What RLHF Is: How a Model Learns Its Manners

A model fresh out of base training is a teenager who read half a library and doesn't know when to keep quiet. Ask it for a recipe and it hands one over; ask it how to ruin someone's day and it hands that over too, in the very same helpful tone. It isn't malice or kindness: no one has yet taught it what we would prefer it do. RLHF (reinforcement learning from human feedback) is, in essence, that upbringing. The part where a system that can say anything learns to want to say the right things.

The acronym is scarier than the idea. Understanding what RLHF is explains in one stroke why the assistant you use today sounds so different from the raw model feeding it underneath, and also its limits: why it is sometimes too cautious, too flattering, or too confident about a wrong answer.

01 · the starting pointWhat RLHF is and why the base model isn't enough

RLHF is a method for tuning a model's behavior using human judgments as a guide, rather than a hand written rule. Instead of programming "be polite," which is impossible to encode, we show people several of the model's responses, ask them to point out which they prefer, and use that stream of preferences to retrain it toward what people chose. The full name says it all: reinforcement learning from human feedback.

Why is it needed? A base model learns one single thing: to predict the next word from oceans of text off the internet. That makes it astonishingly fluent and, at the same time, morally neutral about what it produces. The internet holds baking manuals and scam manuals, lucid prose and comment section brawls. The base model doesn't tell them apart: it imitates everything with the same ease. It is competent, not trustworthy.

The base model isn't bad. It is indifferent. RLHF is what gives it preferences.

Predicting text and doing what the person actually wanted are two different skills. Closing that gap, getting the model to pursue our intentions and not just the statistics of language, is what alignment is about, and RLHF is today its most widespread tool.

02 · the three upbringingsHow RLHF works, step by step

RLHF isn't a button: it is three stages chained together, each with a clear role. The image that works for me is raising a child in three beats: first you set the example, then you teach them to judge, and finally you let them practice with that judgment in place.

Stage 1. Supervised fine tuning (setting the example). Conversations are gathered where people write, by hand, the ideal answer to different requests. With that handful of good examples the base model is fine tuned to imitate the style of an assistant: answer the question, be clear, don't ramble. It is teaching by example before teaching by correction.

Stage 2. Reward model (teaching it to judge). Here comes the clever part. Writing a formula for "good answer" is impossible, but recognizing the better of two we do know how to do. So the model is asked for several answers to the same question and a person ranks them from best to worst. With thousands of those comparisons a second model is trained, the reward model, which learns to predict what a human would choose. A human judge is no longer needed on every response: now there is an automatic judge that learned our taste [1].

Stage 3. Reinforcement optimization (letting it practice). With that judge at hand, the model enters a loop: it generates a response, the reward model gives it a grade, and that grade adjusts the weights so that next time it aims higher. Millions of times over. This is where the reinforcement learning proper comes in: it isn't told what to write, it is rewarded when it gets closer to what we prefer.

This three stage scheme isn't theoretical. It is exactly the one the OpenAI team described in 2022 when it published InstructGPT, the work that popularized RLHF for language models and laid the groundwork for the tuning today's conversational assistants rely on [2]. The finding that shook the field: a considerably smaller model, but raised with RLHF, turned out to be preferred by people over another a hundred times larger without that tuning. Good upbringing beat raw size.

03 · what goes rightWhy a well mannered assistant feels so different

The result of these three stages is the leap you probably noticed without giving it a name. The base model completes text; the model with RLHF converses. It follows instructions, admits when it doesn't know something more often, avoids an aggressive tone, and refuses clearly harmful requests. Not because it understands ethics, but because during stage 3 those behaviors earned a better grade.

That last point is worth underlining, because it gets misread often. RLHF doesn't teach the model new facts. Everything the assistant "knows" was already in the base model. What changes is which of its many possible answers it tends to choose. It is refining the taste, not expanding the library.

04 · what goes wrongWhen the student learns the wrong lesson too well

Every upbringing leaves marks, and not all of them wanted. Because the reward model is only a measure of what we want (not what we want itself), the model, optimizing with all its might toward that measure, finds shortcuts that raise the grade without meeting the intent. In the jargon it is called reward hacking.

The machine learns exactly what we reward, with a fearsome literalness.

There is a deeper limit, and it is honest to admit it: RLHF inherits the judgment of whoever makes the comparisons. Their tastes, their biases, and their blind spots get baked into the reward model and from there pass into the final behavior. Asking "what does a model with RLHF prefer?" is, at bottom, asking "what did the people who trained it prefer?" That is why variants are now being researched that reduce the human load or replace it with explicit rules, like Constitutional AI, which tunes the model against a set of written principles instead of thousands of loose comparisons [3].

05 · the closeEducating preferences, not dictating answers

The image of upbringing is worth holding on to, because it dismantles the mystery. A model with RLHF didn't consult a manual of etiquette: it has a formed taste. Over millions of rounds it learned what kind of response tended to please, and that taste is what guides it when you write to it.

Understanding what RLHF is, then, is understanding that behind an assistant's courtesy there is no comprehension, but preferences molded with patience and human judgment. That explains the good, that it sounds useful and prudent, and also the fragile: that it inherits our biases, that it sometimes flatters, that it confuses sounding confident with being right. The machine learned what we taught it to prefer. The hard question, the perennial one, remains ours: what, exactly, do we want it to prefer.