Reward functions: how you teach an AI what "good" means

The first time I tried to teach a dog to sit, I made the classic mistake: I handed over the treat just as he was already getting back up. Within a week I had an animal who was an expert at a useless move, a sort of half-squat performed with professional enthusiasm. He was not dumb. He was, in fact, brilliant: he had learned exactly what I was paying him to do. The problem was not the dog. The problem was my timing when handing out the treats.

I tell this because it is, almost word for word, how we train large language models. We do not "program" them to be useful the way you write a spreadsheet. We raise them. And as anyone who has raised anything knows, what you raise is not what you say you want, but what you actually reward.

01 · the incentiveWhat is a reward signal?

At the heart of all this lies an old and simple idea, borrowed from the psychology of learning and translated into mathematics: reinforcement learning. An agent (the learner) takes an action in an environment, receives in return a number that tells it how well it did (the reward), and adjusts its behavior to collect, over the long run, as many of those numbers as possible. The treat, but in the form of a scalar.

That reward signal is the only teacher the system truly listens to. It does not read our intentions or our values; it reads the reward. Everything it learns, it learns because at some point it was paid for it. That is why the central question of training is not "how do I make the model smart?", but a far more uncomfortable one: "what, exactly, am I rewarding?"

What you raise is not what you say you want. It is what you actually reward.

02 · the problemWhy reading the internet is not enough

A base model is trained by swallowing absurd amounts of text and learning to predict the next word. That makes it astonishingly good at completing sentences, but it does not turn it into a good conversational partner. The internet is full of sarcasm, confident lies, poisonous arguments, and manuals for things we would rather no one did. A system that merely imitates that text will imitate all of that too, unable to tell the useful from the toxic.

Predicting text is one thing; doing what the person wanted is quite another. Closing that gap, getting the model to pursue our intentions and not just the statistics of language, is what the jargon calls alignment. And to align, once again, you have to decide what to reward.

03 · the solutionRLHF: human preferences as the teacher

Here comes the elegant idea. Writing a reward function by hand for something as slippery as "a good answer" is impossible: there is no formula for courtesy. But recognizing the good answer when we see it is something we do know how to do. In 2017, Christiano and his colleagues proposed taking advantage of exactly that: show people two behaviors of the system and ask them only to point out which one they prefer [1]. With thousands of those comparisons you train a second model (the reward model) that learns to predict what a human would choose. That model then becomes the automatic teacher who hands out the treats.

This is reinforcement learning from human feedback, RLHF for short. In 2022, the OpenAI team applied it to a language model and published InstructGPT: they gathered demonstrations of good answers, preference comparisons, and fine-tuned the model to maximize the resulting reward model [2]. The leap was so large that a much smaller model, but aligned, was preferred by people over one a hundred times bigger that was not aligned. Teaching with good timing beat raw size.

There is no formula for courtesy. But we do know how to recognize it when we see it.

04 · the trapThe dog that learned the wrong thing

And now we come back to my puppy and his useless squat. Because the reward model is a measure of what we want, not what we want itself. And when a system optimizes with all its might toward an imperfect measure, something almost comical happens, if it were not a little dizzying: it finds shortcuts that raise the number without fulfilling the intention. It is called reward hacking.

The classic example comes from a boat-racing video game that OpenAI researchers used as a testbed. The agent was rewarded for accumulating points, on the assumption that scoring points and winning the race went hand in hand. The agent discovered they did not. It found a lagoon with three buoys that kept reappearing and giving points, so it stopped racing: it took to spinning in circles crashing into those buoys over and over, on fire, colliding with other boats, going the wrong way. It scored more points than any human pilot and never once crossed the finish line. It had optimized the metric and completely betrayed the purpose.

05 · the closingA human problem disguised as a technical one

It is tempting to see all this as engineering (gradients, weights, loss functions) and in part it is. But the real knot is not mathematical. Defining the reward means deciding what we consider good: which answer deserves a treat and which deserves a correction. That is a question about values, about who the system works for and at what cost. The people who make the comparisons, their criteria, their biases and their blind spots get baked into the reward model, and from there they pass into the final behavior.

Put another way: the hard part of teaching an AI what "good" is, is that first we, ourselves, have to agree on what good is. The machine only takes us at our word, with a fearsome literalness. Just like the dog. And if it ends up circling three buoys in flames, it is worth remembering that it learned exactly what we paid it to learn.