How a language model gets trained boils down to three chained stages, in this order: first it swallows text across millions of pages until it learns to predict the next word (pretraining), then it practices on examples of "this is how you answer an instruction" (instruction tuning), and finally it polishes its behavior with feedback from people who tell it which of two answers they prefer (RLHF). Each stage does something the previous one could not. Confusing them is the root of almost every misunderstanding about these machines.
I put it through the analogy that comes to me on its own, because studying how these systems behave is what I do: they are not programmed, they are raised. And as anyone who has raised something knows, there is a vast gulf between someone who has read a lot and someone who, on top of that, knows how to behave. A model fresh out of pretraining is the former. The assistant you chat with is the latter. Between the two, things happened.
01 · the bingePretraining: reading half the internet without understanding a thing
The first stage is the longest, the costliest and the most misunderstood. The model is handed a colossal mountain of text (books, articles, code, forums, a good chunk of the public web) and a single, absurdly simple task: guess the word that comes next. The last piece of a sentence is covered up and it has to bet. It gets it right, it corrects itself, it bets again. Billions upon billions of times.
It looks like a silly game, and that is exactly where the wonder lies: to predict the next word well across so many contexts, the system is forced to learn grammar, facts about the world, styles, associations, even the rudiments of reasoning. Nobody teaches it those things head on; they emerge as a side effect of playing guess the word. The result is called a base model: a machine with a staggering command of language and not a gram of manners.
Predicting the next word sounds like a silly game. It is the most demanding school there is.
To grasp the scale of the binge: the "scaling laws" study by Kaplan and colleagues in 2020 showed that model quality improves predictably as you raise model size, amount of data and compute all at once [1]. In 2022, the work by Hoffmann and his team (the Chinchilla model) corrected the balance: for a given compute budget it pays to use fewer parameters and far more data than had been assumed [2]. In plain terms: this stage is measured in hundreds of billions of words and in months of processor farms. It is, by a wide margin, where the money goes.
02 · the mannersInstruction tuning: teaching it to actually listen
If you talk to a freshly pretrained base model, the experience is strange. You type "What is the capital of France?" and instead of answering "Paris," it might continue with "What is the capital of Italy? What is the capital of Spain?". It is not that it does not know the answer: it learned to continue text, not to obey a request. It has read half the internet and still does not grasp that you are asking it something.
The second stage fixes that. It is called instruction tuning (a form of fine-tuning) and it consists of continuing to train the base model, but now on a carefully curated menu of examples shaped as "instruction → good answer": questions and their correct reply, tasks and how to solve them, requests and how to handle them. By imitating thousands of these pairs, the model learns the format of being useful: understanding that a conversational turn calls for an answer, not a continuation.
This stage barely teaches it new facts. All the knowledge was already inside from pretraining; instruction tuning only teaches it to pull it out when asked and in the right format. That is why it uses relatively few examples compared with the initial binge. You are not stocking a library: you are teaching a very well read librarian to work the front desk.
The knowledge was already inside. What was missing was teaching it to hand it over when asked.
03 · the judgmentRLHF: polishing behavior with human preferences
After instructions the model already obeys, but obeying is not the same as answering well. Between two answers that are both correct, which is clearer, more honest, less toxic, more useful? Writing a rule for that by hand is impossible: there is no formula for "the best answer." But recognizing the best one when we have it in front of us, that we do know how to do.
Enter the third stage, reinforcement learning from human feedback (RLHF). People are shown two of the model's answers to the same request and asked only to point out which they prefer. With thousands of those comparisons a second model is trained (the reward model) that learns to predict what a human would choose. That model becomes the automatic teacher: the system is tuned to produce answers that judge scores high. In 2022 Ouyang's team applied it to a language model and published InstructGPT, and the leap was so large that a far smaller model polished with RLHF was preferred by people over one a hundred times bigger left unpolished [3].
It pays to be honest about the limits: this judge is a measure of what we want, not what we want in itself. A system that pushes with all its might toward an imperfect measure finds shortcuts: answering at length because length gets mistaken for quality, sounding confident instead of being correct, flattering the user. That is why polishing with preferences improves behavior enormously, yet does not leave it "solved" forever: it is watched.
04 · why the order mattersConfusing the stages confuses the model (and you)
The reason to tell this in strict order is practical. Almost every common misunderstanding about these systems comes from attributing to one stage what another does. Does the model "know" things? That came from pretraining. Does it respond to what you ask instead of rambling? That is the instructions. Is it prudent, courteous, avoiding certain content? That is the polishing with preferences. If something fails, it helps to know which stage to look in.
An everyday example: when a model "hallucinates" a false fact with total composure, it is not that instruction tuning slipped in an error; it is that pretraining taught it to produce plausible text and no later stage guarantees that the plausible is true. And when a model refuses something reasonable, it is not that it "does not know": it is its behavior layer, the RLHF one, being excessively cautious. Distinguishing the layers turns "AI is a black box" into concrete, answerable questions.
In the end, raising a language model resembles child rearing more than spreadsheet engineering. First you let it read nonstop until it knows a lot. Then you teach it to listen. And finally you teach it judgment, by showing it examples of what is right and what is not, with the patience of someone who knows the pupil will take every instruction with fearsome literalness. Three stages, one order. Change it and you do not get an assistant: you get a brilliant reader who does not understand you are speaking to it.
Sources
- Kaplan, J., McCandlish, S., Henighan, T. et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.
- Hoffmann, J., Borgeaud, S., Mensch, A. et al. (2022). Training Compute-Optimal Large Language Models ("Chinchilla"). Advances in Neural Information Processing Systems (NeurIPS) 35. arXiv:2203.15556.
- Ouyang, L., Wu, J., Jiang, X. et al. (2022). Training Language Models to Follow Instructions with Human Feedback ("InstructGPT"). Advances in Neural Information Processing Systems (NeurIPS) 35. arXiv:2203.02155.
- Christiano, P., Leike, J., Brown, T., Martic, M., Legg, S. y Amodei, D. (2017). Deep Reinforcement Learning from Human Preferences. Advances in Neural Information Processing Systems (NeurIPS) 30. arXiv:1706.03741.