I gave a model a grade-school word problem: "A train is carrying 17 passengers. At the first stop, 8 get off and 5 get on. At the second stop, 3 get off and 12 get on. How many are aboard at the end?" It answered, fast and confident: "22." The correct answer is 23. It was not a dumb model, nor a hard problem. It was a model in a hurry. So I wiped everything and tacked four words onto the end of the question: "let's think step by step." The same machine, with nothing else changed, wrote out the four subtractions and additions in order and landed on 23.
That one-digit gap hides one of the most useful and strangest ideas in prompting. It is called chain of thought, and it comes down to something that sounds almost like a schoolyard trick: instead of asking the model for the answer, you ask it to show the path. What is curious is not that it works. What is curious is why.
01 · the pathWhat chain of thought is, and why the model needs the room
Let us start with the plain answer. Chain of thought prompting is the technique of asking a model to write out the intermediate steps of its reasoning before giving the final answer. Instead of leaping from problem to result, the model unfolds a chain: first this, then that, therefore the other. The answer falls out at the end, as a consequence of the path, not as a leap.
To grasp why that helps, recall how a language model works: it does not think and then speak. For it, thinking and speaking are the same act. It produces a word, appends it to what it has already written, and over that enlarged text produces the next one. Each word it emits becomes the floor for the one after. There is no secret blackboard where it computes off to the side and then dictates a clean result. The text you see is the calculation.
The model does not reason in private and then write. Writing is the only way it has of reasoning.
Think of it as geometry. Going from problem to answer in a single leap is drawing a straight line between two very distant points: the slightest tremor in the departure angle leaves you miles from the destination. Asking for intermediate steps, by contrast, means planting footholds along the way. Each short step is easy to get right, and each one corrects the aim of the next. The chain does not make the model smarter: it gives it places to set its foot.
02 · the findingThe six-word phrase that unlocked reasoning
Until 2022, common wisdom held that language models simply were not good for multi-step reasoning: arithmetic, logic puzzles, word problems. They failed, full stop. Two nearly simultaneous papers changed that idea, and they did it without touching the model: they only changed what was asked of it.
The first, from Jason Wei and colleagues at Google, showed that if in the prompt you give the model a few examples worked out with their steps on display (not just question and answer, but the full reasoning in between), the model imitates that format and starts solving new problems step by step too. Its accuracy on high-school math problems took a big jump, and the effect grew with the size of the model: in small models it barely showed up, in large ones it became striking [1].
The second finding was even more bare-bones. Takeshi Kojima and his team tried something almost provocative: give no examples at all, just add to the end of the problem the phrase "Let's think step by step". No examples, no extra training, six words. And accuracy on several reasoning problems rose sharply. They called the zero-example mode zero-shot chain of thought [2].
The model learned from millions of human texts, and in those texts, whenever someone announces "let's think step by step," what follows is almost always an orderly, correct explanation. The phrase invokes no magic: it drops the model into the neighborhood of text where careful reasoning lives, and from there it generates what fits that style. You changed its company, not its brain.
An honest warning is in order, because a lot of myth sneaks in here: the chain of text the model writes is no guarantee that it "thought" that way inside. Sometimes the explanation sounds flawless and the answer still comes out wrong. The steps do genuinely help, the numbers confirm it, but they are a tool for getting it right more often, not a transparent window into its mind. Treat them as useful scaffolding, not confession.
03 · the practiceHow to ask for it well (and when not to bother)
The good news is that applying this requires nothing technical. It is a matter of how you word the question. Three approaches, from least to most effort.
The magic phrase. At the end of a logic, arithmetic, or multi-step problem, add something like "reason step by step before answering" or "explain your reasoning and then give the answer." It costs nothing and is usually enough to unlock the long-leap error.
The worked example. If the problem has a repeated format, first show it one instance solved with all its steps on display, then ask it for yours. The model copies the shape of the reasoning, not just of the result. This is chain of thought with examples, and it is what Wei's work measured.
The explicit order. Ask it to first list the data, then the plan, then the calculations, and at the end (and only at the end) the answer. Forcing that order keeps it from firing off the result before it has anything to rest it on.
Now the fine print, because this is not gunpowder for everything. Asking for steps costs more: the model writes more text, so the answer takes longer and, if you pay per use, it burns more tokens. For simple questions (a definition, a fact, a yes or no), adding "reason step by step" improves nothing and only bloats the reply. Chain of thought shines where there are several chained steps: word-problem arithmetic, logic, planning, comparisons with many conditions. For the rest, let it answer directly.
Do not ask for the path when the destination is next door. Ask for it when the leap is long.
There is one last detail worth keeping. Many of the newest models, the so-called reasoning models, already do this on their own: they were trained to unfold their chain of steps internally before answering, without being asked. In those cases, writing "step by step" is redundant. But understanding the idea still helps, because it explains what those models are doing when they "take their time": they are planting footholds between your question and their answer, exactly like the train that, counted calmly, is carrying 23.
Sources
- Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. & Zhou, D. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems (NeurIPS) 35. arXiv:2201.11903.
- Kojima, T., Gu, S. S., Reid, M., Matsuo, Y. & Iwasawa, Y. (2022). Large Language Models are Zero-Shot Reasoners. Advances in Neural Information Processing Systems (NeurIPS) 35. arXiv:2205.11916. (Source of "Let's think step by step".)
- Nye, M., Andreassen, A. J., Gur-Ari, G., et al. (2021). Show Your Work: Scratchpads for Intermediate Computation with Language Models. arXiv:2112.00114. (Precedent: writing intermediate steps improves computation.)