Imagine you hire the most well-read assistant in the world. It has absorbed entire libraries, quotes Borges and Bengio with equal ease, and answers in seconds. It has just one condition: at any given moment, it can only see a handful of pages of what the two of you are discussing. Everything else (what you said ten minutes ago, the document you handed it at the start) fades away off the edge of the desk. It is not forgetful out of carelessness. It is simply that its desk has a size.
That desk has a technical name: the context window. It is one of those ideas that, once you see it, reorganizes everything you thought you understood about why artificial intelligence behaves the way it does. And almost no one explains it well.
01 · The unitFirst things first: the model does not read words
To a person, the phrase “memory has limits” is four words. To a language model it is a string of tokens: fragments of text, sometimes a whole word, sometimes a piece of one. The tokenizer breaks language into frequent chunks and assigns a number to each one. Everything the AI reads or writes passes first through that grinder.
This matters because the limit of the context window is not measured in words or characters, but in tokens. When you read that a model has a “128,000 token window”, it is saying how many of those pieces it can keep in view (both what you give it and what it answers) at a single moment.
02 · The shapeA window that slides over a river
The word “window” is exact, and it is worth taking seriously. Picture a very long text passing beneath a rigid frame. The model can only pay attention to what falls inside the frame right now. If the conversation grows longer than the window, the beginning starts to fall outside the frame: technically it still exists, but the model can no longer see it. Hence that feeling that it “forgot” what you agreed on at the start.
Under the hood, what happens inside the frame is a mechanism called attention, introduced in 2017 in the work that founded the current era, “Attention Is All You Need”. Each token looks at all the others inside the window and decides which ones to heed. It is enormously powerful and expensive: the cost grows with the square of the number of tokens. Doubling the window does not double the cost: it quadruples it. That is the economic reason, not a whimsical one, why windows were so small for years.
A bigger window is not just “more memory”. It is a bill that grows with the square.
03 · The growthFrom a postcard to a whole book
The good news is that this bill has become payable at a dizzying pace. In just a few years we went from models that could barely sustain a short conversation to models that swallow a whole book in one sitting.
| Model (year) | Context window | ≈ equivalent to |
|---|---|---|
| GPT-3 (2020) | 2,048 tokens | ~1,500 words · a short article |
| GPT-4 (2023) | 8,192 tokens | ~6,000 words · a chapter |
| GPT-4 Turbo (2023) | 128,000 tokens | ~96,000 words · a short novel |
| Claude 3 (2024) | 200,000 tokens | ~150,000 words · a thick book |
| Gemini 1.5 Pro (2024) | 1-2 million | ~1.5 million words · an entire saga |
04 · The trapHaving the page is not having read it
And here is the nuance almost no one mentions. That a model can fit a whole book into its window does not mean it pays equal attention to all of it. In 2023, a study with a memorable title (“Lost in the Middle”) showed that models remember very well what sits at the beginning and the end of the context, and lose track of what is in the middle. Just like us with a two-hour meeting.
Filling the window to the brim does not always improve the answer; sometimes it worsens it. Burying the key fact in the middle of a gigantic context is an almost sure way to have the model overlook it. More context is not more comprehension.
That is why serious systems do not dump everything on the model and pray. They do something smarter: they store knowledge outside, in an organized memory, and for each question they retrieve only the few relevant fragments and place them (fresh, short, well positioned) inside the window. That technique is called retrieval-augmented generation (RAG), and it was born in 2020 [4]. It is, at bottom, the engineering answer to the limitation this article describes: if the desk is small, do not pile up; bring only the sheet you need, when you need it.
Intelligence is not only in the model. It is in what you put in front of it, and when.
05 · What you take awayThree ideas not to forget
The context window is the AI’s working memory: it is measured in tokens and it has a real edge. Enlarging it costs, quadratically, which is why it was small for so long and why its recent expansion is a feat. But the most counterintuitive thing is that filling it is not understanding it: what decides the quality of an answer is not how much text fits, but which text, well chosen, falls inside the frame.
The next time an AI “loses the thread”, you will no longer read it as a mysterious failure. You will see it for what it is: a desk of a certain size, a river of words flowing beneath it, and the art, still very human, of deciding what deserves to be on the table.
Sources
- Vaswani, A. et al. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS).
- Technical specs and release notes from OpenAI (GPT-3, GPT-4, GPT-4 Turbo), Anthropic (Claude 3) and Google DeepMind (Gemini 1.5), 2020-2024.
- Liu, N. F. et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics.
- Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS.