What is a token? The LEGO brick AI uses to build language

Picture handing a sentence to an AI model and, before it can "understand" it, someone runs it through a grinder. The sentence goes in whole and comes out in pieces: bits of words, sometimes whole words, sometimes just three letters and a space. Those pieces are the tokens. And here is the strange part: the model never sees your sentence. It sees the pieces.

We have been told that "AI reads text". It is a white lie. What it reads is a row of numbered chips. Like when you build something out of LEGO: you do not handle "a house", you handle bricks. The house is what emerges when you snap them together.

01 · the brickA token is a chunk of text, not a word

The first thing to let go of is the idea that a token equals a word. Sometimes it does. Sometimes a token is a short, common word like "of" or "house". But very often it is a fragment: a root, an ending, a syllable, even a single character with the space in front of it.

Why break words apart? Because language is infinite and the model's memory is not. If every possible word had its own chip, the model's dictionary would be enormous and useless for words it never saw: proper names, jargon, typos, "supercalifragilistic". With reusable pieces, on the other hand, it can build any word by snapping together parts it already knows.

The model does not memorize words. It learns to build them from pieces.

02 · the grinderTokenization: how the cut point is decided

The cut is not random, nor is it the syllables you learned in school. It is done with a method learned from the texts themselves. The most famous one is called BPE (Byte Pair Encoding).

The idea, at a high level, is surprisingly simple. You start with the text reduced to loose characters. Then you look for the pair of pieces that appears together most often and fuse them into a single new piece. You repeat: you find the next most frequent pair, fuse it, and again. Thousands of times. In the end you have an inventory of pieces where the very common bits ("tion", "un", "ing") ended up as chips of their own, while the rare stuff stays broken into small chunks.

Curiously, BPE was born in 1994 as a data compression trick (Gage), and had nothing to do with language. It was Sennrich, Haddow and Birch who in 2016 adapted it to break words apart in machine translation [1][2]. From there it jumped to nearly every language model you use today.

03 · the billWhy this matters to you: cost and limits

This is where it stops being trivia and starts being practical. Models do not charge or measure in words: they measure in tokens. Two direct consequences.

The cost. When you pay to use a model through an API, you pay per input token and per output token. A text with many long or uncommon words burns more tokens than one of the same length made of simple words. Writing "utilize" costs, literally, more than writing "use".

The context limit. Every model has a cap on how many tokens it can hold "in its head" at once: your message plus its reply. That cap is counted in tokens, not in pages. That is why a long conversation eventually "forgets" the beginning: it fell outside the token window.

How much is a token in real life? As a rule of thumb, in English 1 token ≈ 4 characters ≈ 0.75 words (that is, 100 tokens ≈ 75 words) [3]. In Spanish it comes out a bit pricier: our words are longer and have more endings, so 1 token ≈ 0.7 words and often less. Translated: the same text in Spanish tends to burn more tokens than in English. It is not your imagination.

To the model, a paragraph has no words. It has a price.

None of this changes how you write. But knowing there is a chip grinder behind it all explains three mysteries at once: why it sometimes chokes on proper names, why long conversations lose their memory, and why the bill climbs faster in Spanish than in English.

The next time you type "cat" and the model answers you, remember: it never saw a cat or the word "cat". It saw a chip. And with chips, as with LEGO, you can build almost anything.