Picture a ruler. A fine, laboratory ruler marks every millimeter: it can tell you that a screw measures 12.473 mm. Another ruler, the pocket kind, only has marks every half centimeter: with it the same screw measures "about 12 and a half." Both measure the screw. One is heavy and precise; the other fits in your hand. Quantizing an AI model means switching rulers: no longer storing each number with extremely high precision and starting to round it to coarser marks.
And here comes the dizzying question: a large model is billions of those numbers. If you strip precision from every one of them, does the model still think the same way, or are you breaking it bit by bit without noticing?
01 · the weight of the numberWhat exactly is being squeezed
Inside, a language model is a mountain of numbers called parameters or weights. Each one describes how strongly one connection pulls on another. No words are stored there: there are coordinates, millions of them, placed in an immense space. When the model responds, what it does is multiply and add those coordinates over and over.
By default, each of those numbers is stored in 16 or 32 bits: floating point formats that reserve plenty of room for extremely fine decimals. That is the laboratory ruler. A model with 7 billion parameters in 16 bits takes up about 14 gigabytes just to exist in memory. Your laptop does not have that RAM to spare. That is the wall.
We are not squeezing text or knowledge. We are squeezing the ruler that measures each coordinate.
Quantizing means rewriting each number with fewer bits: going from 16 to 8, to 4, sometimes fewer. Eight bits give 256 possible marks; four bits, only 16. Each number is pushed to the nearest mark. The model weighs half, a quarter, an eighth. And that is where the art begins: how far you can widen the marks before the screw can no longer be measured.
02 · the error that piles upWhy it sometimes breaks and sometimes does not
Rounding a number seems harmless. The trouble is that a model does not use one number: it chains millions. Each multiplication drags along the small rounding error from the previous step, and the errors can add up or cancel out across dozens of layers. The question is not "how far off am I on one number?" but "where does the error land after crossing the whole model?".
Geometry helps you see it. The parameters are not spread out evenly: the vast majority are small values, crowded near zero, and a few are outliers, very large, that live far away. If you stretch a single uniform ruler to cover from the smallest to the most gigantic, the marks end up so far apart that the millions of values in the center all get crushed against the same few marks. That is where the model breaks.
That was the key finding of Tim Dettmers and his team in 2022: in large models, a few outliers concentrate almost all the damage. Their method, LLM.int8(), does something elegant: it stores the vast majority of the weights in 8 bits and sets aside the very few outliers to handle them at full precision. This way it manages to run enormous models in 8 bits without losing measurable quality [1]. The geometric lesson: not every number deserves the same ruler.
Quantization does not remove parameters or trim the model: it keeps the same millions of coordinates, just described with fewer decimals. That is why a quantized model still "knows" the same thing in broad strokes. Truly trimming (removing whole connections) is another technique, pruning, and it answers a different question. Here we are only changing the ruler, not the map.
03 · where the bargain endsHow far you can squeeze in practice
Compression is not free, but for a good stretch it almost looks like it. Going from 16 to 8 bits is usually practically painless: the model weighs half and responds almost identically. Going to 4 bits is barely noticeable but noticeable: the model weighs a quarter and, on most everyday tasks, the difference is hard to perceive. Below 4 bits things become expert territory: the rounding error starts to show up in longer or harder answers.
The first half of the squeeze costs almost nothing. The real price lives in the last stretch.
That is why 4 bits became the sweet spot for running models at home. Methods like GPTQ and NF4 (the latter part of the QLoRA work) achieve that compression by carefully adjusting where to place the ruler marks for each block of weights, instead of using a single ruler for everything [2]. The table sums up the trade.
| Precision | Marks per number | Approx. size in memory | Typical quality |
|---|---|---|---|
| 16 bits (original) | ~65,000 | ~14 GB | reference |
| 8 bits | 256 | ~7 GB | almost identical |
| 4 bits | 16 | ~3.5 GB | slight difference |
| 3 bits or fewer | ≤8 | ~2.6 GB or less | visible degradation |
What makes this matter outside the lab is direct: quantization is what allows a model born in a data center with giant GPUs to run on your laptop, on a phone or on a modest server. It is the difference between "cloud only" and "on your own machine." A 4-bit model that fits in the video memory of a consumer GPU can answer you without sending your text anywhere.
There is an honest nuance worth stating plainly: squeezing too hard does not fail in a scandalous way, it fails in a subtle way. An over-quantized model does not start writing in strange symbols; it keeps sounding fluent, but it becomes a little less precise on the hard stuff, makes mistakes more often on long chains of reasoning, forgets a detail here and there. It is a silent wear, and that is why the choice of how many bits to use deserves to be measured on your specific task, not accepted blindly.
In the end, quantizing is an exercise in practical geometry: choosing where to place the ruler marks so that millions of coordinates fit in less space without moving from their spot. When it is done well, the model shrinks and keeps measuring the same screw. When it is squeezed too much, the ruler becomes so coarse that it can no longer tell 12.4 from 12.6. The trick, as with almost everything in these systems, is knowing exactly how far to go.
Fuentes
- Dettmers, T., Lewis, M., Belkada, Y. & Zettlemoyer, L. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. Advances in Neural Information Processing Systems (NeurIPS) 35. Preprint: arXiv:2208.07339.
- Dettmers, T., Pagnoni, A., Holtzman, A. & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. Advances in Neural Information Processing Systems (NeurIPS) 36. Preprint: arXiv:2305.14314. (Introduces the 4-bit NF4 data type.)
- Frantar, E., Ashkboos, S., Hoefler, T. & Alistarh, D. (2023). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. International Conference on Learning Representations (ICLR). Preprint: arXiv:2210.17323.