A parameter in an AI model is a knob. A number the model can turn up or down to shift, just a little, the answer it produces. When someone says a model has "70 billion parameters", they mean it has seventy billion knobs, and that during training each one found the setting where, taken together, the model was wrong as rarely as possible. There are no hand written rules in there. There are tuned knobs.
The word sounds technical, but the idea is geometric and simple: a parameter is a coordinate. Move it and the surface the model draws over the data tilts a little differently. With two or three knobs you draw a straight line or a clumsy curve. With billions, you draw something able to tell "this sentence sounds like seventeenth century Spanish" apart from "this one sounds like an installation manual". Everything that follows is about understanding where those knobs come from, how many you need, and why more is not always better.
01 · the knobA parameter is a number the model tuned, not one it was programmed with
It is worth separating two things that often get confused. One is the weight: the number that multiplies each signal entering a neuron. The other is the bias: a small value added at the end to shift the result. Weights and biases together are the parameters. They are what the model learns. And learning, here, means exactly one thing: turning each knob in the direction that reduces the error.
Do not confuse them with the hyperparameters, which are the decisions a person makes before training (how many layers, how large each one is, how fast to adjust). The hyperparameters are the design of the box; the parameters are the knobs the box turns on its own. Nobody writes by hand the value 0.0374 of knob number forty million. That number appeared because, data point after data point, training pushed it there.
Nobody programs a knob's value. The model turns it on its own, data point after data point, until the error stops falling.
02 · the tuningWhere they come from: training turns the knobs
At first, the knobs start in almost random settings. The model, using them, produces nonsense. Then the cycle begins: it is shown a data point, it compares its answer with the correct one, it measures how wrong it was, and it adjusts each knob a little in the direction that would have reduced that error. That direction is computed with a technique called backpropagation, which spreads the blame for the error across all the responsible knobs. Then another data point, another adjustment. Millions of times.
That method of spreading the error and turning each knob was popularized by Rumelhart, Hinton and Williams in a 1986 paper now considered foundational [1]. The idea is old; what is new is the scale. When you have billions of knobs and trillions of words of training, that same humble cycle (predict, measure the error, adjust) produces a model able to write, translate and reason in ways that surprise even those who build it.
A natural question comes up here: if more knobs give more capacity, why not always use the maximum possible? Because each knob is also a chance to get it wrong. A model with too many parameters for too little data starts to memorize instead of learning: it knows the examples by heart but fails on anything new. That flaw is called overfitting, and it is the reminder that the big number is not the goal, balance is.
A common mistake is to think that "70 billion parameters" means the model stored 70 billion texts. It did not. The parameters are the knobs of the finished model; the data are the words used to tune them, and they are usually far more numerous. A reference guide suggests that, to train well, it pays to feed the model on the order of twenty words for every parameter [2].
03 · the scaleBillions: what the announced number means
When a company announces the size of a model, it almost always means the parameter count. It is their way of saying "how many knobs this thing has". The scale jump of the last decade is real: it went from models with millions of parameters to models with hundreds of billions. And the finding that drove that race has a name: the scaling laws, which observed that performance improves predictably when you increase parameters, data and compute all at once [3].
| Model (year) | Parameters (approx.) | Order of magnitude |
|---|---|---|
| BERT base (2018) | 110 million | 10⁸ |
| GPT-2 (2019) | 1.5 billion | 10⁹ |
| GPT-3 (2020) | 175 billion | 10¹¹ |
| Llama 3 (2024) | 8 to 70 billion | 10⁹-10¹⁰ |
And there is the trap of the big number. A model with fewer parameters, but trained on more data and better tuned, can beat a much larger one. The work on the Chinchilla family of models showed exactly that: many giant models were undertrained, and a smaller one fed with more data beat them with less compute [2]. The lesson is uncomfortable for marketing: the parameter count measures potential capacity, not delivered intelligence.
The parameter count measures how many knobs the model has, not how well it turned them.
There is also a very practical reason to prefer models with fewer parameters: they fit in less memory and respond faster and cheaper. A model with seventy billion knobs needs a lot of hardware just to keep them loaded. That is why quantization exists, storing each knob at lower precision so the model weighs less without losing too much. But that is another story, and it deserves its own drawing.
Next time you read "a model with X billion parameters", you already know how to translate it: X billion knobs that training set, one by one, chasing the lowest error. A huge number and, even so, only half the story. The other half (what data, how much tuning, how fine the balance) is what decides whether those knobs, in the end, sound in tune.
Sources
- Rumelhart, D. E., Hinton, G. E. & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, pp. 533-536. DOI: 10.1038/323533a0.
- Hoffmann, J. et al. (2022). Training Compute-Optimal Large Language Models ("Chinchilla" models). arXiv:2203.15556. (Ratio of ~20 training words per parameter for compute optimal training.)
- Brown, T. B. et al. (2020). Language Models are Few-Shot Learners (GPT-3, 175 billion parameters). arXiv:2005.14165. See also Kaplan et al. (2020), Scaling Laws for Neural Language Models, arXiv:2001.08361.
- Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL-HLT 2019, pp. 4171-4186. ACL Anthology: aclanthology.org/N19-1423.