I asked a model for the reference to a study I half remembered. It gave it to me in a second: an author with a credible surname, a prestigious journal, a year, a page number. The citation was perfect. It was also nonexistent. Not the paper, not the volume, not that page. The model did not hesitate, did not stumble, did not add an "I think". It looked me in the eye, so to speak, and invented a source with the same calm a child uses to explain where the cat went after the vase broke.
We call that a hallucination: when an AI system produces information that sounds plausible and is, quite simply, false. And the word misleads us, because it suggests a breakdown, a short circuit, something gone wrong. It is not that. The hallucination is not a system failure. It is the system doing exactly what it knows how to do, applied to ground where it has nothing to stand on.
01 · the habitWhy AI hallucinates: it predicts, it does not look things up
To understand why AI hallucinates you have to let go of a deeply rooted idea: that the model "searches" for the answer in some internal file, the way you open a drawer. There is no drawer. A language model does not store facts on tidy index cards; it stores an enormous statistical sense of which word tends to follow which others. Faced with your question, it does not retrieve a fact: it predicts the most likely continuation, word by word.
It is a habit, not a lookup. Like the friend who always has an anecdote ready for any topic: he is not lying on purpose, he is completing the conversation with whatever his experience tells him "fits right there". When he really lived that anecdote, he gets it right. When he did not, he builds it from scraps of others and tells it with the same ease. The model does exactly that, on an industrial scale and without the bother of blushing.
The model does not look for the truth. It looks for the most likely continuation. Sometimes they coincide. Sometimes they do not.
02 · the blind spotThe model does not know what it does not know
Here is the heart of the matter, and it is more unsettling than it seems. A model has no internal gauge that lights up when it enters unknown ground. It does not distinguish "I know this well" from "I am improvising this". For its machinery, generating a fact it memorized a thousand times and generating one it never saw are the same operation: choosing likely words. The confidence with which it answers does not measure how much it knows; it measures how fluent the sentence came out.
That is why a hallucination sounds so convincing. It comes with no label of doubt because, for the system, there was no doubt. The poise is not a mask that hides the ignorance: it is that the ignorance, in there, does not feel any different from knowledge. It is a structural blind spot, not a bad attitude.
A widely cited work by Ji and colleagues reviewed hundreds of studies and proposed a useful distinction [1]. There is intrinsic hallucination, when the model contradicts the very source it was given (you hand it a text and it summarizes it saying the opposite). And there is extrinsic hallucination, when it asserts something that cannot be verified against any source: the invented citation from my anecdote. The second is the more treacherous, because there is nothing in the text with which to catch it in the act.
The term comes from psychiatry and suggests a false perception. Several researchers prefer to call it confabulation or plainly a factual error, because the model does not "perceive" anything: it fills gaps with plausible material, just as a person with certain memory injuries completes what they cannot recall without realizing they are doing it. The label matters little; the mechanism matters a great deal.
03 · the slippery groundWhen it is more likely to invent
Not every question carries the same risk. Hallucination grows exactly where the statistical pattern thins out: very specific and infrequent facts (an exact date, a figure, a page number), niche topics with little presence in the training data, facts later than the model's knowledge cutoff, and those questions whose premise is already false: if you ask it to explain an event that never happened, it will explain it happily, because its job is to continue, not to audit your premise.
There is a parenting rule that applies just as well to models: the ground where they invent the most is the one where you most want an answer and there is least data to give it. Academic citations are the perfect case. They have a very regular shape (surname, year, title, journal) that the model imitates flawlessly, and a very specific content that it almost never memorized. Solid form, hollow substance: the exact recipe for a convincing invention.
| Situation | Risk | How to defend yourself |
|---|---|---|
| Exact citation, DOI or page number | Very high | Verify the source before using it |
| Fact later than the model's cutoff | High | Give it the source or use retrieval (RAG) |
| Niche, poorly documented topic | High | Ask it to mark its level of certainty |
| Question with a false premise | High | Check the premise yourself before asking |
| General and frequent knowledge | Low | Even so, cross-check what matters |
04 · the leashHow to reduce ChatGPT inventing facts
The good news, following the parenting analogy, is that a model can be put on a leash without stripping it of usefulness. None of this eliminates hallucination entirely (it is wise to distrust anyone who promises you that), but it does lower the frequency a lot and, above all, gives you something to catch it with.
Give it the source, do not ask it to recall it. The most effective technique is called RAG, retrieval-augmented generation: instead of trusting the model's memory, the system first searches for real documents and hands it that material so it can answer leaning on it [2]. It is the difference between asking someone from memory and giving them the book open to the right page. The model keeps predicting, but now it predicts over a text that does exist.
Ask it to cite and verify the citation. When you demand a claim with a source, do not settle for it "having the shape of a source": check that the link opens and that it says what the model asserts. A well-formatted citation is not a true citation.
Let it doubt out loud. Instructing the model to answer "I don't know" when it has no basis, and to separate what it knows from what it supposes, reduces the automatic poise. It does not cure the blind spot, but it teaches it to raise its hand.
Do not ask the AI to recall the fact. Give it the fact and ask it to work with it.
In the end, understanding why AI hallucinates changes the relationship you have with it. It stops being an oracle that sometimes fails and becomes what it truly is: an astonishingly fluent predictor, plenty useful, that does not distinguish between what it knows and what it improvises. That is its blind spot. And as with any brilliant, slightly tall-tale-telling companion, the rule is simple and as old as the world: in what matters, trust, but verify.
Sources
- Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., et al. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12), art. 248. DOI: 10.1145/3571730. Available at arXiv:2202.03629.
- Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems (NeurIPS) 33. arXiv:2005.11401.
- OpenAI. Models | Limitations. Official documentation on the knowledge limits and the possibility of incorrect answers from the models. platform.openai.com/docs.