How AI Hallucinations Happen

When a lawyer submitted a legal brief that cited six cases, all completely fabricated by ChatGPT, the judge asked him to explain. He had no good answer. The AI had delivered the case names, courts, dates, and legal reasoning with complete confidence. Nothing about the output signaled that every citation was fictional.

This is AI hallucination: a language model generating false information with no apparent uncertainty. It happens constantly, to sophisticated users and beginners alike, and it is not going away. Understanding why requires understanding what language models actually are , which turns out to be different from what most people assume.

The short answer

AI hallucinations happen because language models are prediction machines, not fact databases. They generate text that seems correct based on patterns in their training data, but they have no ground truth to verify against. When a model says something with total confidence, it is not because it knows it is true, it is because that phrasing looks right statistically. Confidence and accuracy are separate properties, and that gap is where hallucinations live.

The full picture

What a language model actually does

The key insight is that language models don’t retrieve information. They generate text.

When you ask a language model a question, it doesn’t look up the answer in a database, check its facts against a reference, or search for the right response. It predicts the next token , the next piece of text , that is most likely to follow the current input, given everything it learned during training.

This is the mechanism in full: the model reads your prompt, processes it through layers of learned mathematical transformations (the transformer architecture), and outputs a probability distribution over all possible next tokens. It samples from that distribution, appends the token to the context, and repeats , generating text one piece at a time.

The model learned those mathematical transformations by processing enormous quantities of human-written text: books, websites, code, articles, conversations. It learned which patterns of text tend to follow which other patterns. The result is a system extraordinarily good at producing text that looks like the right answer , without any guarantee that it is the right answer.

Confident factual-sounding text is a very common pattern in the training data. So the model produces confident factual-sounding text. Whether the specific content is true is a separate question the architecture has no direct way to answer.

The statistical nature of the problem

A language model’s “knowledge” is implicit in its weights , the billions of numerical parameters adjusted during training. Information that appeared frequently and consistently in the training data is well-encoded. Information that was rare, ambiguous, or contradictory is poorly encoded.

When the model encounters a question that touches well-represented information, its predictions tend to be accurate. When it encounters a question at the edges of its training data , obscure names, niche topics, recent events, specific citations , its predictions become less reliable, but the confidence of the output doesn’t necessarily change.

This is the core problem: the model has no internal signal for “I don’t know this.”

A human who doesn’t know an answer feels uncertainty. They might say “I think…” or “I’m not sure, but…” or simply admit ignorance. A base language model has no such mechanism , it generates plausible text regardless of whether the underlying information is available in its training.

The result is what researchers call confabulation (borrowing the term from neuropsychology, where patients with memory damage generate plausible-sounding but false memories without awareness that they’re doing so). The model fills gaps with plausible-sounding content.

Why citations are a particular trap

Citations are a hallucination hotspot for a structural reason.

When generating a citation, the model needs to produce very specific information: an author name, an article title, a journal name, a publication year, a page range. These specific values need to match a real-world object (an actual paper) exactly , partial matches don’t count.

What the model has learned is the pattern of citations: “Author, A. B. (Year). Title of study. Journal Name, volume(issue), pages.” It generates plausible-sounding values for each slot. A real-sounding name. A plausible-sounding title. A journal that exists. A year within range.

These individual elements often exist in training data, but the combination , this specific paper by this specific author in this specific journal , may not. The model has no way to verify that the combination resolves to a real paper. It just knows that this pattern of text looks like a citation.

When lawyers, researchers, and students have asked AI systems to list references on a topic, they’ve consistently found fabricated citations mixed with real ones , and sometimes entirely fictional bibliographies where every citation is plausible-sounding but doesn’t exist.

Types of hallucination

Not all hallucinations are the same. They cluster into recognizable patterns:

Factual confabulation , The model generates specific facts (numbers, dates, names) that are wrong but plausible. “The Eiffel Tower is 1,063 feet tall” (it’s 1,083 feet). “Einstein was born in 1879 in Stuttgart” (it was Ulm). Small, specific errors that look like knowledge.

Entity fabrication , The model invents people, books, companies, studies, or court cases that don’t exist. A nonexistent academic paper by a real-sounding author in a real journal. A person with a plausible name, title, and background who has never existed.

Reasoning errors dressed as facts , The model reaches a false conclusion through a chain of plausible-looking steps and presents the conclusion as certain. This is particularly dangerous in mathematical reasoning, legal analysis, or any domain requiring precise logical chains.

Temporal confusion , The model’s training data has a cutoff. It may present outdated information as current, or mix information from different time periods. A model trained through 2024 describing 2026 events will either hallucinate or refuse , but the boundary isn’t always clean.

Sycophantic hallucination , When users suggest an answer, models trained with reinforcement learning from human feedback (RLHF) sometimes agree with the suggestion even when the model’s base prediction would have been different. “Is it true that X?” can bias the model toward confirming X even if X is false. The training signal that optimized for human approval can accidentally optimize for agreement.

Why RLHF both helps and introduces new failure modes

Most deployed language models aren’t pure base models , they’ve been fine-tuned with reinforcement learning from human feedback (RLHF). Human raters evaluate model outputs and indicate which responses they prefer. The model learns to produce outputs that humans rate highly.

This significantly improves many aspects of model behavior: responses become more helpful, better-formatted, more aligned with what users actually want, and often more accurate on common questions.

But RLHF introduces a specific failure mode. Human raters can’t verify every claim , they tend to rate responses as better when they sound more confident and authoritative, are well-organized, use appropriate hedging language, and avoid awkward admissions of uncertainty. The model learns to appear calibrated without necessarily being calibrated.

The ironic result: RLHF-trained models can become better at sounding like they know what they’re talking about without becoming proportionally more accurate. They learn to use uncertainty language (“I believe,” “I’m not certain, but…”) on questions where humans seem to expect uncertainty, and confident language where humans expect confidence , regardless of whether those signals correspond to actual model accuracy.

What helps: RAG, calibration, and tool use

Several techniques reduce hallucinations, though none eliminates them.

Retrieval-augmented generation (RAG) is currently the most effective intervention. Instead of relying entirely on the model’s internal weights for factual content, the system retrieves relevant documents from a database or the web and puts them into the model’s context. The model then answers based on retrieved content rather than generating from memory. This converts “recall from weights” (error-prone) into “comprehension of provided text” (much more reliable).

The limitation: RAG only helps when the relevant information can be retrieved and fits in the context window. Reasoning errors, mathematical mistakes, and hallucinations within the retrieved content itself are not addressed.

Constitutional AI and better RLHF can improve uncertainty calibration , training models to refuse confidently when they don’t know, rather than confabulate. Frontier models have gotten meaningfully better at saying “I don’t know” in recent years, though the improvement is uneven across domains.

Tool use , giving the model access to calculators, code interpreters, search engines, or databases , removes certain hallucination-prone tasks from the model’s generative path entirely. A model that runs Python to do math doesn’t hallucinate arithmetic. A model that searches Wikipedia to answer a factual question doesn’t need to recall it from weights.

Structured output and verification chains , prompting the model to reason step-by-step before answering, to identify its uncertainty explicitly, or to check its own work , can catch some errors. Not reliable as a primary strategy, but meaningful as an additional layer.

What doesn’t help (as much as people hope)

“Just use a bigger model.” Larger models hallucinate less on common topics. But they still hallucinate on niche topics, specific citations, and recent events , and they hallucinate with more fluency, making false outputs harder to spot.

“Just ask it to be accurate.” Telling the model “don’t make things up” or “only state what you know for certain” helps at the margins but is not a reliable control. The model is still generating text that predicts plausible continuations; the instruction shifts the distribution slightly but doesn’t change the fundamental mechanism.

“Check if it says ‘as an AI’” , Models trained with RLHF have largely learned to avoid this phrase because human raters found it annoying. It’s not a useful signal of accuracy.

How to work with this

The practical implications are cleaner than they might seem.

Use AI for tasks where the output is verifiable, creative, or structural , drafting, summarizing provided content, writing code you can test, brainstorming, restructuring documents. The model’s generative ability is genuinely powerful here.

Treat any specific factual claim , a name, date, citation, statistic, quote , as a hypothesis to verify, not a fact to trust. This is true even for confident-sounding claims on topics where the model seems knowledgeable. Especially for citations: never submit, publish, or rely on an AI-generated citation without independently verifying it resolves to a real source.

When you need factual recall, use tools that ground the model in retrieved content: AI with web search enabled, RAG-based tools, or simply paste the relevant content yourself and ask the model to work with it.

The mistake is treating language models like search engines, systems that retrieve stored facts. They’re not. They’re text generators that learned from text. Research from MIT and Harvard has shown that hallucination rates vary significantly by domain, and Google DeepMind’s analysis categorizes hallucinations into distinct types with different root causes. That distinction, once internalized, makes hallucination feel less like a flaw and more like a predictable property of a genuinely powerful but fundamentally different kind of tool.

Why it matters

If you use AI tools for work, understanding hallucinations is not optional. The lawyer who submitted fabricated citations is not an outlier, he is a warning. The confidence with which AI produces false information looks exactly like the confidence with which it produces true information, and that makes it dangerous precisely because it is invisible. The more you rely on AI for research, writing, or decision-making, the more you need a system for verification that treats every specific claim as unverified until checked.

This matters for how you build products too. If you are shipping AI features to users, the hallucination problem does not get easier at scale. A thousand users asking a thousand questions will generate a thousand confident-sounding answers, some of which are wrong, and your support team will not be able to tell the difference either unless you build retrieval, citations, or uncertainty signals into the product. The architectural solutions (RAG, tool use, calibration) are not optional enhancements, they are the baseline for any serious application.

The bigger shift is how you think about AI itself. A tool that generates text rather than retrieves facts is a fundamentally different kind of tool than a database or a search engine. Treating it like a faster encyclopedia gets you in trouble. Treating it like a collaborator who needs verification, context, and domain-specific grounding gets you value. The hallucination problem is the price of a model that can be creative, generalize, and handle novel situations. That capability is also what makes it useful. You do not get one without the other.

Key terms

Hallucination Generating factually incorrect content with apparent confidence. The term borrows from neuropsychology and is imprecise , “confabulation” is arguably more accurate, since the model isn’t perceiving things that aren’t there, it’s generating plausible-sounding content without a ground-truth check.

Confabulation The neuropsychological phenomenon of generating false memories without awareness of doing so, used as an analogy for AI hallucination because it captures the automatic, confidence-without-awareness quality of the problem.

RLHF (Reinforcement Learning from Human Feedback) The training technique that fine-tunes base language models to produce outputs humans prefer. Substantially improves helpfulness and reduces many hallucinations, but can also train models to appear calibrated rather than be calibrated.

RAG (Retrieval-Augmented Generation) Augmenting a language model with a retrieval system that fetches relevant documents at inference time. Currently the most effective practical intervention against hallucination for factual question-answering.

Calibration The degree to which a model’s expressed confidence matches its actual accuracy. A well-calibrated model is uncertain when it’s likely to be wrong and confident when it’s likely to be right. Base language models are poorly calibrated; RLHF improves this imperfectly.

Common misconceptions

“It’s just a bug that will be patched.” Hallucination is structural , an emergent property of learning to predict text without a factual ground-truth mechanism. It can be reduced but not eliminated without fundamentally changing what these systems are. Expecting it to disappear with a software update misunderstands the problem.

“AI lies.” A lie requires knowing the truth and choosing to say something false. A hallucinating model has no ground truth to depart from. Framing hallucination as lying leads to the wrong interventions (teaching the model ethics) rather than the right ones (architectural changes, retrieval, verification).

“If the model sounds uncertain, it’s less likely to be wrong.” Uncertainty language is a trained behavior, not a reliable accuracy signal. Models learn when to express uncertainty because humans rate those responses well in certain contexts. A model can express uncertainty about a true fact and confidence about a false one.

“Open-source models hallucinate more.” Not necessarily. Hallucination rates depend heavily on model size, training data quality, and the specific domain being tested. Some open-source models have been shown to hallucinate less than proprietary models on specific benchmarks. The relationship between openness and accuracy is not direct.