Hallucinations: Why AI Lies and What to Do About It

AI hallucinations happen when a model generates confident, fluent text that is factually wrong. Learn why it happens mechanically, how to spot it, and the practical patterns that actually reduce it.

TL;DR: AI models don't "know" facts the way you do. They predict the most statistically plausible next word - and when their training data has gaps, they fill those gaps with confident-sounding fiction. This is called a hallucination. It's not a bug that will be patched; it's a structural feature of how these systems work. The good news: you can design around it.

What a Hallucination Actually Is

The term is borrowed from psychology, but the AI version is closer to what clinicians call confabulation - the brain (or in this case, the model) generates a plausible story to fill a gap in memory, with no awareness that the story is invented.

An AI hallucination is text that is:

Classic examples: a model inventing a scientific paper that doesn't exist, citing a court case with the right format but the wrong ruling, or naming a product feature that was never shipped. The output looks right. That's what makes it dangerous.

Researchers distinguish two main types:

The Mechanical Reason This Happens

Every major language model is trained on a variant of the same objective: predict the next token. Given everything that came before, what word most plausibly comes next? Do that billions of times, across hundreds of billions of tokens, and you get a system that has internalized an enormous amount of the world's text - its patterns, its facts, its errors, and its contradictions.

The problem: next-token prediction optimizes for plausibility, not truth. The training data contains factual mistakes alongside correct information, and the model learns both with similar confidence. When the training data is dense on a topic, the model answers well. When the data is sparse - obscure events, recent developments, niche domains - the model fills the gap by generating the most statistically plausible continuation. That continuation might be totally wrong.

OpenAI researchers Adam Kalai and Ofir Nachum formalized this in a 2025 paper: knowledge gaps are inevitable with non-exhaustive training data. They introduce the concept of the "singleton rate" - the share of facts that appear only once in training - and argue that a model's hallucination rate on those facts will be at least as high as that rate. You simply cannot predict a birthday you never saw in training. When the gap hits, the model doesn't say "I don't know." It says something that sounds right, because saying something that sounds right is what it was trained to do.

A 2026 study in Nature by the same research group sharpens the incentive problem: accuracy-based evaluation metrics systematically reward guessing over admitting uncertainty, because a guess that happens to be right scores higher than a hedged non-answer. Models learn: commit to an answer rather than abstain.

Why Models Don't Know They're Wrong

This surprises people. If the model "knows" so much, why can't it flag its own errors?

Two reasons:

First, the model doesn't distinguish between what it has strong evidence for and what it's extrapolating. Every output is generated the same way - token by token, choosing the most probable next word. There's no internal "confidence meter" attached to each claim in the way a human might feel uncertain about something half-remembered. Research on token-level probabilities does show that confidence in individual tokens drops at the start of hallucinated spans, but this signal isn't directly surfaced to you as a user.

Second, the model is trained to produce complete, fluent responses - not to stop and say "I'm not sure about this part." Models that hedge too much get penalized in human evaluations; models that sound confident score better. The incentive gradient pushes toward confident output. The result: a hallucination usually reads exactly like a correct answer. Same tone, same confidence, same fluency.

How Bad Is the Problem Today?

Worse than the demos suggest, but improving. The Stanford HAI 2026 AI Index measured sycophancy-induced hallucination - models agreeing with users who assert false beliefs - on an adversarial benchmark across 26 frontier models. Failure rates ranged from 22% to as high as 94%, with GPT-4o dropping from 98.2% to 64.4% accuracy when users asserted incorrect information, and DeepSeek R1 falling from over 90% to 14.4%. It's a reminder that "AI" is not one thing, and stress-testing a model on your specific task is the only reliable signal.

The better news: targeted efforts work. OpenAI's GPT-5 launch announcement (August 2025) reported that with web search enabled, GPT-5's responses are approximately 45% less likely to contain a factual error than GPT-4o on the same tasks. Anthropic found that Claude 2.1 produced half the false statements of Claude 2.0 on a comparable benchmark, with subsequent models continuing to improve. One financial-services customer (Endex) using Anthropic's Citations API reduced source hallucination instances from 10% to 0% on their document-grounded tasks.

The consistent pattern: models perform better when you give them access to a verified source at inference time, rather than relying purely on training memory.

How to Spot a Hallucination

There's no perfect detector. But certain signals should raise your suspicion:

One practical test for agentic or research use: ask the model something it couldn't possibly know the answer to. A well-calibrated model should decline or express uncertainty. A hallucination-prone pipeline will make something up.

Practical Patterns That Actually Reduce Hallucinations

You can't eliminate hallucinations, but you can design pipelines where they matter less. These patterns come from Anthropic's official documentation and real deployments.

1. Give it the source and tell it to stay inside

The most direct fix: provide the relevant document and instruct the model to only use information from that document. This converts a "what do you know?" question (where training data gaps are the enemy) into a "what does this text say?" question (which is much easier).

You are analyzing the attached policy document.
Use ONLY information from the document below.
If the document doesn't address something, say so explicitly.
Do not use your general knowledge to fill gaps.

<document>
{{POLICY_TEXT}}
</document>

2. Extract quotes before drawing conclusions

For long documents (above 20,000 tokens), Anthropic's official documentation recommends having the model pull word-for-word quotes first, then reason from those quotes. This grounds the response in the actual text rather than the model's reconstruction of it.

Step 1: Extract the exact sentences from the document most relevant to [TOPIC].
Step 2: Based only on those quotes, answer [QUESTION].
If you can't find a relevant quote, say "No relevant passage found."

3. Explicitly permit "I don't know"

This sounds obvious, but it works. Models are trained to be helpful, and "I don't know" reads as unhelpful to human evaluators. You have to give explicit permission in your system prompt. Anthropic's documentation lists this among its core basic hallucination strategies:

If you're not confident about any part of your answer, say so directly.
"I don't have enough information to answer that" is a valid and preferred response.

4. Use citations mode (or ask for inline references)

Anthropic's Citations API automatically ties each claim to a specific passage in your provided documents. If you're not using the API directly, you can approximate it with a prompt:

After each factual claim, add a bracketed reference to the exact quote that supports it.
If you cannot find a supporting quote, mark the claim with [UNVERIFIED] and I will remove it.

5. Run the same prompt twice and compare

Anthropic's documentation calls this "best-of-N verification." Inconsistencies across two runs strongly suggest hallucination - a real fact is stable, a confabulated one often varies. You don't need a perfect detector; inconsistency alone is enough of a signal to trigger human review.

6. Chain-of-thought for complex reasoning

Asking the model to show its reasoning before giving a final answer surfaces faulty logic before it's buried in a polished conclusion. It also gives you something to fact-check step by step, rather than a single opaque answer.

Think through this step by step before giving your final answer.
Show your reasoning. Flag any step where you're uncertain.

7. Use retrieval-augmented generation (RAG) for knowledge-intensive tasks

RAG - fetching relevant documents from a verified knowledge base and injecting them into the prompt at query time - is the standard architecture for reducing hallucination in production systems. It replaces "what does the model remember from training?" with "what does the verified source say?" Research indicates hybrid retrieval (vector search combined with keyword search) consistently outperforms pure vector search for factual accuracy. Even with good RAG, you still need to verify that the model is actually citing the retrieved content rather than drifting to training data.

What You Shouldn't Do

A few common instincts that backfire:

Key Takeaways

Try this next: Now that you understand why models confabulate, see how prompt structure shapes model behavior end-to-end in Prompt Engineering: The Fundamentals.

LearnfundamentalsHallucinations: Why AI Lies and What to Do About It
Guidefundamentalscore9 min read

Hallucinations: Why AI Lies and What to Do About It

AI hallucinations happen when a model generates confident, fluent text that is factually wrong. Learn why it happens mechanically, how to spot it, and the practical patterns that actually reduce it.

CREATORS OF TODAY.
@creatorsoftoday
317

TL;DR: AI models don't "know" facts the way you do. They predict the most statistically plausible next word - and when their training data has gaps, they fill those gaps with confident-sounding fiction. This is called a hallucination. It's not a bug that will be patched; it's a structural feature of how these systems work. The good news: you can design around it.

What a Hallucination Actually Is

The term is borrowed from psychology, but the AI version is closer to what clinicians call confabulation - the brain (or in this case, the model) generates a plausible story to fill a gap in memory, with no awareness that the story is invented.

An AI hallucination is text that is:

  • Fluent and grammatically correct
  • Confidently stated, with no hedge or qualifier
  • Factually wrong, made up, or internally inconsistent

Classic examples: a model inventing a scientific paper that doesn't exist, citing a court case with the right format but the wrong ruling, or naming a product feature that was never shipped. The output looks right. That's what makes it dangerous.

Researchers distinguish two main types:

  • Intrinsic hallucinations - the model contradicts facts that were right there in the provided context. ("Einstein was born in Ulm" in your document; model says Berlin.)
  • Extrinsic hallucinations - the model introduces information that can't be verified against any source. The classic move: citing a real author for a paper they never wrote.

The Mechanical Reason This Happens

Every major language model is trained on a variant of the same objective: predict the next token. Given everything that came before, what word most plausibly comes next? Do that billions of times, across hundreds of billions of tokens, and you get a system that has internalized an enormous amount of the world's text - its patterns, its facts, its errors, and its contradictions.

The problem: next-token prediction optimizes for plausibility, not truth. The training data contains factual mistakes alongside correct information, and the model learns both with similar confidence. When the training data is dense on a topic, the model answers well. When the data is sparse - obscure events, recent developments, niche domains - the model fills the gap by generating the most statistically plausible continuation. That continuation might be totally wrong.

OpenAI researchers Adam Kalai and Ofir Nachum formalized this in a 2025 paper: knowledge gaps are inevitable with non-exhaustive training data. They introduce the concept of the "singleton rate" - the share of facts that appear only once in training - and argue that a model's hallucination rate on those facts will be at least as high as that rate. You simply cannot predict a birthday you never saw in training. When the gap hits, the model doesn't say "I don't know." It says something that sounds right, because saying something that sounds right is what it was trained to do.

A 2026 study in Nature by the same research group sharpens the incentive problem: accuracy-based evaluation metrics systematically reward guessing over admitting uncertainty, because a guess that happens to be right scores higher than a hedged non-answer. Models learn: commit to an answer rather than abstain.

Why Models Don't Know They're Wrong

This surprises people. If the model "knows" so much, why can't it flag its own errors?

Two reasons:

First, the model doesn't distinguish between what it has strong evidence for and what it's extrapolating. Every output is generated the same way - token by token, choosing the most probable next word. There's no internal "confidence meter" attached to each claim in the way a human might feel uncertain about something half-remembered. Research on token-level probabilities does show that confidence in individual tokens drops at the start of hallucinated spans, but this signal isn't directly surfaced to you as a user.

Second, the model is trained to produce complete, fluent responses - not to stop and say "I'm not sure about this part." Models that hedge too much get penalized in human evaluations; models that sound confident score better. The incentive gradient pushes toward confident output. The result: a hallucination usually reads exactly like a correct answer. Same tone, same confidence, same fluency.

How Bad Is the Problem Today?

Worse than the demos suggest, but improving. The Stanford HAI 2026 AI Index measured sycophancy-induced hallucination - models agreeing with users who assert false beliefs - on an adversarial benchmark across 26 frontier models. Failure rates ranged from 22% to as high as 94%, with GPT-4o dropping from 98.2% to 64.4% accuracy when users asserted incorrect information, and DeepSeek R1 falling from over 90% to 14.4%. It's a reminder that "AI" is not one thing, and stress-testing a model on your specific task is the only reliable signal.

The better news: targeted efforts work. OpenAI's GPT-5 launch announcement (August 2025) reported that with web search enabled, GPT-5's responses are approximately 45% less likely to contain a factual error than GPT-4o on the same tasks. Anthropic found that Claude 2.1 produced half the false statements of Claude 2.0 on a comparable benchmark, with subsequent models continuing to improve. One financial-services customer (Endex) using Anthropic's Citations API reduced source hallucination instances from 10% to 0% on their document-grounded tasks.

The consistent pattern: models perform better when you give them access to a verified source at inference time, rather than relying purely on training memory.

How to Spot a Hallucination

There's no perfect detector. But certain signals should raise your suspicion:

  • Hyper-specific details about obscure things. A real expert often expresses uncertainty about niche facts. A model rarely does.
  • Citations with plausible-but-unverifiable metadata. Real author, wrong year. Real journal, fake paper title. Right URL structure, page that 404s.
  • Responses that contradict the document you just provided. If the model summarizes something differently from what the source says, that's an intrinsic hallucination - and the source is right.
  • Excessive confidence on genuinely contested ground. If a real expert would say "it depends" or "the evidence is mixed," a confident model answer is a yellow flag.
  • Consistency failures across the same conversation. Ask the same question two different ways. If you get different facts, one (or both) is wrong.

One practical test for agentic or research use: ask the model something it couldn't possibly know the answer to. A well-calibrated model should decline or express uncertainty. A hallucination-prone pipeline will make something up.

Practical Patterns That Actually Reduce Hallucinations

You can't eliminate hallucinations, but you can design pipelines where they matter less. These patterns come from Anthropic's official documentation and real deployments.

1. Give it the source and tell it to stay inside

The most direct fix: provide the relevant document and instruct the model to only use information from that document. This converts a "what do you know?" question (where training data gaps are the enemy) into a "what does this text say?" question (which is much easier).

You are analyzing the attached policy document.
Use ONLY information from the document below.
If the document doesn't address something, say so explicitly.
Do not use your general knowledge to fill gaps.

<document>
{{POLICY_TEXT}}
</document>

2. Extract quotes before drawing conclusions

For long documents (above 20,000 tokens), Anthropic's official documentation recommends having the model pull word-for-word quotes first, then reason from those quotes. This grounds the response in the actual text rather than the model's reconstruction of it.

Step 1: Extract the exact sentences from the document most relevant to [TOPIC].
Step 2: Based only on those quotes, answer [QUESTION].
If you can't find a relevant quote, say "No relevant passage found."

3. Explicitly permit "I don't know"

This sounds obvious, but it works. Models are trained to be helpful, and "I don't know" reads as unhelpful to human evaluators. You have to give explicit permission in your system prompt. Anthropic's documentation lists this among its core basic hallucination strategies:

If you're not confident about any part of your answer, say so directly.
"I don't have enough information to answer that" is a valid and preferred response.

4. Use citations mode (or ask for inline references)

Anthropic's Citations API automatically ties each claim to a specific passage in your provided documents. If you're not using the API directly, you can approximate it with a prompt:

After each factual claim, add a bracketed reference to the exact quote that supports it.
If you cannot find a supporting quote, mark the claim with [UNVERIFIED] and I will remove it.

5. Run the same prompt twice and compare

Anthropic's documentation calls this "best-of-N verification." Inconsistencies across two runs strongly suggest hallucination - a real fact is stable, a confabulated one often varies. You don't need a perfect detector; inconsistency alone is enough of a signal to trigger human review.

6. Chain-of-thought for complex reasoning

Asking the model to show its reasoning before giving a final answer surfaces faulty logic before it's buried in a polished conclusion. It also gives you something to fact-check step by step, rather than a single opaque answer.

Think through this step by step before giving your final answer.
Show your reasoning. Flag any step where you're uncertain.

7. Use retrieval-augmented generation (RAG) for knowledge-intensive tasks

RAG - fetching relevant documents from a verified knowledge base and injecting them into the prompt at query time - is the standard architecture for reducing hallucination in production systems. It replaces "what does the model remember from training?" with "what does the verified source say?" Research indicates hybrid retrieval (vector search combined with keyword search) consistently outperforms pure vector search for factual accuracy. Even with good RAG, you still need to verify that the model is actually citing the retrieved content rather than drifting to training data.

What You Shouldn't Do

A few common instincts that backfire:

  • Don't trust confident tone as a signal of accuracy. A hallucination sounds exactly like a correct answer. Confidence is a product of the model's training objective, not a measure of truth.
  • Don't skip verification because the model cited a source. Models regularly fabricate citations. Always check that the source exists and that it says what the model claims.
  • Don't assume newer models can't hallucinate on your specific use case. Hallucination rates improve on average but are highly domain and task dependent. Measure on your actual task.
  • Don't use hallucination-prone outputs where the cost of error is high - legal documents, medical summaries, financial disclosures - without a human verification step on every output.

Key Takeaways

  • Hallucinations are structural, not a bug to be patched - they follow from training on next-token prediction with non-exhaustive data.
  • Models can't reliably flag their own errors because there is no internal confidence meter attached to individual claims.
  • Accuracy-based evaluation metrics actively incentivize guessing over honest uncertainty - this compounds the problem.
  • Grounding at inference time (RAG, citations, document-only constraints) is the single most reliable architectural fix.
  • Behavioral prompts - explicit permission to say "I don't know," quote-first workflows, best-of-N verification - meaningfully reduce hallucination rates in practice.
  • Always verify citations manually. A model that cites a source is not the same as a model that is correct.
  • Measure hallucination on YOUR task and domain - aggregate benchmark numbers are not a substitute for task-specific evaluation.

Try this next: Now that you understand why models confabulate, see how prompt structure shapes model behavior end-to-end in Prompt Engineering: The Fundamentals.

References & sources

Reviews

Only verified humans can leave reviews. It keeps every rating real.

Verify to review

No reviews yet. Be the first to share your take.