Hallucination

What it means

The model generates plausible-sounding content that does not match facts, retrieved sources, or user-provided data.

Why designers should care

Without citations, edit steps, and low-confidence states, users act on wrong answers in high-stakes workflows (finance, health, legal, shipping code).

Example

A research assistant labels unsourced claims as “Unverified,” requires citations for factual bullets, and offers “Compare to source” hover on each sentence tied to RAG snippets.

Common mistakes

• • Presenting every reply with equal visual weight regardless of evidence.
• • Blaming users for not checking when the UI implies authoritative truth.
• • No recovery path (“fix this claim”) after a hallucination is spotted.

Related terms

• Retrieval-Augmented Generation (RAG): RAG = search step + generation step: find chunks likely to contain the answer, inject them into the prompt, then produce a reply grounded in those sources.
• Guardrails: Guardrails include system prompts, classifiers, allow/deny lists, output validators, rate limits, and human review gates applied before or after generation.
• Human-in-the-Loop: The product pauses at defined checkpoints for human judgment (approve send, merge PR, publish article, charge card), even if AI drafted the action.
• Moderation: Automated classifiers and human reviewers flag content categories (hate, violence, sexual content, PII) and enforce actions (block, blur, warn, escalate).
• Prompt Injection: Malicious or accidental instructions embedded in retrieved or pasted content override intended rules (“ignore previous instructions and…”).

Explore more

• Agentic UX framework