A chatbot can name a book that was never written, supply a precise date for an event that never happened, and present both in calm, polished prose. The error is unsettling because the answer has the shape of a researched explanation. Its tone is not evidence that the claim was checked.
Researchers often call this a hallucination or confabulation: plausible output that is false, unsupported, or internally inconsistent. The word describes the result, not a human-like experience inside the model. The useful question is what in the system lets a false statement pass as a finished answer.
A language pattern is not a fact check
During pretraining, a language model learns to predict text from vast collections of examples. This teaches it grammar, common facts, styles of explanation, and relationships between ideas. It does not attach a durable verification certificate to every sentence it can produce. The NIST generative AI risk profile describes confabulation as a natural risk of systems that approximate patterns in training data. The same mechanism that produces coherent prose can produce a coherent error.
Imagine asking for a famous scientist’s birth year. That detail may appear often enough to be reproduced correctly. Ask instead for the precise date of a little-known conference talk. If the answer is absent, ambiguous, or inconsistent in the material the model learned from, a likely-looking date can still fit the sentence. The completed wording may sound just as assured in both cases.
This is why exact names, dates, page numbers, quotations, and paper titles deserve special attention. They are easy to present with apparent precision and easy to get wrong. OpenAI’s research on hallucinations specifically discusses how hard it is to infer arbitrary, infrequent facts from text patterns alone.
Why the answer sounds certain
Confidence in the writing style is different from calibrated confidence in a claim. A model can learn that direct answers tend to be rewarded as useful. Its final wording may therefore omit the uncertainty that a careful researcher would express. A benchmark that counts only correct answers can intensify the problem: a lucky guess scores better than declining to answer. OpenAI’s analysis argues that evaluating appropriate abstention matters if systems are to learn when to say “I don’t know.”
The prompt also changes the result. “Give me the exact date” pressures a system toward a specific value even if the evidence is weak. A better request asks for the original source, distinguishes verified facts from inference, and permits an explicit unknown. That is a useful precaution, but it does not make the response self-verifying. A fabricated citation may accompany a fabricated date.
The different ways an answer goes wrong
Some errors come from missing information. Others come from blending two real things: a genuine person and a different person’s work, or an actual law and a rule from another jurisdiction. A stale answer may once have been accurate. An ambiguous question may have several defensible interpretations, yet the model quietly picks one. These cases need different corrections, which is why simply telling a model to “be accurate” has limited value.
Adding web search or document retrieval can help with up-to-date and specialized questions. The model can then use text that is outside its training data. But retrieval introduces its own failure points. A search result may be irrelevant, a page may be outdated, or a snippet may omit an important condition. The model may cite a document that discusses the topic without supporting the particular sentence. A tool can also fail while the answer continues as if the lookup succeeded.
The distinction matters most when an answer affects a decision. A false summary of a film may waste a minute. An invented legal exception, medical dosage, financial rule, or security setting can do real harm. NIST flags the risk of users acting on confidently presented false content, including false logic and citations that seem to justify it.
A practical verification pass
Start with the claims that would change what you do. Do not spend equal effort checking every sentence. For a consequential answer, isolate the exact proposition: “This account permits a withdrawal under condition X,” rather than “the advice sounds sensible.” Open the cited original page and search for that proposition. Check the date, jurisdiction, and qualification language. A source about an account is not automatically a source for every rule attributed to it.
For a quotation, find the words in the original document. For a paper, confirm that the title, authors, and finding match. For a product feature, use current product documentation. If the cited page does not support the claim, treat the claim as unverified even if several summaries repeat it. If there is no source, ask for one, then inspect it rather than trusting the new citation on sight.
It also helps to separate three statements: what the source says, what the model infers, and what remains unknown. An answer can be useful even when it cannot resolve the last category. In high-stakes work, a human with relevant expertise must still judge the evidence and the consequences.
The right mental model
A polished answer is a set of claims, each with its own evidence requirements. Verification should follow those claims back to the relevant passage, rather than stopping at a citation or confident tone. Retrieval-augmented generation can supply documents, but the document still has to be current, relevant, and used faithfully. The decisive question remains: Where, exactly, does the evidence support this sentence?
