A model can write well and still miss the document that matters
Ask a general chatbot for your employer’s current return policy and it may produce a plausible answer. Plausible is not enough if a policy changed last week. Retrieval-augmented generation, usually shortened to RAG, adds a lookup step: find the relevant material first, then give the model that material while it writes its answer.
The distinction is practical. A model’s training gave it broad patterns and language ability. A document search gives it the specific policy, manual, or research passage needed for this question. RAG combines the two at answer time. It does not normally rewrite the model’s weights whenever a file changes.
The document has to be prepared before anyone asks
Suppose a company has hundreds of support pages. Searching an entire library with every question would be slow and noisy. A RAG system first ingests those pages, splits long text into manageable pieces, stores their location and access information, and builds an index. The pieces are commonly called chunks.
Chunk size is an editorial as well as a technical choice. A sentence by itself may omit the condition that limits a refund. A whole chapter may bury the line the user needs. Good chunks keep the relevant qualifier near the claim. Titles, section names, publication dates, and permission labels can help the system avoid a superficially similar but wrong passage.
Many systems create embeddings for chunks. An embedding represents text as numbers learned so related ideas can be searched even when they use different words. A question about a “laptop battery draining quickly” might find a note about “notebook power loss.” This is useful, but it is not a truth detector. Two passages can sound similar while applying to different products, countries, or dates.
Retrieval is the first place an answer can fail
When the question arrives, the system searches its index and selects candidate chunks. Some designs use keyword search, some use embeddings, and many combine both. A reranker may compare the candidates more carefully before the top passages are sent to the model. If the correct source never makes that shortlist, a beautifully worded final response can still be wrong.
Imagine two policies: one for damaged goods and another for ordinary returns. They share words such as “item,” “return,” and “refund.” A search that retrieves the ordinary policy for a damaged-item question has already set the model on the wrong path. Adding a citation to the wrong passage would not repair that mistake.
The original RAG research combined a language model’s learned knowledge with an external index of passages. Later products use many variations, but the central idea remains: let the model condition its answer on retrieved evidence. The system should preserve enough provenance that a reader can inspect the passage, its date, and its owner.
Generation can add a second failure
After retrieval, the model receives the question and selected passages. It may summarize, compare, or explain them. It can also overstate what they say, merge two policies, ignore a footnote, or fill a gap with a likely-sounding detail. “Grounded” means tied to available sources, not guaranteed accurate.
A good answer should distinguish three cases: the source directly supports a statement, the statement is a reasonable inference, or the source does not answer the question. If the policy is silent about international returns, the assistant should say so. A system that always produces a complete paragraph may be less useful than one that admits a missing document.
Access control matters too. A search index can contain sensitive records. Retrieval must enforce the user’s permissions before a passage reaches the model; asking the model to keep an unauthorized passage secret after it has received it is a fragile boundary. And a retrieved web page is data, not a trusted instruction to the assistant.
A concrete retrieval mistake
Suppose a company handbook changed its parental-leave policy in June. An employee asks how many weeks are available. The search layer returns a highly relevant handbook page from March and a less obviously worded June update. If the system ranks the old page first and sends only that passage to the model, the model may answer neatly and cite the obsolete document. The citation proves where the sentence came from; it does not prove the underlying source was current.
A better workflow records document dates and authority, removes or flags superseded material, and shows the retrieved passages to the user where the decision matters. Search quality should be tested with questions that have known answers, including changed policies, similar product names, and questions the documents do not answer. Measure whether the correct passage was retrieved separately from whether the model used it faithfully. Microsoft’s RAG design guide likewise separates retrieval evaluation from evaluation of the generated response. Otherwise, an apparently good final answer can hide the wrong evidence path. RAG is particularly useful when information changes often, but only if the collection and its update process are maintained.
When RAG is the right tool
RAG is particularly useful when answers depend on an identifiable body of changing or specialized material: product documentation, internal procedures, legislation, or a research collection. It can make answers easier to audit and update, because a corrected document can enter the index without retraining the model.
Missing or stale documents call for better source management. When sources conflict, an answer can still help by exposing the disagreement instead of silently choosing a winner. For a wrong answer, trace the two stages separately: did search supply the correct passage, and did the model stay within it? That distinction also helps explain why a fluent AI answer can still be false. A bigger index alone does not resolve either failure.
