A chatbot answer passes through several different steps: representing the input, predicting a continuation, selecting output tokens, and sometimes consulting outside information. A failure in any one of those steps can leave you with a readable answer that does not solve your problem.
Consider a simple request: “Using this delivery policy, tell me whether my order qualifies for a refund.” The useful answer depends on a document, an order date, a delivery date and the policy’s exceptions. Elegant wording adds little if one of those inputs is missing. Following the answer through the system makes it easier to see which part needs checking.
Text becomes something the model can calculate with
The model works with tokens rather than whole sentences as indivisible objects. A token can represent a word, a word fragment or punctuation; the exact split depends on the tokenizer. Those token identifiers are converted into learned numerical representations.
The original Transformer paper describes embeddings, attention and a final conversion into next-token probabilities. Attention combines information from positions in a sequence using calculated weights. It is a way of incorporating context, not a separate fact-checking service. The paper studied translation; it illustrates core machinery rather than documenting every present-day chatbot.
In our refund example, the policy’s “within 30 days” and the customer’s dates need to remain available in the input. The model may connect relevant parts of the text, but the existence of attention does not prove that it applied the correct exception. You still need an answer that identifies the clause and shows the date comparison.
Prediction builds the answer a piece at a time
An autoregressive language model estimates a distribution over possible next tokens given the preceding context. A decoding procedure chooses a token, and generation continues with that token added to the sequence. The completed paragraph is assembled through those successive choices.
Holtzman and colleagues’ study of decoding explains this process and compares ways of selecting continuations. Its experiments concern open-ended text generation, including repetition and diversity. They are not a test of whether a refund decision is correct.
This distinction matters when an answer looks suspiciously tidy. The sentence “You are eligible under section 4.2” has the shape of a useful response. The shape alone says nothing about whether section 4.2 exists. Check the actual policy before treating the number as a citation.
Likewise, asking for a more confident tone changes the presentation you requested. It supplies no new evidence about eligibility. Ask for the clause, dates and calculation instead of asking the system to sound certain.
Assistant behavior is trained separately
Predicting plausible text and following a person’s instruction are different objectives. The InstructGPT research describes one historical approach: train on human demonstrations, collect comparisons between outputs, learn a reward model, and further tune the language model using that reward.
That work showed that training could improve instruction following, while also reporting simple mistakes and invented information. It does not establish the training recipe or reliability of whichever product you are using today.
For the refund question, a helpful style might include a polite explanation and a clear conclusion. Both are useful once the underlying decision is supported. Neither compensates for a missing delivery date. A better response in that situation would state which date is needed and leave the decision unresolved.
You can encourage that structure: “If an input is absent, identify it rather than filling it in.” Treat the instruction as a request for a more inspectable answer, not a guarantee that the system will comply.
Retrieved evidence changes the available material
A language model’s learned parameters are not the same as an editable record of your current delivery policy. Retrieval adds another route: find relevant material and make it available to the generator.
The original retrieval-augmented generation paper combined a document retriever with a text generator and evaluated knowledge-intensive tasks. It also explored replacing the document index to update available knowledge. Its results concern a particular research system and datasets, not a promise that any chatbot with search will answer correctly.
For our example, ask which policy version was retrieved. A perfectly accurate quotation from last year’s policy could still answer the wrong question. Check its effective date, the covered product and any regional limits. Then compare the quotation with the conclusion: does the cited paragraph actually say what the answer claims?
A citation that opens successfully is only the first check. The more useful check is whether the cited words support the relevant step of the decision.
Make calculations and missing inputs visible
Suppose, purely for illustration, a policy allows a claim within 30 days after delivery. An order arrived on September 10 and a claim was submitted on September 28. The elapsed difference is 18 days. That arithmetic answers the timing question under the assumed rule; it does not establish whether the policy has another exclusion or whether those dates are documented.
The answer should expose the following chain:
| Step | What the illustration supplies | What still needs checking |
|---|---|---|
| Rule | A 30-day window after delivery | The applicable policy version and exclusions |
| Inputs | Delivery September 10; claim September 28 | Whether the records support both dates |
| Operation | September 28 minus September 10 = 18 days | Whether delivery is the correct starting event |
| Conclusion | The timing falls within the assumed window | Whether all other eligibility conditions are met |
“Use the quoted policy, use these dates, and show the elapsed-day calculation” gives you this chain to audit. If a calculator or code tool is used, inspect its inputs as well as its output. Correct arithmetic on the order date would still be wrong if the rule starts at delivery.
For a longer conversation, restate the small set of facts the decision depends on. Do not assume that an earlier upload or remembered detail is available in the current answer. Ask the system to list the inputs it used; compare that list with the original records.
A practical way to inspect the result
Read the final answer in two passes. First, check the substance: the real policy version, the actual clause, the supplied dates and the calculation. Second, check the wording for additions that the evidence does not support, such as a promised payment date or a claim that approval is automatic.
A useful final prompt is: “Separate what the document states, what you calculated, and what still requires confirmation.” That produces a review structure you can compare with the material in front of you.
For this order, a defensible answer might establish only that the claim was made 18 days after delivery. If an exclusion remains unchecked, eligibility remains open. Keeping that boundary visible is more useful than a confident “yes” that quietly skips it.
