An irrelevant detail can derail an AI answer before the arithmetic even begins. The system may choose an operation suggested by the wording instead of representing what the question actually asks. A calculator can execute that mistaken operation perfectly. The failure lies in deciding what to calculate.
Consider this invented example: Maya has twelve marbles and receives eight more. Three of the new marbles are unusually small. How many marbles does she now have? The answer is twenty. Size did not remove any marbles. An answer of seventeen would introduce a subtraction the story never requested.
That small example separates three things often bundled together: understanding the situation, carrying out a procedure, and explaining the result. Reliability requires checking which of those steps succeeded.
What the irrelevant-detail experiments showed
Apple researchers’ 2024 GSM-Symbolic paper tested variations of school-level mathematics questions. Changing numerical values affected performance, and adding apparently relevant but mathematically unnecessary clauses caused substantial declines among the models tested.
In the GSM-NoOp condition, details could suggest a familiar operation without justifying it. A model might treat smaller fruit as something to subtract from a count. Its numerical steps could then look tidy while answering a different problem.
These findings concern specified models, prompts and evaluations from that study. They do not establish that every later model will fail a marble question. Their lasting value is methodological: success on the original wording is weaker evidence than success across variations that preserve the underlying problem.
A practical check follows from that result. Before inspecting a calculation, ask which sentence authorizes each operation. In our marble example, receiving eight justifies addition. The statement about size justifies no change to the count. This is a reasoning check a reader can perform without needing access to the model’s internals.
A counterexample can check a conclusion
Here is a second invented problem. Every red box is locked. Some locked boxes are heavy. Must some red boxes be heavy?
No. Two imagined boxes are enough to check the claim:
| Box | Locked? | Heavy? |
|---|---|---|
| Red | Yes | No |
| Blue | Yes | Yes |
Every red box is locked, and some locked boxes are heavy. Both premises hold, yet no red box is heavy. One consistent counterexample defeats the claim that the conclusion must follow.
Notice how little vocabulary the question requires. The work is in preserving the direction and extent of the relationships. “Every red box is locked” puts red boxes inside the locked category. It does not put every locked box inside the red category. “Some” also leaves the rest unspecified.
Now rename the boxes as folders, reorder the two premises, or add a sentence about shelf height. The logical answer should remain unchanged. If the answer changes, there is a concrete variation to investigate. One screenshot cannot reveal a general failure rate, but a controlled family of examples can show where a procedure loses stability.
Learned relationships can still support real intermediate work
It would also be too simple to conclude that a model only repeats complete answers it has seen. Anthropic’s 2025 tracing research found intermediate representations in Claude 3.5 Haiku linking Dallas to Texas and Texas to Austin when answering a question about the relevant state capital. Interventions on those representations changed the answer.
That supports a multi-step process in the investigated case. The same research also found cases where an explanation did not faithfully describe the internal calculation the researchers identified. A plausible written account is therefore useful material to inspect, but cannot serve as an automatic record of the computation that produced it.
Those findings can coexist. A model may combine information productively in one task and give an unreliable account in another. The useful question is which process is supported by evidence in the case at hand, rather than whether one grand label can classify all of its behavior.
Knowing a procedure differs from executing it
Apple’s 2025 study The Illusion of Thinking used controlled puzzles, including Tower of Hanoi, to vary complexity. Reasoning models performed better at intermediate difficulty, but the tested systems eventually reached difficulty ranges where success collapsed. Providing the Hanoi algorithm did not remove the execution failures.
The experiment distinguishes finding a strategy from maintaining a correct sequence of states. A model can describe how disks should move while losing track of which disk occupies a peg. The paper’s provocative title does not turn those observations into a proof that no language model can reason on any task.
A research response by Khan and colleagues argued that the text-only interface and system constraints contributed to the apparent cliff. They reported improvements when models could use tools to carry out procedures. That is evidence for studying the whole setup, including execution facilities, rather than attributing every failure to one inherent cognitive limit.
The two practical lessons fit together: specify the task conditions and inspect how state is maintained. An external workspace can help execute a strategy without guaranteeing that the strategy addresses the right problem.
Independent checking needs the right question
Imagine asking an AI to schedule a meeting. It translates availability into constraints, and a separate program checks candidate times. The program can reject a slot that overlaps a lunch break—provided the lunch break was included.
If the AI omitted it, the checker may certify a schedule that satisfies an incomplete description. In the marble example, a calculator has the same limitation: it can verify that twelve plus eight minus three equals seventeen, while leaving the unjustified subtraction untouched.
This suggests a useful review sequence:
- Write down the requested result and every relevant constraint.
- Check that the proposed representation preserves them.
- Run the calculation, procedure or external checker.
- Compare the checked result with the original request.
These are practical review steps, not a guarantee against all errors. Their purpose is to expose where an unsupported choice entered. Our article on machine-checked software proofs follows the same issue at a larger scale: the property being checked and the assumptions defining it determine what a successful check establishes.
For the next confident AI answer, try a small test with a clear expected result. Remove a decorative detail. Rename the objects. Ask whether a counterexample is possible. Then use an independent method to check the part that can be checked. The explanation becomes more useful when its conclusions survive changes that should not matter.
