An AI agent does useful work when its actions produce the result you actually needed. A confident completion message is only a report about that work. To decide whether the report is justified, you need a way to inspect the result outside the agent’s own explanation.
That is easier when the assignment has a clear scope, a defined output and an independent check. Rather than judging an entire category of software from a demonstration, write one task small enough that you can tell whether it succeeded.
An agent chooses actions inside a loop
Anthropic’s engineering guide distinguishes fixed workflows from agents that dynamically direct their processes and tool use. It emphasizes feedback from the environment, human checkpoints and stopping conditions. It also notes costs and errors that can accumulate over multiple actions.
The guide was first published in December 2024 and now flags changes in the tooling landscape. Here it supports architectural ideas, rather than a claim about the capabilities or prices of a particular current product. You do not need to use Anthropic’s software to apply the distinction.
A fictional document task makes the loop visible. An agent reads a request, opens a folder, selects a file, extracts information and writes a summary. Each action gives it another observation. If it opens the wrong file near the start, a fluent summary can faithfully describe material that was never relevant to the assignment.
More steps do not automatically provide more assurance. Assurance comes from checks that compare the task and its result. The right question is what would expose a plausible wrong answer.
Write the finish condition before starting
Suppose you want a briefing from five specified public documents. “Research the topic” leaves open which documents count, what period matters and whether the result should reconcile conflicting claims.
A more inspectable fictional assignment might be: “Use these five documents to list each author’s position on the stated question. Link each position to its source, identify disagreements and mark anything the documents do not resolve.” Define the output as a short table plus an explanation of the disagreements.
Then define what you will inspect:
- Coverage: all five specified documents are represented, or an inaccessible document is named as a gap.
- Traceability: each author’s position links to the passage that supports it.
- Disagreement: conflicting positions remain distinct; the briefing does not blend them into a conclusion no author supports.
These checks make the finish condition visible in the briefing itself.
These conditions are original examples of task design, not a benchmark proven to eliminate errors. Their purpose is to make failure describable. You can identify a missing document or an unsupported statement more concretely than deciding whether a report “looks comprehensive.”
Give the task an explicit boundary
State what material and actions belong to the assignment. For the briefing, that might be the five supplied public files and a draft saved for review. If the task does not require sending messages or changing a system of record, those actions need not be included in its permissions.
Also record assumptions that could affect the answer: the date range, intended audience and meaning of the question. A briefing for a beginner may need terms explained; a briefing comparing technical proposals may need the assumptions exposed instead. Missing context can produce an output that is polished but aimed at the wrong reader.
A useful boundary includes a stop condition. For example, if one of the five documents is unavailable, the agent should produce a clearly partial result or ask for the missing material according to the agreed process. Continuing indefinitely or quietly substituting another document changes the assignment.
This boundary is especially useful for repeated tasks. A saved instruction should describe the stable purpose, while each run should still use current inputs and record unresolved issues. Old context is evidence about an earlier state, not permission to assume that state persists.
Inspect the product of the actions
For the fictional briefing, open at least the passages behind the central conclusions. Check that a cited document actually supports the claim and that the text preserves qualifications. A link to a real page establishes its existence; it does not prove that the sentence beside it is accurate.
For a spreadsheet task, inspect the cells that should have changed and check a worked result. For a code change, examine the change and whether relevant tests exercise the reported behavior. For a draft message, compare the recipient, factual details and requested action before sending it.
The check should match the failure you care about. A spelling review will not discover that a summary used the wrong reporting period. A successful file save will not establish that a calculation is correct. A test that merely repeats the same mistaken assumption may offer little assurance.
For a task that can change records or send information, add the separate review of each tool’s permissions. A correct briefing and an authorized action are different things to establish.
Benchmarks demonstrate a problem under specified conditions
The original WebArena paper introduced reproducible web tasks evaluated for functional correctness. Its best GPT-4-based baseline achieved 14.41% end-to-end success, compared with 78.24% for human performance in the reported experiments.
Those are historical benchmark results from the paper first released in 2023, not the success rate of today’s agents or your own workflow. Their value here is methodological: completion was assessed against a task outcome, rather than how convincing an agent’s final answer sounded. The gap also prevents treating one impressive demonstration as proof of general reliability.
When evaluating your own task, record the inputs, expected result, observed result and review time. Include failures. If the agent’s draft saves twenty minutes but checking it takes twenty-five, that particular run did not establish a time saving. If it produces an unusually useful analysis, record what made the task suitable rather than assuming the same result everywhere.
Make the handover answerable
Ask the final report to identify what was produced, where you can inspect it, what checks ran and what remains unresolved. “Finished” is hard to assess. “Five documents covered, two disagreements preserved, one inaccessible appendix noted” gives you something to verify.
A bounded task can still demand substantial reasoning. The boundary does not make the work trivial; it makes the result reviewable. Start with one assignment whose completion you can recognize, then expand responsibility only when the evidence from actual runs justifies it.
