An AI leaderboard can tell you which system performed well on a particular test. Choosing a system for your own work takes another step: checking whether that test resembles the job you need done.
Consider an assistant that turns customer emails into draft support tickets. It must extract the order number, identify the request, preserve uncertainty and leave missing fields empty. A model that excels at difficult mathematics might still put the wrong number in the ticket. You would reject that result even if the prose were excellent.
The useful question is concrete: what counts as a successful ticket, and what evidence would show that the assistant can produce one consistently?
Read the test before reading the rank
A score has a task, a dataset and a grading rule behind it. These details determine what the number can mean.
The original SWE-bench paper, for example, defines a software task using a repository and an issue description. A proposed patch is applied and tested; success means resolving the issue under that test procedure. This is a different demand from writing a short standalone function. It also does not certify every aspect of maintaining a production service.
Before comparing two scores, write down what each system received and what it could do. Were relevant files supplied? Could it search, run code or attempt another answer? Was the result graded by tests, a reference answer or a person? A tool-enabled workflow and a single text response answer different evaluation questions.
For the support assistant, a suitable test would contain emails and a ticket specification. A general writing preference vote would leave the order-number problem unresolved. It could be useful for judging tone, but tone needs its own place in the decision.
Keep accuracy separate from useful behavior
The HELM research framework makes a case for evaluating multiple dimensions in the same scenarios, including accuracy, robustness, calibration and efficiency. It also documents how prompt formatting and adaptation choices enter an evaluation. The paper is a historical framework, not a current ranking of products.
Apply that distinction to your task by keeping a small scorecard:
| Check | Example failure | How to judge it |
|---|---|---|
| Field accuracy | Wrong order number | Compare with the email |
| Missing information | Invented delivery date | Require an empty field and a question |
| Policy compliance | Refund promised without authority | Compare with the supplied policy |
| Format | Invalid ticket structure | Check the agreed field format |
| Time and cost | Draft needs excessive retries | Count the complete attempt |
These are proposed checks for this example, not a universal standard. A document-summary assistant would need different rows. A medical or financial workflow would require more demanding validation and human oversight than this small exercise provides.
Do not let a good average silently compensate for a failure you cannot accept. If correct identifiers are essential, decide that before seeing the results. Otherwise it is easy to invent a generous weighting after your favorite model performs badly.
Ask when the questions became available
A public test can become part of the material a model encounters during training. Familiarity with the questions complicates the interpretation of a high score; it does not automatically prove deliberate cheating.
LiveCodeBench addresses this problem by collecting programming-contest problems over time and comparing performance across release periods. Its paper reports evidence consistent with contamination in some older problems and evaluates several code-related tasks. Those results concern its tested systems and periods; they are not an accusation against every model with a strong score.
For a claim you are assessing, look for a stated relationship between the test dates and the training cutoff. Fresh questions can help, but freshness alone does not establish that the questions cover your work or that the grading is sound.
For your own support test, reserve examples that have not been used to tune the prompt. If you repeatedly rewrite instructions around every failed test email, that collection becomes a development exercise. Keep a separate set for checking the final workflow. Avoid placing private customer records in a public benchmark or in a service that is not approved to receive them.
Build a comparison you can inspect
Start by listing a few distinct cases: an ordinary refund request, an email with two order numbers, a forwarded chain with conflicting dates, and a request that the policy does not answer. These are illustrative cases. The right mix should come from the actual work and its errors.
For each case, write the expected ticket before running the assistant. Identify which statements come directly from the email, which require a policy lookup, and which must remain uncertain. An answer key that says only “looks reasonable” will not catch the invented delivery date.
Run each candidate with the same available information and the workflow you expect to use. If one gets retries or retrieval, give the comparison a clear label and account for those resources. You can compare complete systems, but describing the result as a pure model comparison would hide part of the explanation.
Read failures individually. A blank field and a fabricated field may both lose a point, yet they demand different responses. The first may need a clarification step. The second may prevent you from using the workflow until it changes.
Decide what the result permits
Suppose two assistants each pass eighteen of twenty fictional cases:
| Assistant | Failed cases | What the headline score hides |
|---|---|---|
| A | Two visible formatting errors | The ticket needs a format correction |
| B | Two silently substituted order numbers | The ticket may be attached to the wrong order |
Both score 90 percent. The second failure can survive a quick visual review because the field is filled and looks plausible. Keep failure types beside the score when deciding which workflow to test further.
Twenty examples would also be far too few to establish a rare-error rate. A successful small trial can justify further testing; it cannot establish that the system will remain reliable across new customers, policies and input formats.
Keep the test cases, settings, outputs and reasons for each judgment. The task-verification exercise shows how to make a finish condition inspectable before a run. When the model or policy changes, those records let you compare behavior rather than rely on a remembered impression.
A leaderboard is useful evidence for narrowing a search. The decision becomes stronger when you can point to a real input, an agreed success condition and a checked result. For the ticket assistant, that means knowing which field is right, which is uncertain and who must review the draft before it becomes an action.
