Start with a complete attempt

An agent can use many tools and still fail to finish the work. A long trace demonstrates activity, not success. Define an attempt as a bounded task with a starting state, a permitted set of actions and an observable end condition.

If the task is to prepare an approved document, measure whether the document meets its requirements and reaches the correct destination. Do not substitute the number of tool calls or the length of the final explanation for that result.

Keep different outcomes visible

Record completed tasks, partially completed tasks and failures. Note whether a person intervened. Separate a wrong answer from a provider outage, a permission failure or a tool that stopped responding. Those outcomes may require different fixes even when they all interrupt the user.

Security and access boundaries deserve their own checks. A useful result obtained from data the user should not see is not a successful attempt. Presentation quality cannot cancel that failure.

Count the whole cost

Include retries and unsuccessful attempts when comparing cost. If one system succeeds once after several failures, reporting only the successful call makes it look cheaper than the experience actually was.

Keep measured charges separate from estimates. Record model usage, paid tools and any human review time you can observe. When the provider does not expose a cost, report it as unknown rather than zero. A subscription changes the billing arrangement; it does not remove the work performed.

Test ordinary difficulty

Use examples that reflect the real workflow: missing fields, conflicting evidence, an unavailable dependency and a user who lacks permission. Keep a separate set of examples for evaluation so changes are not judged only on cases used during development.

Repeat important cases. One impressive success or one unusual failure is a starting point for investigation, not a dependable rate. Report the number of attempts alongside any percentage, and preserve the context that could explain a change.

Make release decisions explicit

Decide which failures block release, which can be handled by review, and when a user should fall back to the existing process. The application should make those boundaries visible.

A useful evaluation is a decision tool. It should tell the team whether the system is ready for the next scope of work, what it costs to operate, and which uncertainty matters most. It does not need a single score that hides those tradeoffs.