How to Know Whether an AI Agent Actually Works

The agent says the task is complete.

That statement is not evidence.

An agent can produce a coherent plan, call the expected tools, and end with a polished summary while the underlying job remains partly undone. It can also reach the correct outcome through a path that is too expensive, fragile, or risky to operate.

Evaluation has to cover the completed job, the decisions inside it, and the conditions under which it ran.

Define success outside the model

Begin with an observable state.

Weak: “The agent handled the request well.”

Stronger: “The required fields were collected, the account was updated with the approved values, the confirmation was sent, and the case contains the identifiers proving each action.”

The completion condition should be checkable by code or an experienced reviewer without asking the agent whether it succeeded.

For research work, success may include an evidence set, coverage requirements, correct calculations, explicit uncertainty, and a deliverable that answers the decision question. For software work, it may include tests, build output, behavior in the application, and a reviewable change record.

Build cases from real work

A useful evaluation set contains:

common cases,
high-value cases,
rare but consequential cases,
missing or malformed inputs,
conflicting instructions,
permission boundaries,
unavailable tools,
stale or contradictory sources,
and cases where the correct answer is to stop.

Do not rely only on historical successful examples. They underrepresent the conditions that make operation difficult.

Keep a held-out set for release decisions. Add production failures after they are understood and sanitized. Version the cases alongside the system.

Measure three levels

1. Outcome

Did the job finish correctly?

This is the primary measure. It may be binary, rubric-based, or tied to an underlying business result. When possible, verify directly against systems of record.

2. Trajectory

Did the agent make sound intermediate decisions?

Inspect tool selection, arguments, retrieval, planning, retries, approvals, and stop conditions. A lucky outcome through an unsafe path is not reliable performance.

3. Operation

Could the organization afford and support this behavior?

Measure latency, model and tool cost, number of steps, retry rate, human review time, failure recovery, and queue behavior. A 95% success rate may still be unusable if the remaining 5% consumes all operator attention.

| Measure | Example | | --- | --- | | Task success | Verified final state matches requirement | | Tool correctness | Correct tool and valid arguments | | Grounding | Material claims supported by allowed evidence | | Boundary behavior | Stops or requests approval when required | | Recovery | Resumes safely after transient failure | | Cost | Total task cost within budget | | Latency | User or queue completion time acceptable | | Human effort | Review and correction time reduced |

Separate component failure

When the task fails, identify the layer:

The required source was absent.
Retrieval missed the source.
The model misinterpreted correct context.
The wrong tool was selected.
The arguments were invalid.
The tool failed.
Verification was missing.
The interface caused the user to supply the wrong information.

This matters because each failure has a different repair. A larger model does not fix a deleted document or a tool that returns ambiguous success.

Use model graders carefully

Models can help evaluate open-ended output at scale, especially for criteria like completeness, tone, or evidence use. But the grader is another model-dependent system.

Calibrate it against human judgments. Give it a narrow rubric. Blind it to irrelevant information. Track disagreement. Do not use a model grader as the only authority for high-consequence behavior.

Deterministic assertions should cover structure, permissions, expected records, tool results, calculations, identifiers, and other facts the application can check directly.

Evaluate change, not only launch

Every model, prompt, retrieval, tool, or workflow change should run against the same core cases. Compare the candidate with the current system. Require explicit acceptance for tradeoffs: perhaps quality improves while latency rises, or tool success increases while cost doubles.

Use online measures after release, but do not experiment casually with consequential behavior. Start with internal traffic, shadow runs, or decision support. Expand authority as evidence accumulates.

METR’s time-horizon research is a useful reminder that agent success varies with task length and reliability threshold. A model’s ability to sometimes complete long work does not establish that it can reliably run your workflow. Your cases remain the relevant instrument.

Observe production outcomes

Tracing is necessary but insufficient. A trace can show every step succeeded according to the software while the customer’s objective failed.

Connect runs to downstream state and user correction. Review:

cases reopened after apparent completion,
actions reversed by operators,
repeated failures around one source or tool,
cost and latency outliers,
escalation volume,
and changes in the type of work users attempt.

The evaluated boundary should evolve with actual use. If users begin relying on the agent for a new decision, the system has changed even if the code has not.

The release question

Do not ask, “Is the agent intelligent enough?”

Ask:

Which jobs can it complete at the required reliability?
Which cases require review or refusal?
Can every consequential action be verified?
Can operators understand and recover failure?
Are cost and latency acceptable at expected volume?
Does the evaluation represent what users are actually doing?

An agent works when the organization has evidence that a bounded job completes correctly, observably, and economically, with a controlled response when it does not.

Everything else is a demonstration.

Evaluate the handoff, not just the agent

Many evaluations stop at the moment the agent asks for help. In production, that is only the middle of the workflow. The quality of the handoff determines whether human review is a real control or a ritual.

An evaluation case should inspect whether the system explains the intended task, preserves completed work, identifies the uncertainty, presents the relevant evidence, and asks for a decision a person can reasonably make. It should also test what happens after the response. Does the workflow resume from the correct state? Can a rejection be honored without repeated prompting? Is the reviewer’s correction captured as evidence for future tests?

Measure review burden directly. Useful signals include the share of tasks requiring review, time to understand a review request, correction rate, approval reversals, and the number of times a person has to leave the review surface to reconstruct context. A system that routes every ambiguous detail to a person may be safe in a narrow sense while still failing to reduce work.

Build an error budget around consequences

Not every error deserves the same threshold. A missing internal tag, an incorrect customer-facing statement, and an unauthorized financial action should not be averaged into one accuracy score.

Group evaluation cases by consequence and reversibility. Low-consequence formatting or categorization may tolerate occasional correction. Material claims may require evidence and review. Irreversible actions may require deterministic validation, explicit approval, or complete exclusion from the agent’s authority.

This creates an operating error budget. The team can decide where imperfection is acceptable, where fallback is required, and where one failure should block a release. It can also avoid the opposite mistake: demanding near-perfect model behavior for harmless work and making the system too expensive or slow to use.

| Failure class | Example | Reasonable response | |---|---|---| | Cosmetic | Inconsistent formatting | Monitor and repair in batches | | Recoverable | Wrong internal category | Validate, allow correction, measure rate | | Material | Unsupported external claim | Require evidence and review | | Consequential | Unauthorized system change | Block execution and investigate |

Run a shadow period

Before an agent receives meaningful authority, let it operate in shadow mode. Give it the same triggers and context as the real workflow, record the actions it would take, and compare them with what experienced people actually did.

Shadowing reveals cases the initial test set missed. It also shows whether the evaluation criteria reflect the organization’s real judgment. Disagreements should not automatically be counted as model failures. Sometimes experts disagree with one another, policy is implicit, or the workflow contains a choice no one has named. Those findings are part of the system design.

After shadowing, move through staged authority: proposals, reversible actions, bounded completion, and wider use. Continue sampling successful tasks, not just failures. A system can drift into an easier subset of work and appear reliable because difficult cases are quietly abandoned or escalated.

The final standard is practical: can the organization explain what the agent is responsible for, show how that responsibility is measured, detect when performance changes, and recover when the system gets it wrong? If those answers are clear, the agent may be ready to do real work. If they are not, another demo will not fix the problem.