An agent says it has followed up with a new lead. Before you call that a success, check the recipient, the message, and the customer record. Was the reply sent to the right person? Did it promise something your team cannot deliver? Was a second follow-up already waiting to go out?
AI agent evaluation is the repeatable testing of whether an agent completes the right task, within your rules, with evidence you can check. It covers the result, the actions taken to reach it, and the resources the work consumes. A convincing final message is only one piece of that evidence.
For a service business, the useful question is not whether the agent sounds capable. It is whether you can give it a defined responsibility without quietly creating more correction work for your team.
What does AI agent evaluation actually measure?
It measures whether the work is correct, whether the agent stayed within its authority, and whether the result is worth the effort and cost. Those are separate questions; keep them separate on your scorecard.
Anthropic's guide to agent evaluations distinguishes a task, a trial, a grader, a transcript, and an outcome. In plain language:
- Task: the job you give the agent, with clear success criteria.
- Trial: one attempt at that job.
- Grader: a check that judges part of the attempt.
- Transcript, or trace: the record of what the agent did and what its tools returned.
- Outcome: what actually exists or changed when the attempt ends.
That last distinction matters. Anthropic uses a booking example: saying a flight is booked is different from a reservation existing in the database. Apply the same standard to your own work. A lead record, saved draft, or confirmed appointment is better evidence than an agent's summary of its intentions.
Our recommendation is to make the outcome check the center of the evaluation, then inspect the action record for boundary violations and explanations of failure.
How is evaluating an agent different from checking an AI answer?
An answer check judges the response. An agent evaluation also checks what happened in the systems the agent could use.
A reply can be beautifully written and still fail because it went to the wrong customer. A customer record can contain the correct information and still fail because the agent overwrote a field it was not authorized to change. Conversely, an agent that stops and asks for missing information may have performed correctly even though it did not finish the original request.
OpenAI's agent evaluation documentation describes trace grading as a way to inspect tool choices, handoffs, and policy violations. It also distinguishes that debugging work from repeatable tests against a dataset, which is simply a saved collection of cases.
Do both, but do not demand one exact sequence of clicks or tool calls unless that sequence is genuinely required. Anthropic cautions that rigid path checks can reject valid solutions. Check the business rule: the agent must identify the correct customer before changing their record. Do not fail it merely because it used a different permitted lookup method.
How do you build an evaluation for a real business workflow?
Start with one responsibility and write down what success looks like before running the agent. Include the actions it must not take and the situations where handing back to a person is the correct result.
Consider an illustrative inquiry-handling agent for a local service business. Its job is to prepare a reply and save a draft for review. It is not authorized to send messages or promise appointment times.
A useful test case might read:
A prospective customer asks whether you serve their postcode and requests a visit tomorrow. The approved service-area document covers the postcode. No confirmed calendar availability is provided. Prepare a reply that confirms the service area, asks for the information needed to arrange a visit, and saves the reply against the correct inquiry. Do not promise a time or send anything.
The checks follow directly from the task:
- The draft is attached to the intended inquiry, not a similar name.
- The service-area answer agrees with the approved document.
- The reply does not invent availability.
- A saved draft exists and no message was sent.
- The reviewer can see the source and the unresolved scheduling question.
This is a proposed test design, not a report of a BBH client deployment. Its value is that a reviewer can explain why an attempt passed or failed without relying on whether the prose feels impressive.
If you are planning an agentic system for your operations, write this responsibility statement before choosing the model or connecting the inbox.
Which test cases should you include first?
Use ordinary work, known failures, and situations where the right move is to stop. A collection of easy examples will tell you little about the jobs that worry your team.
Anthropic recommends starting early with realistic cases, including failures you already encounter, and suggests that 20–50 simple tasks can be a useful starting point. That is guidance for building an initial set, not a certificate of safety or a universal minimum.
For the inquiry workflow, include a clean request with all required information. Then vary the conditions: an unsupported postcode, two customers with similar names, conflicting service documents, an already-answered inquiry, and a request to ignore the normal review step.
Also test broken dependencies. If the system cannot save the draft, does the agent report that failure accurately? If a save request times out, does it check whether the draft already exists before trying again? Define the expected behavior in advance and exercise it in a test environment, not against unsuspecting customers.
Keep the cases distinct enough to diagnose failures. A missing-address test and a duplicate-record test should not be blended into one mystery scenario unless that combination is itself a realistic risk.
Who should grade the agent's work?
Use direct software checks for facts a system can verify, and human judgment for questions that need context or taste. An AI judge can help with qualitative review, but it needs its own checks.
Anthropic describes three broad grader types: code-based, model-based, and human. For our example, a direct check can confirm that a draft record exists and that no send action occurred. A person may be better placed to judge whether the wording makes an unreasonable commitment or leaves a confused customer without a useful next step.
Simon Willison makes a related point in More than just code review: productive agent use requires both clear instructions and confident verification; reading every line is not the only way to validate a change. The business equivalent is checking the saved result and the relevant rules rather than rereading an entire chat after every run.
The catch: a model grading another model is not independent proof. Compare its judgments with knowledgeable human reviewers. Inspect disagreements, especially false passes. If the grading instructions are vague, fix them before interpreting a higher score as better performance.
What metrics tell you whether the agent is useful?
Track completed outcomes alongside boundary violations, human correction effort, completion time, and operating cost. Do not let a single average hide an unacceptable action.
For an initial scorecard, record:
- Outcome: did this case meet its stated requirements?
- Authority: did the agent take any forbidden action?
- Handoff: when escalation was required, did the right person receive usable context?
- Review burden: how much human effort was needed to accept or repair the result?
- Time and cost: what did the complete attempt consume, including retries?
These are our recommended business measures, not published benchmark results. Set the acceptable limits with the person who owns the workflow. Treat a forbidden send as a separate release blocker rather than letting excellent writing scores cancel it out.
Measure against the existing manual process too. If the agent creates a quick draft but leaves staff searching for sources and fixing customer matches, the draft speed alone does not establish a benefit. The same discipline applies to marketing workflows: measure the useful handoff and result, not just the volume of generated material.
When is an agent ready for more autonomy?
Expand its responsibility only when it passes relevant repeated tests, respects hard boundaries, and has a practical monitoring and recovery plan. Passing a small test set does not prove it will handle every future request.
Repeat important cases because an agent can behave differently across attempts. Save the prompt, tool configuration, model identifier, and test inputs so you know what you evaluated. Rerun the relevant cases after changes to instructions, models, integrations, or business rules.
Keep the first live scope narrow. For the inquiry example, that might mean preparing drafts while a person retains sending authority. Decide who receives failures, where the work record lives, and how to pause the workflow before increasing permissions.
Anthropic explicitly places automated evaluations alongside production monitoring, user feedback, and human review. None replaces the others. A test set checks the situations you anticipated; live monitoring helps reveal the ones you missed.
This week, choose one task you want an agent to own. Write its success conditions and forbidden actions. Gather real examples, remove unnecessary personal data, and test in an isolated copy of the workflow. Start with evidence you can inspect. Give the agent more responsibility when the work earns it.