Grade the outcome, not the transcript

Ask most teams how they test an agent and they show you transcripts. A reviewer, or another model, reads the conversation and scores it: polite, on topic, followed the steps. It feels thorough. It misses the thing that matters.

An agent's job is rarely the conversation. It's the refund that was issued, the record that was updated, the ticket that was routed. A transcript can be perfect while the database is wrong, and a transcript grader will pass it every time.

Check what changed

The alternative is simple to say: grade the state the agent leaves behind. Before the task, you know what the world should look like afterwards. After the task, you look. Was the refund issued, for the right amount, to the right customer, once? Did the order move to the right status? Did anything else change that shouldn't have?

This week Microsoft and Hugging Face described exactly this approach.1 Their ThinkingBox sandbox gives agents a real backend and tools, then compares the final database state with the expected outcome. Each check returns 1 or 0.

A test that reads the outcome can't be charmed by a good answer.

A test that is also a reward

A clean 1 or 0 is more than a better test. It is a reward signal. If you can check the outcome automatically, you can train on it: let the agent try a task many times, reward the attempts that leave the right state, and update the model. In the ThinkingBox write-up, the team did this and reported the pass rate on its benchmark rising from 51.7% to 60.8%.

The detail worth copying is how they chose what to train on. Tasks the base model always solves give no signal; neither do tasks it never solves. They kept the tasks in the middle, the ones the agent sometimes gets right, because that's where an attempt can be better or worse. It's the same shape as a good review band: people, and training, should spend their effort where the outcome is uncertain.

Where a judge still helps

Not everything is a database field. Whether a reply was clear, or whether the agent explained a refusal well, can't be read from the final state. That's where a judge, a model or a person, still earns its place. The point is to use it for what the state can't show, not as the main test.

And a passing score is still a probability, not a promise. An agent that leaves the right state six times in ten is right six times in ten on cases like those. Treat its success rate like any other score: check it on your own tasks, and keep the costly and uncertain cases in front of a person.

What to do this week

Pick the three actions your agent takes most often. For each one, write down what the world should look like after it's done correctly, and write a check for it. Run your existing test cases through those checks instead of reading the transcripts. You will find failures you didn't know you had, and you will have the start of a reward you can train on later.