Check the checker before you trust it
A model that reviews documents can save most of a reviewer’s week. Before it does, run it beside the people it will help and count what it misses.
The 60-second version Open
- Using a model as a first-pass reviewer can cut review time sharply. The question is what it misses.
- Measure that before you rely on it: run the model on work people have already reviewed, and compare.
- Count two things: issues people found that the model missed, and issues the model found that people missed.
- Then change the job, not the safeguards: people review what the model flags, plus a sample of what it passes.
- Keep a final human check on anything that goes to a regulator, a customer or production.
The fastest way to lose trust in a model reviewer is to switch it on and wait for the first thing it misses. The slower, better way is to find out what it misses before anyone depends on it.
Anthropic published an example of the better way last week. Working with Enveda on manufacturing documentation, Claude reviewed 1,061 pages. It caught all 11 issues that human reviewers had found, and flagged seven more. Human review time fell 89%, from over 25 hours to under three.
The headline is the 89%. The part worth copying is the 11 of 11, because it is how you know a time saving like that is safe.
Compare against work people already did
The test only means something because there was an answer key: documents people had already reviewed, with the issues they found written down. Run the model on the same pages and you can count exactly what it caught and what it didn't.
You can do this with almost any review job. Take the last few weeks of support tickets, contracts or pull requests that people have already checked. Run the model over them without showing it the human notes. Then compare.
A reviewer you haven't tested against an answer key is a guess, however good it sounds.
Count both kinds of mistake
There are two numbers, and they mean different things. Issues people found that the model missed are the dangerous ones: that is what would slip through if you trusted it. Issues the model found that people missed are a bonus, but check them. Some will be real, and some will be noise that costs review time.
Don't stop at one batch. Eleven issues is a small sample, and a model that catches all of them on one set of documents can still miss the twelfth kind. Keep running the comparison as the work changes.
Change the job, not the safeguards
Once misses are rare, the model can take the first pass. People then review what it flags, plus a random sample of what it passes, so you keep measuring its miss rate instead of assuming it. In the same write-up, a regulatory document Claude drafted still went through standard human review, which found one minor error. That final check is the point: it stays.
What to do this week
Pick one review task your team does every week. Collect 50 items people have already checked, with their findings. Run a model over them cold, and count what it missed and what it added. If it misses things that matter, you have learned that cheaply. If it doesn't, you have the start of a review process that is faster and still checked.
A new essay arrives every Sunday.
Discussion