Every team that puts an AI agent in front of customers hits the same wall. At first a person reads every answer before it goes out. That’s safe, and it’s slow. A few weeks in, someone asks the obvious question: “Do we really need to read all of these?” The usual answer is to either keep reviewing everything (and burn out) or stop reviewing (and hope).
I think there’s a third option: treat human review as the start of an eval system. Every time a person approves or fixes an answer, that decision is data. Collect it properly and it does three jobs: it tests the agent, it teaches the agent, and it tells you which answers no longer need a human at all. If it works, the review queue shrinks every week and quality holds.
I’ll work through it with a simple example: a support bot for an online store that answers questions about orders, returns and delivery.
The core idea: a review is a label
When a reviewer approves an answer, they’ve said “this is what good looks like for this question”. When they fix one, they’ve said “this is wrong, and here’s the right version”. Most teams throw that away: the fixed answer goes to the customer and the lesson is lost.
Instead, save every decision as a row:
| Question | Bot’s answer | Verdict | Fixed answer | Reason |
|---|---|---|---|---|
| “Can I return a sale item?” | “Yes, within 30 days.” | Fixed | “Sale items can be exchanged, not refunded.” | Policy |
| “Where’s my order #4471?” | “It ships tomorrow.” | Approved | ||
| “Do you ship to Canada?” | “Yes, in 5 to 7 days.” | Fixed | “Not yet. We ship within the US only.” | Made up a fact |
Honestly, that table is most of the system. Everything below is just different ways of reading it.
Three things one review does
1. It becomes a test. Every fixed answer is a question you know the bot got wrong, plus the right answer. Add it to a test set. Before any change to the prompt or the model, rerun the whole set. If an old mistake comes back, you know before a customer does.
2. It becomes a lesson. Group the fixes by reason. If “Policy” shows up twelve times in a week, the bot doesn’t need twelve corrections: it needs the returns policy in its instructions. One rule can fix the whole group. (This is the same loop as my post on the prompt that learns: feedback flows back into the thing that decides behaviour.)
3. It calibrates trust. Over time you learn which kinds of question the bot gets right nearly every time. Order status? Approved 99 times out of 100. Refund exceptions? Fixed half the time. That’s how you decide where a human is still needed.
Letting a second model do the first pass
A person can’t read every answer forever, so add a reviewer model (often called a judge). It reads the question, the bot’s answer and the relevant policy, and gives a score with a reason: “confident this is correct”, “unsure”, or “likely wrong”.
The judge doesn’t replace the human. At first, it runs alongside the human, and you compare them. If the judge says “confident” and the human approves, they agree. If the judge says “confident” and the human fixes it, that’s the dangerous case: a confident mistake.
You only let the judge’s “confident” skip human review once that dangerous case is rare enough, say under 1 in 200, for a given kind of question. This is a measurement you can do with a simple count, not a feeling.
The shrinking funnel: four stages
Trust is earned per kind of question, not for the bot as a whole. Order status can be fully automatic while refunds still go to a person.
| Stage | What a human reviews | Moves on when |
|---|---|---|
| 1. Shadow | Everything. The judge scores silently. | A few hundred reviews collected for this kind of question |
| 2. Assisted | Everything the judge isn’t confident about, plus a random 20% of the rest | Confident mistakes stay under 1 in 200 for a month |
| 3. Spot check | Unsure answers, plus a random 5% | Same bar holds |
| 4. Audit | A random 1 to 2%, plus anything a customer complains about | Never “done”: the sample stays forever |
Read left to right, human effort goes where the bot is weakest and moves away as the bot proves itself.
The numbers worth watching
Three numbers tell you if the system is working:
- Auto-approve rate: the share of answers that go out without a human. This should rise.
- Confident-mistake rate: of the answers the judge waved through, how many did the random sample catch as wrong? This should stay flat and low. If it rises, drop that kind of question back a stage.
- Repeat mistakes: how often a mistake already in the test set happens again. This should be close to zero. If it isn’t, the lessons aren’t reaching the prompt.
The business will love the first number. The second one keeps it honest. If you only watch the first, you’ll automate yourself into trouble.
The traps
Rubber-stamping. When a reviewer sees “the judge is confident” next to an answer, they approve faster and look less closely. Fix: in the random sample, hide the judge’s score from the reviewer, so their verdict is independent.
The sample misses rare disasters. A 2% sample will almost never catch the one answer a month that promises a refund you don’t offer. Fix: some topics (money, legal, safety) always go to a human, whatever the stats say.
The judge drifts. If you change the judge’s model or prompt, its old track record no longer counts. Fix: treat a judge change like a bot change. Rerun the test set, and drop back one stage until it re-earns trust.
Grading the wrong thing. If the judge only checks “is this polite and on topic”, it will pass answers that are polite, on topic and wrong. Fix: give the judge the policy and the order data, and have it check facts, not tone.
Closing thought
Human review usually feels like the cost of using AI safely. Set up like this, it’s also what makes the bot better every week. So instead of asking “how do we review less?”, I’d ask “how do we make every review count?” Get that right and reviewing less follows.