The gate that trains its own replacement
Human review is usually framed as a temporary cost you remove once the model is good enough. Built differently, it is the thing that makes the model good enough, and the thing that tells you when to stop.
There is a standard story about automating judgement work. Phase one, humans do it. Phase two, a model does it and humans check. Phase three, the model does it alone.
The story is appealing and it produces bad systems, because it treats the human phase as scaffolding to be removed rather than as the mechanism that generates the evidence for removing it.
One pipeline, not two eras
The alternative is to build the automated path on day one and put the human at a decision point inside it, rather than beside it.
The seat proposes. The human approves, overrides, or corrects. The work then flows onward through exactly the same path it will use when the gate eventually relaxes. There is no "manual mode" later replaced by an "automated mode," because there was only ever one pipeline with a decision point in it.
This matters more than it sounds. Systems built as two eras have a migration problem: everything downstream was shaped by the manual process, and switching is a rewrite. Systems built as one pipeline with a gate have no migration at all. The gate simply fires less often.
Supervision is the training data
Here is the part that changes the economics. Every gated decision - the proposal, the verdict, and the gap between them - is a labelled example, produced as a byproduct of work that had to happen anyway.
You are not running a separate annotation project. The cost of supervision is the cost of doing the job, and the output is a corpus that tells you precisely where the seat agrees with a person and where it does not.
That corpus is also the only honest basis for relaxing anything. "The model seems good now" is not evidence. "On this class of decision, across this many cases, proposal and verdict have matched" is.
Relax per class, not globally
Gates come down by category, not all at once.
Some decisions are easy and high-volume; the seat will agree with the human almost immediately and continuing to review them is pure cost. Others are rare, ambiguous, and consequential. Those stay gated far longer, possibly forever.
Treating "should this be automated" as one question about the whole system forces you to answer it at the level of the hardest case, which means either nothing gets automated or something dangerous does.
Irreversibility sets the order
Not all decisions cost the same when wrong.
A misrouted lead is recoverable - someone notices, it gets rerouted. A decision that permanently excludes someone is not. So the class that is hardest to undo is the last to have its gate relaxed, regardless of how well the seat performs on it. Performance is measured on cases you know about, and the failure mode here is a case you never hear about.
Ordering by consequence rather than by accuracy is the discipline. It is also the part most often skipped, because irreversible decisions are frequently the easiest to classify, which makes them look like obvious early wins.
They are not. They are the last thing you touch.