I've had this exact conversation with three different clients now, always around the same point in a project: someone, usually senior, asks when the human review step can be removed once the AI's accuracy improves. It's a reasonable question to ask. It's also, in our view, the wrong question, and getting the answer wrong is how organisations end up with AI systems that work fine for eighteen months and then cause a genuinely bad outcome in month nineteen.
The framing problem
There's an assumption baked into that question — that human review is a stopgap for an immature system, something you graduate out of once the AI proves itself. That framing makes sense for some kinds of software. It doesn't make sense for decisions that have real consequences for real people, because the question was never really "is the AI accurate enough." The question is "who is accountable when it isn't, and how do we catch that case before it causes harm."
Accuracy and accountability are different problems. You can have a 99.7% accurate fraud detection model and still need a human checkpoint on the 0.3%, because that 0.3% is exactly where the consequential, hard-to-reverse mistakes live. Improving the model's accuracy doesn't make the checkpoint less necessary — if anything, a more confident model that's occasionally wrong is more dangerous than a visibly uncertain one, because people stop double-checking it.
What "human in the loop" should actually mean
Done properly, this isn't a human rubber-stamping every single output the AI produces — that's not oversight, that's theatre, and it burns out the human reviewer within a month because they're approving things they have no real ability to meaningfully evaluate at that volume. Real human-in-the-loop design means the AI handles full autonomy on the cases it's confident about and within policy bounds, and routes — automatically, with full context — only the cases that are genuinely uncertain, high-stakes, or outside its training to a human who has the time and information to actually think about them.
That's a very different ratio than people imagine. In a well-tuned KYC system we've deployed, roughly 85% of cases never reach a human at all — they're confidently within policy and the agent handles them end to end. The other 15% reach a human, and crucially, they reach that human with the specific reason for escalation already attached, not a generic "please review." The human's time goes entirely toward judgment calls, not toward re-doing work the AI already did correctly.
Why this is permanent, not a phase
The honest case for keeping a human checkpoint isn't that the AI might be wrong today and right in two years. It's that the kinds of decisions worth automating in banking and regulated enterprise almost always have a tail of genuinely novel, ambiguous, or high-stakes cases that no model — however well trained — should be making alone, because the cost of a wrong autonomous decision in that tail is disproportionate to any efficiency gained by removing the checkpoint. The threshold for what counts as "the tail" can tighten as trust is established. The existence of a tail that needs a human doesn't go away.
What this looks like in an audit
When CBUAE or an internal audit team asks how a given AI decision was made, the answer that holds up isn't "the model was 94% confident." It's "here is the policy threshold the model operated within, here is the escalation path for anything outside it, and here is the named human who has override authority at every point in this workflow." That's not a weaker answer than full autonomy would be. It's the only answer that actually survives scrutiny, because it shows a system designed around the assumption that it will sometimes be wrong — which, eventually, every system is.