What you need to know
"Put a human in the loop" is not a design. The questions are which decisions need a person, where in the flow the person sits, and what they need to do the job well.
Route by stakes, not by volume
Three properties of a decision set the need for review:
- Reversibility: can a wrong decision be undone cheaply? A draft email can; a sent payment or a denied benefit often cannot.
- Cost of a wrong decision: a mislabelled ticket costs little; a wrong clinical or financial decision costs a lot.
- Confidence: how likely the system is to be wrong on this case. Confidence is only useful if it is calibrated against real outcomes; a model can be confidently wrong.
| Confidence | Reversible? | Cost of error | Review |
|---|---|---|---|
| High | Yes | Low | None; monitor |
| Low | Yes | Low | None or sampled; monitor |
| High | No | High | Often still required, or sampled at a high rate |
| Low | No | High | Always, before the action |
Where the person sits
| Placement | How it works | Fits | Cost |
|---|---|---|---|
| Pre-action approval | Nothing happens until a person approves | Irreversible or high-cost actions | Latency and reviewer capacity |
| Post-action audit | The action happens; a person reviews afterwards and can reverse it | Reversible, lower-cost actions at volume | Errors take effect before they are caught |
| Sampled review | A share of decisions is reviewed to measure quality | Monitoring routine work | Individual errors can slip through |
Many systems combine them: pre-approval for the risky band, post-action audit for the middle, sampling for the rest.
Keep oversight meaningful
Sending everything to people looks safe and fails in practice. Reviewers facing long queues of routine approvals start approving without reading (consent fatigue or approval fatigue), so the risky cases get the least attention. Design the reviewer's job:
- Route only what needs judgment, so each review matters.
- Show the evidence: the inputs, the sources the system used, why it was flagged, and what the system recommends.
- Make disagreeing easy, and record overrides with a reason.
- Track reviewer agreement and time per case; a fall in review time with rising volume is a warning sign.
- Feed overrides back into evaluation (lesson 4.5).
Diligence is the human counterpart: the reviewer remains accountable for what they approve, which only works if the system gives them what they need to judge.
In the exam
Review questions give a decision type, volume, reversibility and cost, and ask which cases go to a person and when. Apply the routing rule, pick pre-action approval for irreversible or high-cost actions, post-action audit or sampling for reversible ones, and prefer designs that give reviewers evidence and avoid flooding them.
Check your understanding
A system drafts replies to internal IT tickets; agents can edit any draft before sending. Volume is 5,000 a day. What review design fits?
Show answer
Answer: D. Drafts are reversible and low-cost, and agents already see them. Sampling monitors quality without creating a bottleneck; no monitoring at all would hide drift.
Which decision most clearly needs pre-action human approval?
Show answer
Answer: C. Account closure on a weak signal is hard to reverse and costly if wrong. That combination is exactly what pre-action approval is for.
Reviewers approve 99.8% of a growing queue in about four seconds each. Which TWO changes restore meaningful oversight?
Show answer
Answer: A, C. The pattern is consent fatigue. Fewer, better-chosen cases with the evidence attached make each review real; more staff on the same queue or reminders do not change it.
Apply it: Design review for Larkfield's glass fast-track
Larkfield wants to fast-track payment of small motor glass claims (see the case). Regulation forbids automated denials, not automated payments, but paying a fraudulent or uncovered claim costs money and is hard to recover.
- Classify the decisions in the fast-track (pay, refer, deny) by reversibility and cost.
- Write the routing rule: which glass claims pay without a person, which go to an adjuster first, and which are audited afterwards.
Done when: no denial is made by the system. - Define what the adjuster sees when a claim is referred.
- Say how you will detect consent fatigue among the adjusters.
Show model answer
1. Decisions
| Decision | Reversible? | Cost if wrong |
|---|---|---|
| Pay a covered claim | Hard to recover once paid | About $420 each; larger if fraud is organized |
| Refer to an adjuster | Fully | A delay of a day or two |
| Deny | Not allowed for the system | Regulatory breach |
2. Routing rule
- Pay automatically when Policy Admin confirms glass cover on the date of loss, the amount is within the policy's glass limit and under $1,000, the fraud engine score is below the threshold, the photo check passes, and the customer has had fewer than two glass claims in 12 months.
- Refer before paying when any condition fails, when the system's confidence on the photo or invoice is low, or when the amount is $1,000 or more.
- Never deny: anything that might be uncovered goes to an adjuster.
- Audit afterwards: 5% of automatic payments are sampled weekly, plus all payments to repairers new to Larkfield.
3. Referral view. The claim, photos, invoice, extracted fields next to the source, the Policy Admin result, the fraud score with its top factors, the customer's claim history, the specific condition that caused the referral, and buttons to pay, request information, or pass to the special investigations team.
4. Consent fatigue. Track referrals per adjuster per day, median time per referral, and the share approved unchanged. A falling review time with a rising approval rate triggers a review of the routing rule (too many low-value referrals) before it becomes rubber-stamping.