What you need to know
Aggregates hide segments
"97% accurate" can mean 99.5% on typed invoices and 70% on handwritten receipts, or 99% on totals and 80% on tax IDs. Before reducing human review, break accuracy down by document type and by field, and check every segment meets the bar.
| Segment | Volume | Accuracy |
|---|---|---|
| Typed invoices | 70% | 99.1% |
| Contracts | 10% | 96.0% |
| Receipts (photos) | 15% | 91.5% |
| Handwritten invoices | 5% | 78.0% |
Overall this is about 97%, and one segment is far below it.
Keep measuring high-confidence output
Once high-confidence extractions skip review, nobody sees their errors. Stratified random sampling keeps a small, regular sample of high-confidence output from every segment going to reviewers. It measures the ongoing error rate and catches new error patterns, such as a supplier changing its layout.
Calibrated, field-level confidence
Have the model output a confidence per field, not one per document. Then calibrate the thresholds against a labelled validation set: find the confidence level above which a field is right, say, 99.5% of the time, and use that as the routing threshold. Uncalibrated self-reported confidence is not reliable on its own.
Spend scarce reviewer time well
If reviewers can check only 10% of documents, reviewing every tenth in arrival order wastes most of their effort on easy documents. Route instead:
- Low-confidence fields.
- Documents with ambiguous or contradictory content (
conflict_detected). - Segments with known weak accuracy.
- Plus the small stratified sample of high-confidence output.
In the exam
Scenario 6: leadership wants to cut review based on overall accuracy, reviewers are assigned in arrival order, or confidence scores are used without calibration.
Check your understanding
Accuracy is 97% overall and leadership wants to stop reviewing high-confidence extractions. What must happen first?
Show answer
Answer: D. An aggregate can hide segments that perform badly. Each segment must meet the bar before review is reduced for it.
How should review thresholds for field-level confidence be set?
Show answer
Answer: A. Self-reported confidence needs calibration. A labelled set shows which confidence level corresponds to the accuracy you need.
Reviewers can check 10% of documents and currently take every tenth in arrival order. What is better?
Show answer
Answer: C. Directing review to likely errors, plus a stratified sample of the rest, makes limited capacity count.
Apply it: Bayview's review routing
Bayview claims 97% accuracy, reviewers can check 10% of documents, and they currently review every tenth document. Design the review workflow.
- Define the segments you would measure before any automation.
- Write the routing rules for the 10% capacity.
- Design the stratified sample for high-confidence output.
- Describe how thresholds are calibrated.
Show model answer
1. Segments. By document type (typed invoice, handwritten invoice, receipt, contract, other) and by field (totals, dates, tax ID, vendor, line items). Accuracy is reported per cell.
2. Routing, in order
- Any field below its calibrated threshold.
conflict_detectedtrue, orcalculated_totaldiffering fromstated_total.- Segments below 95% accuracy (for example handwritten invoices), all documents until the segment improves.
- Remaining capacity: the stratified sample.
3. Sample. 1% of high-confidence documents each week, drawn randomly within each document type in proportion to volume, with a minimum of 30 per type so small segments are measured.
4. Calibration. Label 2,000 documents across all segments. For each field, find the confidence level above which accuracy reaches 99.5%, and use it as the threshold. Recalibrate quarterly and whenever the sample shows a new error pattern.