Human-in-the-loop calibration

Definition

Tuning when a pipeline auto-resolves vs routes to a human, using field-level confidence scores and a fixed threshold rather than gut feel.

Key points

• Automate extractions with model confidence >90%; route everything below to a Human Review Queue, and send at/above 90% to automated downstream processing.
• Implementation: have the model output field-level confidence scores (not one aggregate score), grounded in the solution for reducing semantic errors.
• Critical validation step: analyze accuracy by document type and by field to confirm high-confidence extractions perform consistently across all segments, not just in aggregate, before deploying.
• Related reliability lever: stratified sampling — sample low-confidence / high-risk records at a higher rate for human review.
• Distinct from support escalation: a confidence score is a routing signal for extraction QA, but is an always-wrong signal for human-handoff decisions in support flows.

Why it matters for the exam

• The >90% auto-resolve + field-level scores design is a recurring "how do you calibrate human review?" answer.

Common gotchas

• Aggregate accuracy hiding a weak document type/field is the trap — validate per segment.
• Confidence gating extractions ≠ confidence triggering a human handoff (different domains).

See also

• Escalation triggers
• Wrong escalation signals
• Limits of automated retry
• Minimal footprint principle

Sources

• architects-playbook
• netlight-study-guide