Stratified Sampling And Confidence Scores

Definition

A human-review-calibration pattern: have the model emit field-level confidence scores, route low-confidence items to humans, and use stratified sampling to keep auditing high-confidence items at a higher-than-random rate for high-risk segments.

Key points

• Have the model output field-level confidence scores (0%→100%). Requirement example: automate extractions only above >90% confidence.
• Routing: below 90% → Human Review Queue; at/above 90% → Automated Downstream Processing. Route low-confidence / ambiguous / contradictory extractions to human review.
• Stratified (random) sampling: sample high-confidence extractions for ongoing measurement, weighting low-confidence / high-risk strata at a higher rate — don't sample uniformly.
• Critical validation step: analyse accuracy by document type and field to verify high-confidence extractions perform consistently across all segments (not just in aggregate) before reducing human review.
• Calibrate thresholds with labeled validation sets.
• The model's self-reported confidence is an "always-wrong" routing signal on its own — calibrate against ground truth; also don't route on sentiment/frustration or complexity alone.

Why it matters for the exam

• Human-in-the-loop / confidence-calibration scenarios (D5.5). Tests the >90% threshold pattern, stratified sampling over uniform sampling, and segment-level validation before de-staffing review.

Common gotchas

• Trusting raw self-reported confidence without calibration — a classic wrong answer.
• Validating only in aggregate; you must check per doc-type and per-field before cutting review.
• Uniform random sampling instead of stratifying by risk/confidence.

See also

• Error Propagation In Multi-Agent Systems
• Structured Tool Errors
• Context Rot

Sources

• architects-playbook
• netlight-study-guide
• certsafari