Loading…
Loading…
100 captured onnx-community/SmolLM2-135M-ONNX runs, seeded 42 split 50/50; thresholds picked on the first half, reported on the second: confidence ≥ 0.5, entropy ≤ 6 bits.
Reference model: optional audit, off by default.
Sweeping confidence τ 0.3–0.95 at entropy ≤ 6 bits on the held-out split. The blue marker denotes the calibrated τ = 0.5; the dashed red line is un-gated baseline accuracy (40.0%).
| Regime | Confidence Bar | Coverage | Accuracy | Auto / Held | Gain vs Baseline |
|---|---|---|---|---|---|
| Ultra-Strict | τ ≥ 0.90 | 2.0% | 100.0% | 1 auto (49 escalated) | +60.0% |
| High Precision | τ ≥ 0.75 | 10.0% | 60.0% | 5 auto (45 escalated) | +20.0% |
| Calibrated Gate (τ = 0.5) | τ ≥ 0.50 | 10.0% | 60.0% | 5 auto (45 escalated) | +20.0% |
| Permissive | τ ≥ 0.35 | 12.0% | 66.7% | 6 auto (44 escalated) | +26.7% |
| Always-Trust (Ungated) | None (Ungated) | 100.0% | 40.0% | 50 auto (0 escalated) | 0.0% |
| Split | Auto | Coverage | Accuracy | Baseline | Wrong |
|---|---|---|---|---|---|
| Calibration (50) | 2/50 | 4.0% | 50.0% | 26.0% | 1 |
| Held-out (50) | 5/50 | 10.0% | 60.0% | 40.0% | 2 |
Thresholds were picked on the calibration half; only the held-out half above counts as the honest report.
“The invoice has the wrong company name, can you fix it”
“Do you ship to Taipei, and how long does it take”
No reference model recorded. The numbers above are scored against the author's synthetic gold labels only. Run pnpm reference with a key set to add a second, independent label source; nothing above changes and the app needs no key.
Reproduce every number here: pnpm capture && pnpm calibrate