Treat AI-output quality as a release criterion: a fixed regression set built from past incidents, a distribution diff against the previous release snapshot (20–50 real inputs), a human spot-check before the first production deploy of a new output type — and one person who signs off. Below: the demo-grade anti-pattern, the three-layer method, and a minimal harness that fails CI with a readable exit code.
How to make eval a release criterion for AI features instead of a demo-grade approval
Demo-grade eval greenlights a feature on the same examples as the pitch. Production then shows the long tail. Three layers — incident-born regression, distribution diff vs the last snapshot, human spot-check — plus one named owner. Minimal harness: github.com/dobryakov/eval-harness.