Eval as a release criterion: harness vs demo-grade
An AI feature approved on the same inputs as the pitch breaks in production on the long tail of real cases. An eval harness makes a suite of checks the release criterion and runs the feature against that tail: regression from past incidents, a distribution diff, and a human spot-check — before deploy, not after the incident.
Problem
An AI feature that works in the demo breaks in production unpredictably and inexplicably. Most teams find out after the release. The reason — "demo grade": the eval that approved the feature used the same inputs as the pitch. Curated dataset, controlled conditions. The long tail of real traffic was never run.
Methodology
Make AI output quality a release criterion, not a retrospective finding. Minimum eval at release — three layers.
-
1. Layer 1 — Regression (non-negotiable).
A fixed set of reference cases with known expected output: edge cases from past
incidents, cases from the demo (that got the feature approved), at least one
adversarial input per output type (prompt injection, empty, malformed). Any new
failure blocks the release.
# cases/rag-stale-corpus.yaml id: rag-stale-corpus origin: incident-2026-03 # RAG hallucinated on a full corpus input: query: "current rate for tariff X" expected: must_contain: - "current document" must_not_contain: - "status: archived" retrieved_chunks: min_count: 1 must_contain: - "active" must_not_contain: - "archived" pass_criteria: - no_archived_docs - answer_groundedThe case is born from an incident (
origin) — that's what separates a regression set from a demo dataset: the inputs come from what has already broken in production. -
2. Layer 2 — Distribution check (RAG/classifiers).
20–50 fresh real inputs through the new version. Not a check against an "ideal,"
but against a diff vs the previous release
snapshot: distribution of length/format, change in retrieved chunks, confidence
shift (shift toward the extremes = brittleness).
The mechanism is not a statistical test — it's thresholds on summary statistics against the snapshot:
# layers/distribution.py LENGTH_DRIFT_THRESHOLD = 0.30 # rel-change of mean > 30% → flag CONFIDENCE_STD_DRIFT_THRESHOLD = 0.20 # confidence std growth > 20% → brittleness std_growth = (current_std - baseline_std) / baseline_std if std_growth > CONFIDENCE_STD_DRIFT_THRESHOLD: flags.append(f"confidence std grew {std_growth:.1%} — fragility signal") # mean confidence shifting toward the edges [<0.35 | >0.85] with movement > 0.15 edge_shift = abs(current["confidence_mean"] - baseline_mean) if edge_shift > 0.15 and (mean < 0.35 or mean > 0.85): flags.append("confidence mean shifted toward edges")Why thresholds and not KS/PSI: on 20–50 samples the two-sample Kolmogorov–Smirnov test is underpowered and throws false alarms; thresholds on
mean/pstdevare a cheap gate at that N. The price: you don't catch changes in shape (bimodality, a fattening tail) — coarse shifts in mean/spread show up, subtle ones don't. Upgrade path as the set grows: PSI or a two-sample KS on the length and confidence distributions. - 3. Layer 3 — Human spot-check. Required before the first production deploy of a new type of output: a domain expert reads 10–20 real outputs with specific questions (any data the model shouldn't have access to; would an expert consider this correct; any adversarial exploitation).
- 4. Owner. One person signs off. If you can't name the owner before release — that's already the first red flag.
Artifact
Minimal harness for this checklist: a Python script that runs the regression set against a snapshot and compares distribution (Layer 1 + Layer 2). The system under test plugs in through a thin adapter — the harness knows nothing about the model, only the contract:
# adapter.run(input) -> {"output": str, "chunks": list, "confidence": float}
response = adapter.run(raw_input)
Wired as a release gate through exit codes, so CI can block the merge:
python run_eval.py --all --inputs inputs.jsonl
# exit 0 — both layers passed, release allowed
# exit 1 — regression failed → hard block (a known-bad came back)
# exit 2 — distribution drift → softer: a review signal, not an auto-block
The split is deliberate: regression is "a known thing broke," an unambiguous stop;
distribution drift is "behavior moved," a reason for a human, not an automatic
rejection. The snapshot updates only via an explicit
--update-snapshot — the
baseline moves on purpose, not on its own.
- github.com/dobryakov/eval-harness — regression + distribution check, adapter interface, demo case from a real incident. The harness is vibe-coded and illustrative: a skeleton of the practice, not a library.
Where it breaks
- Eval dataset = demo dataset — the most common anti-pattern: "eval" turns out to be the stakeholder demo, not a measurement.
- Pass criteria defined after the run — "looks reasonable" instead of a concrete measurable threshold.
- No owner — sign-off delegated to "it looked fine in staging."
- Real failure modes that produced this methodology: RAG correct on curated corpus, hallucinates on full; classifier silently degrades after upstream input schema change; AI-generated code passes automated tests but introduces a security pattern the tests didn't cover.
- A threshold misses a change in shape. The distribution layer sees a shift in mean and spread, but will miss bimodality and a fattening tail at the same mean. Mitigation: once the set grows to hundreds of samples, swap thresholds for PSI or a two-sample KS. What to monitor in production alongside eval: the share of outputs near the confidence edges and p95 answer length — early signals of drift between releases.
For whom and why
Risk: eval green, prod red — the standard story. Teams find out after release and fix it more expensively than if they'd caught it before.
Solution: a feature doesn't ship until three eval layers pass — not because it "looks good," but because it meets a criterion.
Metric: when the board asks "what's our AI quality story," you don't answer "we tested it" — you show the regression and distribution diff.
Closest to a production playbook: a practice-level artifact for Head of AI and technical CTO. Vibe-coding the harness turns the methodology into a reproducible code artifact — closing the code gap, not just the playbook.
Keep reading
About the author
Grigoriy Dobryakov — fractional / interim eng + AI leadership
Want AI release quality as a first-class engineering criterion?
Structured eval before every release that changes model behavior — regression, distribution check, human spot-check, named owner.
Discuss a fractional audit Email meOther breakdowns
An engineering breakdown series: real task → methodology → working artifact → honest breakdown of where it fails.
Back to series →