Treat AI-output quality as a release criterion: a fixed regression set built from past incidents, a distribution diff against the previous release snapshot (20–50 real inputs), a human spot-check before the first production deploy of a new output type — and one person who signs off. Below: the demo-grade anti-pattern, the three-layer method, and a minimal harness that fails CI with a readable exit code.
Most AI features ship in demo grade. Not because teams do not care — because the eval that approved the feature used the same inputs as the pitch. Curated dataset. Controlled conditions. The long tail of real production traffic never ran.
Then production shows what the demo missed:
- a RAG system that retrieves correctly on a staging corpus and hallucinates on the full corpus with stale entries;
- a classifier that silently degrades after an upstream input-schema change;
- AI-assisted code that passes automated tests but introduces a security pattern the tests never covered.
Three stacks. One failure mode: no structured eval, no named owner, approval by “looked fine in the demo.”
Agent drift is when model or agent behavior leaves what you treated as “verified” — without a red alert at release time. Sometimes the prompt or corpus changed. Sometimes the input distribution shifted quietly. The symptom is the same: green eval on the demo set, red production on the live tail.
When AI-output review is actually required
Not every deploy. The trigger is simple: does this release change what the model sees, or what it outputs?
Review before release when any of the following is true:
- the output is used in a customer-facing flow (response, recommendation, classification, search);
- the output influences a business-critical decision (approval, routing, pricing, alert);
- the output is built from retrieval (RAG) — hallucination risk scales with retrieval quality;
- the model, prompt, or retrieval corpus changed since the last release.
Infrastructure-only deploys, pure UI, config that does not touch prompt/retrieval — a full eval is optional. Saving cost here is fine; self-deception is when “infra” quietly swaps the index or the system prompt.
Ownership: one person, not “the team”
One person owns the eval. Not “everyone.” Not “it looked fine in staging.”
Typical pattern:
| Role | Responsibility |
|---|---|
| Feature owner (engineer or PM) | defines “good output” in business terms |
| ML/AI lead | validates methodology and sign-off criteria |
| Release approver (EM / tech lead) | final gate; sign-off cannot be replaced by “looked fine in staging” |
If you cannot name the owner before release, that is already the first risk signal. The later argument about “who should have caught it” is almost guaranteed.
Three layers of a minimal eval
Layer 1 — Regression (non-negotiable)
A fixed set of reference cases with known expected outputs. The set must include:
- edge cases from previous incidents;
- cases from the demo that were used to approve the feature (yes — run them, but not only them);
- at least one adversarial input per output type: prompt injection, empty input, malformed input.
Pass criterion: every reference case stays inside the defined acceptable range. Any new failure blocks release.
The key: a case is born from an incident. The origin field in YAML is not bureaucracy — it is what separates a regression set from a demo dataset. Inputs come from what already broke in production, not from a board slide.
Example from the bundled rag-stale-corpus case (eval-harness repo):
id: rag-stale-corpus
origin: incident-2026-03
input:
query: "действующая ставка по тарифу X"
expected:
must_contain:
- "актуальный документ"
must_not_contain:
- "archived"
retrieved_chunks:
min_count: 1
must_contain:
- '"status": "active"'
must_not_contain:
- '"status": "archived"'
pass_criteria:
- no_archived_docs
- answer_groundedmust_contain / must_not_contain check the answer text. retrieved_chunks check serialized chunks (often JSON). Needles must match your real RAG format: a log-style "status: archived" will not hit {"status": "archived"}, and the test will pass vacuously.
The same repo ships adversarial cases: empty, malformed, injection (ignore previous instructions…). Their job is to lock in a refusal — not a “nice answer to an attack.”
Layer 2 — Distribution check
For RAG and classifiers. Take 20–50 recent real inputs, run the new version, and compare not to an “ideal,” but to a diff against the previous release snapshot:
- output length / format — outliers often mean a broken prompt or schema change;
- retrieved chunks — did they change, and why;
- confidence distribution — a shift toward the edges means brittleness.
In eval-harness the drift flags are explicit:
| Signal | Threshold |
|---|---|
mean output length vs snapshot |
shift > 30% |
confidence std |
increase > 20% |
mean chunks count |
shift > 30% |
| mean confidence toward edges | < 0.35 or > 0.85 with a clear baseline delta |
These are not “scientific truth.” They are a working gate: if the distribution moved without an explanation, the release stops until someone owns the reason.
After an accepted ship, update the snapshot separately (--update-snapshot). Do not pass that flag in the release pipeline — otherwise the baseline quietly absorbs the degradation.
Layer 3 — Human spot-check
Process, not code. Mandatory before the first production deploy of any new output type: a domain person reads 10–20 real outputs with specific questions:
- does the output contain information the model should not have access to;
- would a domain expert consider it correct;
- is there adversarial misuse sitting in plain sight.
On later releases, trigger spot-check on model, prompt, or corpus change, or after an incident. Without this layer you automate blindness: regression green, distribution “in band,” meaning already garbage.
The demo-grade anti-pattern
Demo grade: output that looks correct under controlled conditions and fails on the real input distribution.
How it usually ships:
- the feature is built and “tested” against the brief examples;
- those examples were written to show capability, not to stress-test;
- staging runs on a curated or synthetic dataset;
- a reviewer approves on “looks reasonable”;
- production opens the long tail the demo never saw.
Symptoms you are shipping demo-grade:
- eval inputs = pitch or sprint-review examples;
- nobody ran the feature on real production inputs before release;
- the “eval” is a stakeholder demo, not a measurement;
- pass criteria are written after the run (“yeah, seems fine”).
The fix is short and uncomfortable: demo dataset ≠ eval dataset. The demo shows the best case. The eval measures the real distribution.
Artifact: eval-harness in CI
A minimal harness for Layer 1 + Layer 2: github.com/dobryakov/eval-harness. Python, YAML cases, JSONL inputs, snapshot, adapter interface.
Exit codes so the gate is machine-readable:
| Exit code | Meaning |
|---|---|
0 |
ship |
1 |
Layer 1 failed (regression) |
2 |
Layer 2 failed (distribution drift) |
Quick start:
pip install -r requirements.txt
python run_eval.py --layer regression
python run_eval.py --layer distribution --inputs inputs.jsonl
python run_eval.py --all --inputs inputs.jsonlWire your model / agent through adapter.py:
def run(input: dict) -> dict:
return {
"output": str, # answer text
"chunks": list, # retrieved chunks
"confidence": float, # 0..1
}In CI:
- run: pip install -r requirements.txt
- run: python run_eval.py --all --inputs inputs.jsonlEvery incident that must not recur becomes a permanent regression case under cases/. Otherwise you “fixed” it once and wait for the same bug under a different prompt.
Repo layout:
eval-harness/
run_eval.py
adapter.py
layers/
regression.py
distribution.py
cases/
*.yaml
snapshots/
latest.json
inputs.jsonlLayer 3 stays out of the code — correctly: a human spot-check cannot be closed by a green script checkbox.
Checklist before a release that changes AI output
- [ ] Owner named: who signs off on AI-output quality
- [ ] Regression set run: all reference cases pass
- [ ] No new failure modes vs the previous release
- [ ] Distribution check done (for RAG / classifiers): no unexplained shift
- [ ] Human spot-check done if: new capability / model change / prompt change / prior incident
- [ ] Eval dataset ≠ demo dataset
- [ ] Pass criteria defined before the run, not after
Limits of the method
- Distribution thresholds are heuristics. 30% / 20% catch coarse drift; subtle semantic degradation with stable length can slip past Layer 2 — that is why Layer 3 remains.
- Empty
originand “pretty” cases. If regression is built from showcase examples, you get demo-grade again — only in YAML. - Vacuous pass on chunks. A wrong needle format in
retrieved_chunksyields a false green. Serialize a real chunk the same way the adapter does, then write the case. - No owner — no gate. A harness in CI without someone accountable for a red result becomes noise people learn to bypass (
continue-on-error: true). - Snapshot after degradation.
--update-snapshotat the moment of “fine, this is the new normal” locks bad in as baseline. - Not every deploy. Mandatory eval on every hotfix kills adoption; keep the trigger “does this change model input/output.”
Who this is for
For CTOs, Heads of AI, and tech leads who already have pilots, and whose board question is no longer “do we have AI” but “what is our AI quality bar.” “We tested” is a weak answer. “Here is the regression set and the distribution diff vs last release” is a working one.
Neighboring layers of the same discipline: agent perimeter without relying on the system prompt and skill onboarding as governance. Eval gates model output; those gate access and the introduction of new assistant behavior.
Bottom line
If your eval matches the pitch, you are not checking production — you are checking a presentation. Split demo from eval, build regression from incidents, measure distribution diff, name one owner, and do not ship until all three layers pass on criteria — not because it “looks good.”