Eval as a Release Criterion: Catching Agent Drift Before Production | Grigoriy Dobryakov

Grigoriy Dobryakov

Howto · breakdown

Breakdown 10

Eval as a release criterion: harness vs demo-grade

An AI feature approved on the same inputs as the pitch breaks in production on the long tail of real cases. An eval harness makes a suite of checks the release criterion and runs the feature against that tail: regression from past incidents, a distribution diff, and a human spot-check — before deploy, not after the incident.

CTO Head of AI Tech Lead

Problem

An AI feature that works in the demo breaks in production unpredictably and inexplicably. Most teams find out after the release. The reason — "demo grade": the eval that approved the feature used the same inputs as the pitch. Curated dataset, controlled conditions. The long tail of real traffic was never run.

Methodology

Make AI output quality a release criterion, not a retrospective finding. Minimum eval at release — three layers.

  1. 1. Layer 1 — Regression (non-negotiable). A fixed set of reference cases with known expected output: edge cases from past incidents, cases from the demo (that got the feature approved), at least one adversarial input per output type (prompt injection, empty, malformed). Any new failure blocks the release.
    # cases/rag-stale-corpus.yaml
    id: rag-stale-corpus
    origin: incident-2026-03   # RAG hallucinated on a full corpus
    input:
      query: "current rate for tariff X"
    expected:
      must_contain:
        - "current document"
      must_not_contain:
        - "status: archived"
      retrieved_chunks:
        min_count: 1
        must_contain:
          - "active"
        must_not_contain:
          - "archived"
    pass_criteria:
      - no_archived_docs
      - answer_grounded

    The case is born from an incident (origin) — that's what separates a regression set from a demo dataset: the inputs come from what has already broken in production.

  2. 2. Layer 2 — Distribution check (RAG/classifiers). 20–50 fresh real inputs through the new version. Not a check against an "ideal," but against a diff vs the previous release snapshot: distribution of length/format, change in retrieved chunks, confidence shift (shift toward the extremes = brittleness).

    The mechanism is not a statistical test — it's thresholds on summary statistics against the snapshot:

    # layers/distribution.py
    LENGTH_DRIFT_THRESHOLD = 0.30          # rel-change of mean > 30% → flag
    CONFIDENCE_STD_DRIFT_THRESHOLD = 0.20  # confidence std growth > 20% → brittleness
    
    std_growth = (current_std - baseline_std) / baseline_std
    if std_growth > CONFIDENCE_STD_DRIFT_THRESHOLD:
        flags.append(f"confidence std grew {std_growth:.1%} — fragility signal")
    
    # mean confidence shifting toward the edges [<0.35 | >0.85] with movement > 0.15
    edge_shift = abs(current["confidence_mean"] - baseline_mean)
    if edge_shift > 0.15 and (mean < 0.35 or mean > 0.85):
        flags.append("confidence mean shifted toward edges")

    Why thresholds and not KS/PSI: on 20–50 samples the two-sample Kolmogorov–Smirnov test is underpowered and throws false alarms; thresholds on mean/pstdev are a cheap gate at that N. The price: you don't catch changes in shape (bimodality, a fattening tail) — coarse shifts in mean/spread show up, subtle ones don't. Upgrade path as the set grows: PSI or a two-sample KS on the length and confidence distributions.

  3. 3. Layer 3 — Human spot-check. Required before the first production deploy of a new type of output: a domain expert reads 10–20 real outputs with specific questions (any data the model shouldn't have access to; would an expert consider this correct; any adversarial exploitation).
  4. 4. Owner. One person signs off. If you can't name the owner before release — that's already the first red flag.

Artifact

Minimal harness for this checklist: a Python script that runs the regression set against a snapshot and compares distribution (Layer 1 + Layer 2). The system under test plugs in through a thin adapter — the harness knows nothing about the model, only the contract:

# adapter.run(input) -> {"output": str, "chunks": list, "confidence": float}
response = adapter.run(raw_input)

Wired as a release gate through exit codes, so CI can block the merge:

python run_eval.py --all --inputs inputs.jsonl
#   exit 0  — both layers passed, release allowed
#   exit 1  — regression failed → hard block (a known-bad came back)
#   exit 2  — distribution drift → softer: a review signal, not an auto-block

The split is deliberate: regression is "a known thing broke," an unambiguous stop; distribution drift is "behavior moved," a reason for a human, not an automatic rejection. The snapshot updates only via an explicit --update-snapshot — the baseline moves on purpose, not on its own.

Where it breaks

For whom and why

Risk: eval green, prod red — the standard story. Teams find out after release and fix it more expensively than if they'd caught it before.

Solution: a feature doesn't ship until three eval layers pass — not because it "looks good," but because it meets a criterion.

Metric: when the board asks "what's our AI quality story," you don't answer "we tested it" — you show the regression and distribution diff.

Closest to a production playbook: a practice-level artifact for Head of AI and technical CTO. Vibe-coding the harness turns the methodology into a reproducible code artifact — closing the code gap, not just the playbook.

About the author

Grigoriy Dobryakov — fractional / interim eng + AI leadership

Want AI release quality as a first-class engineering criterion?

Structured eval before every release that changes model behavior — regression, distribution check, human spot-check, named owner.

Discuss a fractional audit Email me

Other breakdowns

An engineering breakdown series: real task → methodology → working artifact → honest breakdown of where it fails.

Back to series →