{"id":76,"date":"2026-07-26T19:10:34","date_gmt":"2026-07-26T19:10:34","guid":{"rendered":"https:\/\/www.dobryakov.net\/blog\/76\/"},"modified":"2026-07-26T19:10:34","modified_gmt":"2026-07-26T19:10:34","slug":"eval-harness-agent-drift","status":"publish","type":"post","link":"https:\/\/www.dobryakov.net\/blog\/76\/","title":{"rendered":"How to make eval a release criterion for AI features instead of a demo-grade approval"},"content":{"rendered":"<p>Treat AI-output quality as a <strong>release criterion<\/strong>: a fixed regression set built from past incidents, a distribution diff against the previous release snapshot (20\u201350 real inputs), a human spot-check before the first production deploy of a new output type \u2014 and one person who signs off. Below: the demo-grade anti-pattern, the three-layer method, and a minimal harness that fails CI with a readable exit code.<\/p>\n<p><!--more--><\/p>\n<p>Most AI features ship in <strong>demo grade<\/strong>. Not because teams do not care \u2014 because the eval that approved the feature used the same inputs as the pitch. Curated dataset. Controlled conditions. The long tail of real production traffic never ran.<\/p>\n<p>Then production shows what the demo missed:<\/p>\n<ul>\n<li>a RAG system that retrieves correctly on a staging corpus and hallucinates on the full corpus with stale entries;<\/li>\n<li>a classifier that silently degrades after an upstream input-schema change;<\/li>\n<li>AI-assisted code that passes automated tests but introduces a security pattern the tests never covered.<\/li>\n<\/ul>\n<p>Three stacks. One failure mode: no structured eval, no named owner, approval by \u201clooked fine in the demo.\u201d<\/p>\n<p>Agent drift is when model or agent behavior leaves what you treated as \u201cverified\u201d \u2014 without a red alert at release time. Sometimes the prompt or corpus changed. Sometimes the input distribution shifted quietly. The symptom is the same: green eval on the demo set, red production on the live tail.<\/p>\n<h2>When AI-output review is actually required<\/h2>\n<p>Not every deploy. The trigger is simple: <strong>does this release change what the model sees, or what it outputs?<\/strong><\/p>\n<p>Review before release when any of the following is true:<\/p>\n<ul>\n<li>the output is used in a customer-facing flow (response, recommendation, classification, search);<\/li>\n<li>the output influences a business-critical decision (approval, routing, pricing, alert);<\/li>\n<li>the output is built from retrieval (RAG) \u2014 hallucination risk scales with retrieval quality;<\/li>\n<li>the model, prompt, or retrieval corpus changed since the last release.<\/li>\n<\/ul>\n<p>Infrastructure-only deploys, pure UI, config that does not touch prompt\/retrieval \u2014 a full eval is optional. Saving cost here is fine; self-deception is when \u201cinfra\u201d quietly swaps the index or the system prompt.<\/p>\n<h2>Ownership: one person, not \u201cthe team\u201d<\/h2>\n<p>One person owns the eval. Not \u201ceveryone.\u201d Not \u201cit looked fine in staging.\u201d<\/p>\n<p>Typical pattern:<\/p>\n<table>\n<thead>\n<tr>\n<th>Role<\/th>\n<th>Responsibility<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Feature owner (engineer or PM)<\/td>\n<td>defines \u201cgood output\u201d in business terms<\/td>\n<\/tr>\n<tr>\n<td>ML\/AI lead<\/td>\n<td>validates methodology and sign-off criteria<\/td>\n<\/tr>\n<tr>\n<td>Release approver (EM \/ tech lead)<\/td>\n<td>final gate; sign-off cannot be replaced by \u201clooked fine in staging\u201d<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>If you cannot name the owner before release, that is already the first risk signal. The later argument about \u201cwho should have caught it\u201d is almost guaranteed.<\/p>\n<h2>Three layers of a minimal eval<\/h2>\n<h3>Layer 1 \u2014 Regression (non-negotiable)<\/h3>\n<p>A fixed set of reference cases with known expected outputs. The set must include:<\/p>\n<ul>\n<li>edge cases from previous incidents;<\/li>\n<li>cases from the demo that were used to approve the feature (yes \u2014 run them, but <strong>not only<\/strong> them);<\/li>\n<li>at least one adversarial input per output type: prompt injection, empty input, malformed input.<\/li>\n<\/ul>\n<p>Pass criterion: every reference case stays inside the defined acceptable range. Any new failure blocks release.<\/p>\n<p>The key: a case is born from an incident. The <code>origin<\/code> field in YAML is not bureaucracy \u2014 it is what separates a regression set from a demo dataset. Inputs come from what already broke in production, not from a board slide.<\/p>\n<p>Example from the bundled <code>rag-stale-corpus<\/code> case (eval-harness repo):<\/p>\n<pre><code class=\"language-yaml\">id: rag-stale-corpus\norigin: incident-2026-03\ninput:\n  query: &quot;\u0434\u0435\u0439\u0441\u0442\u0432\u0443\u044e\u0449\u0430\u044f \u0441\u0442\u0430\u0432\u043a\u0430 \u043f\u043e \u0442\u0430\u0440\u0438\u0444\u0443 X&quot;\nexpected:\n  must_contain:\n    - &quot;\u0430\u043a\u0442\u0443\u0430\u043b\u044c\u043d\u044b\u0439 \u0434\u043e\u043a\u0443\u043c\u0435\u043d\u0442&quot;\n  must_not_contain:\n    - &quot;archived&quot;\n  retrieved_chunks:\n    min_count: 1\n    must_contain:\n      - &#x27;&quot;status&quot;: &quot;active&quot;&#x27;\n    must_not_contain:\n      - &#x27;&quot;status&quot;: &quot;archived&quot;&#x27;\npass_criteria:\n  - no_archived_docs\n  - answer_grounded<\/code><\/pre>\n<p><code>must_contain<\/code> \/ <code>must_not_contain<\/code> check the answer text. <code>retrieved_chunks<\/code> check serialized chunks (often JSON). Needles must match your <strong>real<\/strong> RAG format: a log-style <code>&quot;status: archived&quot;<\/code> will not hit <code>{&quot;status&quot;: &quot;archived&quot;}<\/code>, and the test will pass vacuously.<\/p>\n<p>The same repo ships adversarial cases: empty, malformed, injection (<code>ignore previous instructions\u2026<\/code>). Their job is to lock in a refusal \u2014 not a \u201cnice answer to an attack.\u201d<\/p>\n<h3>Layer 2 \u2014 Distribution check<\/h3>\n<p>For RAG and classifiers. Take 20\u201350 recent real inputs, run the new version, and compare not to an \u201cideal,\u201d but to a <strong>diff<\/strong> against the previous release snapshot:<\/p>\n<ul>\n<li>output length \/ format \u2014 outliers often mean a broken prompt or schema change;<\/li>\n<li>retrieved chunks \u2014 did they change, and why;<\/li>\n<li>confidence distribution \u2014 a shift toward the edges means brittleness.<\/li>\n<\/ul>\n<p>In eval-harness the drift flags are explicit:<\/p>\n<table>\n<thead>\n<tr>\n<th>Signal<\/th>\n<th>Threshold<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>mean <code>output<\/code> length vs snapshot<\/td>\n<td>shift &gt; 30%<\/td>\n<\/tr>\n<tr>\n<td><code>confidence<\/code> std<\/td>\n<td>increase &gt; 20%<\/td>\n<\/tr>\n<tr>\n<td>mean <code>chunks<\/code> count<\/td>\n<td>shift &gt; 30%<\/td>\n<\/tr>\n<tr>\n<td>mean confidence toward edges<\/td>\n<td>&lt; 0.35 or &gt; 0.85 with a clear baseline delta<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>These are not \u201cscientific truth.\u201d They are a working gate: if the distribution moved without an explanation, the release stops until someone owns the reason.<\/p>\n<p>After an accepted ship, update the snapshot separately (<code>--update-snapshot<\/code>). Do not pass that flag in the release pipeline \u2014 otherwise the baseline quietly absorbs the degradation.<\/p>\n<h3>Layer 3 \u2014 Human spot-check<\/h3>\n<p>Process, not code. Mandatory before the first production deploy of any new output type: a domain person reads 10\u201320 real outputs with specific questions:<\/p>\n<ul>\n<li>does the output contain information the model should not have access to;<\/li>\n<li>would a domain expert consider it correct;<\/li>\n<li>is there adversarial misuse sitting in plain sight.<\/li>\n<\/ul>\n<p>On later releases, trigger spot-check on model, prompt, or corpus change, or after an incident. Without this layer you automate blindness: regression green, distribution \u201cin band,\u201d meaning already garbage.<\/p>\n<h2>The demo-grade anti-pattern<\/h2>\n<p><strong>Demo grade:<\/strong> output that looks correct under controlled conditions and fails on the real input distribution.<\/p>\n<p>How it usually ships:<\/p>\n<ol>\n<li>the feature is built and \u201ctested\u201d against the brief examples;<\/li>\n<li>those examples were written to show capability, not to stress-test;<\/li>\n<li>staging runs on a curated or synthetic dataset;<\/li>\n<li>a reviewer approves on \u201clooks reasonable\u201d;<\/li>\n<li>production opens the long tail the demo never saw.<\/li>\n<\/ol>\n<p>Symptoms you are shipping demo-grade:<\/p>\n<ul>\n<li>eval inputs = pitch or sprint-review examples;<\/li>\n<li>nobody ran the feature on real production inputs before release;<\/li>\n<li>the \u201ceval\u201d is a stakeholder demo, not a measurement;<\/li>\n<li>pass criteria are written <strong>after<\/strong> the run (\u201cyeah, seems fine\u201d).<\/li>\n<\/ul>\n<p>The fix is short and uncomfortable: <strong>demo dataset \u2260 eval dataset<\/strong>. The demo shows the best case. The eval measures the real distribution.<\/p>\n<h2>Artifact: eval-harness in CI<\/h2>\n<p>A minimal harness for Layer 1 + Layer 2: <a href=\"https:\/\/github.com\/dobryakov\/eval-harness\">github.com\/dobryakov\/eval-harness<\/a>. Python, YAML cases, JSONL inputs, snapshot, adapter interface.<\/p>\n<p>Exit codes so the gate is machine-readable:<\/p>\n<table>\n<thead>\n<tr>\n<th>Exit code<\/th>\n<th>Meaning<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><code>0<\/code><\/td>\n<td>ship<\/td>\n<\/tr>\n<tr>\n<td><code>1<\/code><\/td>\n<td>Layer 1 failed (regression)<\/td>\n<\/tr>\n<tr>\n<td><code>2<\/code><\/td>\n<td>Layer 2 failed (distribution drift)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Quick start:<\/p>\n<pre><code class=\"language-bash\">pip install -r requirements.txt\n\npython run_eval.py --layer regression\npython run_eval.py --layer distribution --inputs inputs.jsonl\npython run_eval.py --all --inputs inputs.jsonl<\/code><\/pre>\n<p>Wire your model \/ agent through <code>adapter.py<\/code>:<\/p>\n<pre><code class=\"language-python\">def run(input: dict) -&gt; dict:\n    return {\n        &quot;output&quot;: str,        # answer text\n        &quot;chunks&quot;: list,       # retrieved chunks\n        &quot;confidence&quot;: float,  # 0..1\n    }<\/code><\/pre>\n<p>In CI:<\/p>\n<pre><code class=\"language-yaml\">- run: pip install -r requirements.txt\n- run: python run_eval.py --all --inputs inputs.jsonl<\/code><\/pre>\n<p>Every incident that must not recur becomes a permanent regression case under <code>cases\/<\/code>. Otherwise you \u201cfixed\u201d it once and wait for the same bug under a different prompt.<\/p>\n<p>Repo layout:<\/p>\n<pre><code>eval-harness\/\n  run_eval.py\n  adapter.py\n  layers\/\n    regression.py\n    distribution.py\n  cases\/\n    *.yaml\n  snapshots\/\n    latest.json\n  inputs.jsonl<\/code><\/pre>\n<p>Layer 3 stays out of the code \u2014 correctly: a human spot-check cannot be closed by a green script checkbox.<\/p>\n<h2>Checklist before a release that changes AI output<\/h2>\n<ul>\n<li>[ ] Owner named: who signs off on AI-output quality<\/li>\n<li>[ ] Regression set run: all reference cases pass<\/li>\n<li>[ ] No new failure modes vs the previous release<\/li>\n<li>[ ] Distribution check done (for RAG \/ classifiers): no unexplained shift<\/li>\n<li>[ ] Human spot-check done if: new capability \/ model change \/ prompt change \/ prior incident<\/li>\n<li>[ ] Eval dataset \u2260 demo dataset<\/li>\n<li>[ ] Pass criteria defined <strong>before<\/strong> the run, not after<\/li>\n<\/ul>\n<h2>Limits of the method<\/h2>\n<ul>\n<li><strong>Distribution thresholds are heuristics.<\/strong> 30% \/ 20% catch coarse drift; subtle semantic degradation with stable length can slip past Layer 2 \u2014 that is why Layer 3 remains.<\/li>\n<li><strong>Empty <code>origin<\/code> and \u201cpretty\u201d cases.<\/strong> If regression is built from showcase examples, you get demo-grade again \u2014 only in YAML.<\/li>\n<li><strong>Vacuous pass on chunks.<\/strong> A wrong needle format in <code>retrieved_chunks<\/code> yields a false green. Serialize a real chunk the same way the adapter does, then write the case.<\/li>\n<li><strong>No owner \u2014 no gate.<\/strong> A harness in CI without someone accountable for a red result becomes noise people learn to bypass (<code>continue-on-error: true<\/code>).<\/li>\n<li><strong>Snapshot after degradation.<\/strong> <code>--update-snapshot<\/code> at the moment of \u201cfine, this is the new normal\u201d locks bad in as baseline.<\/li>\n<li><strong>Not every deploy.<\/strong> Mandatory eval on every hotfix kills adoption; keep the trigger \u201cdoes this change model input\/output.\u201d<\/li>\n<\/ul>\n<h2>Who this is for<\/h2>\n<p>For CTOs, Heads of AI, and tech leads who already have pilots, and whose board question is no longer \u201cdo we have AI\u201d but \u201cwhat is our AI quality bar.\u201d \u201cWe tested\u201d is a weak answer. \u201cHere is the regression set and the distribution diff vs last release\u201d is a working one.<\/p>\n<p>Neighboring layers of the same discipline: <a href=\"https:\/\/www.dobryakov.net\/blog\/63\/\">agent perimeter without relying on the system prompt<\/a> and <a href=\"https:\/\/www.dobryakov.net\/blog\/65\/\">skill onboarding as governance<\/a>. Eval gates model output; those gate access and the introduction of new assistant behavior.<\/p>\n<h2>Bottom line<\/h2>\n<p>If your eval matches the pitch, you are not checking production \u2014 you are checking a presentation. Split demo from eval, build regression from incidents, measure distribution diff, name one owner, and do not ship until all three layers pass on criteria \u2014 not because it \u201clooks good.\u201d<\/p>\n<p>Repo: <a href=\"https:\/\/github.com\/dobryakov\/eval-harness\">github.com\/dobryakov\/eval-harness<\/a><\/p>\n<p>Howto: <a href=\"https:\/\/www.dobryakov.net\/howto\/eval-harness-agent-drift.html\">dobryakov.net\/howto\/eval-harness-agent-drift.html<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Demo-grade eval greenlights a feature on the same examples as the pitch. Production then shows the long tail. Three layers \u2014 incident-born regression, distribution diff vs the last snapshot, human spot-check \u2014 plus one named owner. Minimal harness: github.com\/dobryakov\/eval-harness.<\/p>\n","protected":false},"author":0,"featured_media":75,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[24,50],"tags":[52,55,54,29,51,46,53,49],"class_list":["post-76","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-engineering","category-mlops","tag-agent-drift","tag-ci","tag-demo-grade","tag-enterprise","tag-eval","tag-rag","tag-release-gate","tag-vibe-coding"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 4.9.10 - aioseo.com -->\n\t<meta name=\"description\" content=\"Eval as a release gate: incident-born regression, distribution diff vs snapshot, human spot-check. CI exit codes. Example: eval-harness.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<link rel=\"canonical\" href=\"https:\/\/www.dobryakov.net\/blog\/76\/\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 4.9.10\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"Grigoriy Dobryakov - IT+AI Blog - Grigoriy Dobryakov&#039;s blog: management, development and testing\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"How to make eval a release criterion for AI features instead of a demo-grade approval\" \/>\n\t\t<meta property=\"og:description\" content=\"Eval as a release gate: incident-born regression, distribution diff vs snapshot, human spot-check. CI exit codes. Example: eval-harness.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/www.dobryakov.net\/blog\/76\/\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2026-07-26T19:10:34+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2026-07-26T19:10:34+00:00\" \/>\n\t\t<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n\t\t<meta name=\"twitter:title\" content=\"How to make eval a release criterion for AI features instead of a demo-grade approval\" \/>\n\t\t<meta name=\"twitter:description\" content=\"Eval as a release gate: incident-born regression, distribution diff vs snapshot, human spot-check. CI exit codes. Example: eval-harness.\" \/>\n\t\t<script type=\"application\/ld+json\" class=\"aioseo-schema\">\n\t\t\t{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"BlogPosting\",\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/76\\\/#blogposting\",\"name\":\"How to make eval a release criterion for AI features instead of a demo-grade approval\",\"headline\":\"How to make eval a release criterion for AI features instead of a demo-grade approval\",\"author\":{\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/author\\\/#author\"},\"publisher\":{\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/#organization\"},\"image\":{\"@type\":\"ImageObject\",\"url\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/eval-harness-agent-drift.jpg\",\"width\":1200,\"height\":630,\"caption\":\"How to make eval a release criterion for AI features instead of a demo-grade approval\"},\"datePublished\":\"2026-07-26T19:10:34+00:00\",\"dateModified\":\"2026-07-26T19:10:34+00:00\",\"inLanguage\":\"en-US\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/76\\\/#webpage\"},\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/76\\\/#webpage\"},\"articleSection\":\"AI Engineering, MLOps, agent drift, CI, demo-grade, enterprise, eval, RAG, release gate, vibe-coding\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/76\\\/#breadcrumblist\",\"itemListElement\":[{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog#listItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\",\"nextItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/category\\\/ai-engineering\\\/#listItem\",\"name\":\"AI Engineering\"}},{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/category\\\/ai-engineering\\\/#listItem\",\"position\":2,\"name\":\"AI Engineering\",\"item\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/category\\\/ai-engineering\\\/\",\"nextItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/76\\\/#listItem\",\"name\":\"How to make eval a release criterion for AI features instead of a demo-grade approval\"},\"previousItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog#listItem\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/76\\\/#listItem\",\"position\":3,\"name\":\"How to make eval a release criterion for AI features instead of a demo-grade approval\",\"previousItem\":{\"@type\":\"ListItem\",\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/category\\\/ai-engineering\\\/#listItem\",\"name\":\"AI Engineering\"}}]},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/#organization\",\"name\":\"Grigoriy Dobryakov - IT+AI Blog\",\"description\":\"Grigoriy Dobryakov's blog: management, development and testing\",\"url\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/76\\\/#webpage\",\"url\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/76\\\/\",\"name\":\"How to make eval a release criterion for AI features instead of a demo-grade approval\",\"description\":\"Eval as a release gate: incident-born regression, distribution diff vs snapshot, human spot-check. CI exit codes. Example: eval-harness.\",\"inLanguage\":\"en-US\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/#website\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/76\\\/#breadcrumblist\"},\"author\":{\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/author\\\/#author\"},\"creator\":{\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/author\\\/#author\"},\"image\":{\"@type\":\"ImageObject\",\"url\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/07\\\/eval-harness-agent-drift.jpg\",\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/76\\\/#mainImage\",\"width\":1200,\"height\":630,\"caption\":\"How to make eval a release criterion for AI features instead of a demo-grade approval\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/76\\\/#mainImage\"},\"datePublished\":\"2026-07-26T19:10:34+00:00\",\"dateModified\":\"2026-07-26T19:10:34+00:00\"},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/\",\"name\":\"Grigoriy Dobryakov - IT+AI Blog\",\"description\":\"Grigoriy Dobryakov's blog: management, development and testing\",\"inLanguage\":\"en-US\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.dobryakov.net\\\/blog\\\/#organization\"}}]}\n\t\t<\/script>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"How to make eval a release criterion for AI features instead of a demo-grade approval","description":"Eval as a release gate: incident-born regression, distribution diff vs snapshot, human spot-check. CI exit codes. Example: eval-harness.","canonical_url":"https:\/\/www.dobryakov.net\/blog\/76\/","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"BlogPosting","@id":"https:\/\/www.dobryakov.net\/blog\/76\/#blogposting","name":"How to make eval a release criterion for AI features instead of a demo-grade approval","headline":"How to make eval a release criterion for AI features instead of a demo-grade approval","author":{"@id":"https:\/\/www.dobryakov.net\/blog\/author\/#author"},"publisher":{"@id":"https:\/\/www.dobryakov.net\/blog\/#organization"},"image":{"@type":"ImageObject","url":"https:\/\/www.dobryakov.net\/blog\/wp-content\/uploads\/2026\/07\/eval-harness-agent-drift.jpg","width":1200,"height":630,"caption":"How to make eval a release criterion for AI features instead of a demo-grade approval"},"datePublished":"2026-07-26T19:10:34+00:00","dateModified":"2026-07-26T19:10:34+00:00","inLanguage":"en-US","mainEntityOfPage":{"@id":"https:\/\/www.dobryakov.net\/blog\/76\/#webpage"},"isPartOf":{"@id":"https:\/\/www.dobryakov.net\/blog\/76\/#webpage"},"articleSection":"AI Engineering, MLOps, agent drift, CI, demo-grade, enterprise, eval, RAG, release gate, vibe-coding"},{"@type":"BreadcrumbList","@id":"https:\/\/www.dobryakov.net\/blog\/76\/#breadcrumblist","itemListElement":[{"@type":"ListItem","@id":"https:\/\/www.dobryakov.net\/blog#listItem","position":1,"name":"Home","item":"https:\/\/www.dobryakov.net\/blog","nextItem":{"@type":"ListItem","@id":"https:\/\/www.dobryakov.net\/blog\/category\/ai-engineering\/#listItem","name":"AI Engineering"}},{"@type":"ListItem","@id":"https:\/\/www.dobryakov.net\/blog\/category\/ai-engineering\/#listItem","position":2,"name":"AI Engineering","item":"https:\/\/www.dobryakov.net\/blog\/category\/ai-engineering\/","nextItem":{"@type":"ListItem","@id":"https:\/\/www.dobryakov.net\/blog\/76\/#listItem","name":"How to make eval a release criterion for AI features instead of a demo-grade approval"},"previousItem":{"@type":"ListItem","@id":"https:\/\/www.dobryakov.net\/blog#listItem","name":"Home"}},{"@type":"ListItem","@id":"https:\/\/www.dobryakov.net\/blog\/76\/#listItem","position":3,"name":"How to make eval a release criterion for AI features instead of a demo-grade approval","previousItem":{"@type":"ListItem","@id":"https:\/\/www.dobryakov.net\/blog\/category\/ai-engineering\/#listItem","name":"AI Engineering"}}]},{"@type":"Organization","@id":"https:\/\/www.dobryakov.net\/blog\/#organization","name":"Grigoriy Dobryakov - IT+AI Blog","description":"Grigoriy Dobryakov's blog: management, development and testing","url":"https:\/\/www.dobryakov.net\/blog\/"},{"@type":"WebPage","@id":"https:\/\/www.dobryakov.net\/blog\/76\/#webpage","url":"https:\/\/www.dobryakov.net\/blog\/76\/","name":"How to make eval a release criterion for AI features instead of a demo-grade approval","description":"Eval as a release gate: incident-born regression, distribution diff vs snapshot, human spot-check. CI exit codes. Example: eval-harness.","inLanguage":"en-US","isPartOf":{"@id":"https:\/\/www.dobryakov.net\/blog\/#website"},"breadcrumb":{"@id":"https:\/\/www.dobryakov.net\/blog\/76\/#breadcrumblist"},"author":{"@id":"https:\/\/www.dobryakov.net\/blog\/author\/#author"},"creator":{"@id":"https:\/\/www.dobryakov.net\/blog\/author\/#author"},"image":{"@type":"ImageObject","url":"https:\/\/www.dobryakov.net\/blog\/wp-content\/uploads\/2026\/07\/eval-harness-agent-drift.jpg","@id":"https:\/\/www.dobryakov.net\/blog\/76\/#mainImage","width":1200,"height":630,"caption":"How to make eval a release criterion for AI features instead of a demo-grade approval"},"primaryImageOfPage":{"@id":"https:\/\/www.dobryakov.net\/blog\/76\/#mainImage"},"datePublished":"2026-07-26T19:10:34+00:00","dateModified":"2026-07-26T19:10:34+00:00"},{"@type":"WebSite","@id":"https:\/\/www.dobryakov.net\/blog\/#website","url":"https:\/\/www.dobryakov.net\/blog\/","name":"Grigoriy Dobryakov - IT+AI Blog","description":"Grigoriy Dobryakov's blog: management, development and testing","inLanguage":"en-US","publisher":{"@id":"https:\/\/www.dobryakov.net\/blog\/#organization"}}]},"og:locale":"en_US","og:site_name":"Grigoriy Dobryakov - IT+AI Blog - Grigoriy Dobryakov's blog: management, development and testing","og:type":"article","og:title":"How to make eval a release criterion for AI features instead of a demo-grade approval","og:description":"Eval as a release gate: incident-born regression, distribution diff vs snapshot, human spot-check. CI exit codes. Example: eval-harness.","og:url":"https:\/\/www.dobryakov.net\/blog\/76\/","article:published_time":"2026-07-26T19:10:34+00:00","article:modified_time":"2026-07-26T19:10:34+00:00","twitter:card":"summary_large_image","twitter:title":"How to make eval a release criterion for AI features instead of a demo-grade approval","twitter:description":"Eval as a release gate: incident-born regression, distribution diff vs snapshot, human spot-check. CI exit codes. Example: eval-harness."},"aioseo_meta_data":{"post_id":"76","title":"How to make eval a release criterion for AI features instead of a demo-grade approval","description":"Eval as a release gate: incident-born regression, distribution diff vs snapshot, human spot-check. CI exit codes. Example: eval-harness.","keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":"How to make eval a release criterion for AI features instead of a demo-grade approval","og_description":"Eval as a release gate: incident-born regression, distribution diff vs snapshot, human spot-check. CI exit codes. Example: eval-harness.","og_object_type":"default","og_image_type":"default","og_image_custom_url":null,"og_image_custom_fields":null,"og_image_url":null,"og_image_width":null,"og_image_height":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_image_url":null,"twitter_title":"How to make eval a release criterion for AI features instead of a demo-grade approval","twitter_description":"Eval as a release gate: incident-born regression, distribution diff vs snapshot, human spot-check. CI exit codes. Example: eval-harness.","schema_type":"default","schema_type_options":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"limit_modified_date":false,"ai":null,"breadcrumb_settings":null,"seo_analyzer_scan_date":null,"created":"2026-07-26 19:10:58","updated":"2026-07-26 19:16:07"},"aioseo_breadcrumb":"<div class=\"aioseo-breadcrumbs\"><span class=\"aioseo-breadcrumb\">\n\t\t\t<a href=\"https:\/\/www.dobryakov.net\/blog\" title=\"Home\">Home<\/a>\n\t\t<\/span><span class=\"aioseo-breadcrumb-separator\">&raquo;<\/span><span class=\"aioseo-breadcrumb\">\n\t\t\t<a href=\"https:\/\/www.dobryakov.net\/blog\/category\/ai-engineering\/\" title=\"AI Engineering\">AI Engineering<\/a>\n\t\t<\/span><span class=\"aioseo-breadcrumb-separator\">&raquo;<\/span><span class=\"aioseo-breadcrumb\">\n\t\t\tHow to make eval a release criterion for AI features instead of a demo-grade approval\n\t\t<\/span><\/div>","aioseo_breadcrumb_json":[{"label":"Home","link":"https:\/\/www.dobryakov.net\/blog"},{"label":"AI Engineering","link":"https:\/\/www.dobryakov.net\/blog\/category\/ai-engineering\/"},{"label":"How to make eval a release criterion for AI features instead of a demo-grade approval","link":"https:\/\/www.dobryakov.net\/blog\/76\/"}],"_links":{"self":[{"href":"https:\/\/www.dobryakov.net\/blog\/wp-json\/wp\/v2\/posts\/76","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dobryakov.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dobryakov.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dobryakov.net\/blog\/wp-json\/wp\/v2\/comments?post=76"}],"version-history":[{"count":0,"href":"https:\/\/www.dobryakov.net\/blog\/wp-json\/wp\/v2\/posts\/76\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.dobryakov.net\/blog\/wp-json\/wp\/v2\/media\/75"}],"wp:attachment":[{"href":"https:\/\/www.dobryakov.net\/blog\/wp-json\/wp\/v2\/media?parent=76"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dobryakov.net\/blog\/wp-json\/wp\/v2\/categories?post=76"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dobryakov.net\/blog\/wp-json\/wp\/v2\/tags?post=76"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}