{"id":194,"date":"2026-09-29T08:00:31","date_gmt":"2026-09-29T08:00:31","guid":{"rendered":"https:\/\/www.dobryakov.net\/blog\/194\/"},"modified":"2026-09-29T08:00:31","modified_gmt":"2026-09-29T08:00:31","slug":"ai-governance-agentic-governance","status":"publish","type":"post","link":"https:\/\/www.dobryakov.net\/blog\/194\/","title":{"rendered":"Agentic AI Governance: Bounded Autonomy, Policy Gates, and the Human in the Loop"},"content":{"rendered":"

When an AI system stops generating text and starts taking actions, the failure mode changes entirely. A hallucination stops being a bad reply and becomes an executed money transfer. An injection from an ingested email stops being a formatting glitch and becomes a privilege escalation. The agent's power and visibility become the attack surface. Per the 2026 Saviynt report, only 5% of security leaders are confident they could contain a compromised agent. The industry is building systems that act on the world before building the guardrails that bound those actions.<\/p>\n

<\/p>\n

This is the problem the agentic-systems engineer owns. You are not tuning a model for better answers. You are engineering an autonomy plane: letting an agent do useful work on its own while explicitly bounding the scope of that autonomy, requiring approval for irreversible actions, and intercepting dangerous behavior before it causes damage.<\/p>\n

The autonomy plane<\/h2>\n

Autonomy is not an on\/off switch. It is graduated by the action's risk class. The architectural pattern is a Policy-Gated Action Loop<\/strong>: every agent action passes through a deterministic policy gate that decides whether to auto-execute, require human confirmation, or deny.<\/p>\n

This is a direct application of out-of-band policy design. The safety of an action is decided outside the model<\/strong> \u2014 by a deterministic policy over provenance and action class that cannot be talked past with a prompt injection. The model proposes an action; a system the model cannot influence decides whether it runs.<\/p>\n

agent plan \u2192 tool selection\n      \u2502\n      \u25bc\n[classify action risk] \u2192 read | reversible-write | irreversible | financial\n      \u2502\n      \u25bc\n[OPA policy gate] \u2500\u2500 auto \u2500\u2500\u25ba execute (least-priv, short-lived creds)\n      \u251c\u2500\u2500 HITL \u2500\u2500\u25ba approval queue \u2500\u2500\u25ba (human) \u2500\u2500\u25ba execute | reject\n      \u2514\u2500\u2500 deny \u2500\u2500\u25ba block + audit\n      \u25b2\n[kill-switch flag] \u2500\u2500 interrupts the loop immediately<\/code><\/pre>\n
Engineering stack<\/h2>\n\n\n\n\n\n\n\n\n\n
Layer<\/th>\nComponent<\/th>\nRole<\/th>\n<\/tr>\n<\/thead>\n
Orchestration<\/td>\nLangGraph, CrewAI, or custom state-machine runner<\/td>\nExplicit states for agent execution<\/td>\n<\/tr>\n
Tool protocol<\/td>\nMCP (Model Context Protocol)<\/td>\nGoverned server access, scoped tools<\/td>\n<\/tr>\n
Policy<\/td>\nOPA (Rego)<\/td>\nDeterministic action gate<\/td>\n<\/tr>\n
Kill-switch<\/td>\nFeature flags<\/td>\nImmediate loop interruption<\/td>\n<\/tr>\n
HITL<\/td>\nApproval queue + audit trail<\/td>\nHuman confirmation for irreversible actions<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Explicit orchestration states are an architectural requirement, not a preference. If the agent's execution graph is implicit, you cannot insert a deterministic gate into the flow.<\/p>\n
Step 1. An action catalog with risk classes<\/h2>\n
Every tool action gets a risk class and a required execution mode. This catalog is the foundation; the policy gate enforces it.<\/p>\n
actions:\n  search_customer:   {risk: read,             mode: auto}\n  create_ticket:     {risk: reversible-write, mode: auto}\n  update_limit:      {risk: irreversible,     mode: hitl}\n  transfer_funds:    {risk: financial,        mode: hitl, second_approver: true}<\/code><\/pre>\n
The agent does on its own whatever is cheap to undo. Anything irreversible or expensive goes through a human. For financial actions, dual confirmation is mandatory.<\/p>\n
Step 2. The policy-gated loop<\/h2>\n
Before calling a tool, the agent asks OPA: is this allowed, for whom, under what conditions? The policy evaluates the action class, the amount, the customer, the time of day, and the data provenance. The verdict is logged to the audit trail.<\/p>\n
The gate evaluates provenance because the source of the request matters as much as the action itself. An action provoked by an untrusted source \u2014 an ingested email, a scraped webpage \u2014 gets denied regardless of whether the action class would normally auto-execute.<\/p>\n
Step 3. Least-privilege tools via MCP<\/h2>\n
MCP servers receive minimal scopes, short-lived revocable credentials, and an allow-list of tools per agent role. MCP servers are supply-chain components: they go into the AI Bill of Materials. An unmanaged MCP server with broad credentials is the same class of risk as an unmanaged SaaS integration \u2014 except the agent can call it at machine speed.<\/p>\n
Step 4. Human-in-the-loop for the irreversible<\/h2>\n
Irreversible and expensive actions go to an approval queue. Without explicit confirmation \u2014 two confirmations for financial actions \u2014 they do not execute. This is an explicit state transition in the orchestration graph, not "the model decided it was fine." The model proposes. The human disposes.<\/p>\n
Step 5. Memory governance<\/h2>\n
Agent memory requires isolation and validation. Poisoned memory is a mechanism for smuggling an injection across sessions: an attacker corrupts the agent's stored context in one interaction, and the agent acts on that corruption in the next, bypassing the real-time input filters. Memory entries need TTLs, write permissions, and validation on read. The OWASP Agentic Top 10 names memory poisoning explicitly because it is one of the few attack vectors that compounds over time.<\/p>\n
Step 6. Limits and the kill-switch<\/h2>\n
A per-task limit on steps and cost prevents runaway loops. Exceeding the limit triggers a stop and escalation. The kill-switch \u2014 implemented via feature flags \u2014 interrupts the loop immediately and freezes unfinished actions. Any agentic task must be stoppable, and its in-flight actions must be freezable. If you cannot freeze the queue, the kill-switch only stops the next action, not the one already in flight.<\/p>\n
Where it breaks<\/h2>\n
HITL fatigue.<\/strong> Too many confirmations and the human rubber-stamps "OK" without reading the context. Oversight becomes a formality. The fix is structural: HITL only for genuinely irreversible actions, batching low-stakes confirmations, and providing clear, concise context in each approval request. If the human has to read three paragraphs to approve a ticket creation, they will stop reading.<\/p>\n
Injection bypasses the gate through task logic.<\/strong> The agent is convinced the action is legitimate because an indirect injection in ingested data told it so. What saves you is not a guard model arguing with the agent. What saves you is a provenance policy: the action was provoked by an untrusted source, so the gate denies it regardless of the agent's conviction.<\/p>\n
Composition of safe actions produces a dangerous result.<\/strong> Each individual step is permitted by the gate. The chain is harmful. A single step's gate cannot see this \u2014 it needs task-level and session-level invariants that evaluate the cumulative effect of the action sequence, not just each action in isolation.<\/p>\n
Gate latency kills agent liveness.<\/strong> Every policy check adds latency. Synchronous checks slow the agent; reserve them for irreversible and financial actions: a gate that checks everything synchronously makes the agent too slow to be useful. The resolution is asynchronous pre-evaluation for read and reversible-write classes, reserving synchronous gates for irreversible and financial actions.<\/p>\n
Multi-agent accountability diffusion.<\/strong> In a swarm, accountability diffuses across agents. Tracing causality \u2014 which agent triggered which action, and why \u2014 becomes significantly harder. Stopping the swarm requires a kill-switch that propagates to all agents in the graph, not just the one exhibiting the anomaly.<\/p>\n
Standards mapping<\/h2>\n\n\n\n\n\n\n\n\n\n
Framework<\/th>\nProvisions<\/th>\n<\/tr>\n<\/thead>\n
OWASP Top 10 for Agentic Applications<\/td>\nExcessive agency, tool misuse, memory poisoning, hijacking via injection<\/td>\n<\/tr>\n
OWASP LLM<\/td>\nLLM06 (Excessive Agency), LLM01 (Prompt Injection)<\/td>\n<\/tr>\n
EU AI Act<\/td>\nArt. 14 (human oversight), Art. 15 (robustness)<\/td>\n<\/tr>\n
ISO\/IEC 42001<\/td>\nOperations and incident management for AI<\/td>\n<\/tr>\n
NIST AI RMF<\/td>\nManage (oversight, intervention)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
The EU AI Act Article 14 requirement is direct: a human must be able to intervene. The policy-gated loop is the engineering implementation of that legal requirement. Without the gate, the approval queue, and the kill-switch, "human oversight" is a policy document with no mechanism behind it.<\/p>\n
Maturity checklist<\/h2>\n
    \n
  • L1<\/strong>: An agent with an allow-list of tools and action logging.<\/li>\n
  • L2<\/strong>: A risk-class policy gate, HITL for the irreversible, step and budget limits, MCP with least privilege.<\/li>\n
  • L3<\/strong>: Memory governance, a kill-switch with drills, an out-of-band provenance policy, protection against compositional attacks, anti-rubber-stamp measures in oversight.<\/li>\n<\/ul>\n
    Most organizations deploying agents today are at L0 \u2014 an agent with tool access, no policy gate, and logging that runs only if someone remembers to turn it on. The gap between L0 and L2 is where the real engineering work lives.<\/p>\n
    Lab and artifact<\/h2>\n
    Give the agent four tools of different risk classes via MCP. Set up the OPA gate, an approval queue with dual confirmation for financial actions, step and budget limits, memory isolation, and the kill-switch. Run a red-team exercise: an indirect injection from an ingested email attempting to trigger update_limit<\/code> or transfer_funds<\/code>. The artifact is an action catalog with risk classes, Rego policy gates, a HITL log, and a red-team report mapped against the OWASP Agentic Top 10.<\/p>\n
    If the injection succeeds in triggering the financial action, the provenance policy is not working. If the human rubber-stamps the approval, the HITL context is not clear enough. Both failures are fixable. The one failure that is not fixable is discovering you needed these guardrails after the agent already executed the action.<\/p>\n
    \n
    An agent without bounded autonomy is a tool that acts on your infrastructure with your credentials at a speed you cannot match. The question is not whether the agent will attempt something destructive. The question is whether the policy gate stops it before the action leaves the queue \u2014 and whether you built the gate before giving the agent the keys.<\/p>\n","protected":false},"excerpt":{"rendered":"
    When an AI agent stops answering and starts acting, a wrong answer becomes an executed money transfer. Here is the engineering pattern for bounding agent autonomy.<\/p>\n","protected":false},"author":0,"featured_media":193,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[199,147],"tags":[200,201,202,27,204,203],"class_list":["post-194","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-agentic-ai","category-ai-governance","tag-agentic-ai","tag-bounded-autonomy","tag-human-in-the-loop","tag-mcp","tag-opa","tag-policy-engine"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t