Chapter 9. Agentic AI Governance: autonomy, tools, human-in-the-loop | Grigoriy Dobryakov

Grigoriy Dobryakov

Course · Enterprise AI Governance Architecture

Home ← Back to course
Chapter 9AI Governance course

Chapter 9. Agentic AI Governance: autonomy, tools, human-in-the-loop

Up to this chapter, Kovcheg mostly *answered*. Now it *acts*: creates tickets, drafts and sends customer decisions, calls internal APIs, reaches into systems via MCP servers. This is a qualitative jump in risk. While the model generates text, the worst case is a wrong answer (ch. 3, 7). Once the model takes actions, its "power and visibility become the attack surface" (in OWASP Agentic's phrasing): an injection from an email (ch. 3) stops being a bad reply and becomes an executed money transfer.

And here's the uncomfortable part: the industry isn't ready. Per the 2026 Saviynt report, only 5% of security leaders are confident they could contain a compromised agent (ch. 8). This chapter covers the autonomy plane: how to let an agent do useful work on its own while explicitly bounding the scope of that autonomy, requiring approval for irreversible actions, and intercepting dangerous behavior before it causes damage.

Privacy / ZDR1 · Privacy / ZDR Access control2 · Access control Threats3 · Threats Audit4 · Audit FinOps5 · FinOps Regulation6 · Regulation Quality7 · Quality Supply chain8 · Supply chain Agentic9 · Agentic Operating model10 · Operating model

The customer's business goal

Kovcheg's autonomy is graduated by action risk. Promises to the business:

  1. The agent does on its own whatever is cheap to undo; anything irreversible/expensive goes through a human.
  2. An injection or hallucination never turns into an executed harmful action.
  3. Any agentic task can be stopped, and its unfinished actions frozen.

Driver: threat or regulator

Architectural pattern

Bounded Autonomy & Policy-Gated Action Loop — every agent action passes through a deterministic policy gate that decides: auto-execute / require human confirmation / deny. Autonomy isn't an on/off switch — it's graduated by the action's risk class. This is a direct application of the ch. 3 finding (out-of-band policy, CaMeL): the safety of an action is decided outside the model — by a deterministic policy over provenance and action class that can't be talked past with an injection.

agent plan → tool selection
      │
      ▼
[classify action risk] → read | reversible-write | irreversible | financial
      │
      ▼
[OPA policy gate] ── auto ──► execute (least-priv, short-lived creds)
      ├── HITL ──► approval queue ──► (human) ──► execute | reject
      └── deny ──► block + audit (ch. 4)
      ▲
[kill-switch flag] ── interrupts the loop immediately (ch. 8)

Engineering stack & providers

Engineering implementation

### Step 1. An action catalog with risk classes

Every tool action gets a class and a required mode:

actions:
  search_customer:   {risk: read,             mode: auto}
  create_ticket:     {risk: reversible-write, mode: auto}
  update_limit:      {risk: irreversible,     mode: hitl}
  transfer_funds:    {risk: financial,        mode: hitl, second_approver: true}

### Step 2. The policy-gated loop

Before calling a tool, the agent asks OPA: is this allowed, for whom, under what conditions (amount, customer, time, data provenance from ch. 3). The verdict is logged (ch. 4).

### Step 3. Least-privilege tools via MCP

MCP servers with minimal scopes, short-lived revocable credentials, an allow-list of tools per agent role. MCP servers go into the AIBOM (ch. 8) — they're supply-chain components.

### Step 4. Human-in-the-loop for the irreversible

Irreversible/expensive actions go to an approval queue; without confirmation (two, for financial actions) they don't execute. This is an explicit state transition, not "the model decided it was fine."

### Step 5. Memory governance

Isolation and validation of agent memory (defense against memory poisoning per OWASP Agentic), TTLs and write permissions. Poisoned memory is a way to smuggle an injection across sessions.

### Step 6. Limits and the kill-switch

A per-task limit on steps/cost (ties to ch. 5); exceeding it triggers a stop and escalation. The kill-switch (ch. 8) interrupts the loop immediately and freezes unfinished actions.

Where it breaks

Standards and mapping

Lab and artifact

Give the Kovcheg agent 4 tools of different risk classes (from the catalog above) via MCP; set up the OPA gate, an approval queue with dual confirmation for financial actions, step/budget limits, memory isolation, and the kill-switch; run a red-team exercise: an indirect injection from an email (the ch. 3 case) trying to trigger update_limit/transfer_funds. Artifact: an action catalog with risk classes + rego gates + an HITL log + a red-team report against OWASP Agentic.

Maturity checklist

Sources

In practice

How it actually works — engineering breakdowns

Standalone howto from practice, showing this control plane on real code and a working artifact.

Read next

Putting AI into production under regulatory risk?

Designing the control plane for your system: privacy, access, guardrails, audit, EU AI Act / ISO 42001 compliance — as working architecture, not a policy PDF.

Email me

The transition engine

Next Move Engine — the system that takes a team to an autonomous delivery loop.

Next Move Engine →