Chapter 4. Burden II: human oversight, accuracy, logs, post-market monitoring

The situation at Kompas

Kompas's product pride is seamlessness: the recruiter opens a role and immediately sees a ready ranked list of candidates, ready to work. It is precisely this seamlessness that collides with the human oversight requirement. The law asks: where is the human who actually understands what the rank is based on and can override it — rather than just clicking "accept" on what they were shown?

The second discovery of this chapter is that the obligations don't end at release. Kompas is already in production, and it turns out the provider must monitor the system in operation, log its behavior and report serious incidents to the regulator. Compliance went from a point ("shipped — passed") to a process ("shipped — and now we keep it up").

What the rule says

Human oversight (Art. 14). A high-risk system is designed so that it can be effectively overseen by a human: the human must understand the system's capabilities and limitations, notice its failures, not over-rely on its output (protection against automation bias — blind trust in the machine) and be able to intervene or override the result. The key: it is not "a human is present nearby" but a designed capability for real intervention. An "approve" button clicked without looking does not satisfy the requirement.

Accuracy, robustness, cybersecurity (Art. 15). The system must reach an adequate level of accuracy (with stated metrics), be resilient to errors and failures, and protected against attacks, including AI-specific ones (adversarial manipulation, data poisoning). For scoring people, a subtle question arises: the accuracy of the ranking is a metric of what, exactly, and how it relates to fairness.

Record-keeping / logs (Art. 12). The system automatically keeps event logs over its service life — for traceability, incident investigation and proof of operation within bounds. For Kompas this is an immutable trace of every scoring decision.

Post-market monitoring (Art. 72). The provider builds a system for observing the system's behavior in real operation — actively collecting and analyzing data about how it works, rather than waiting for complaints.

Serious incident reporting (Art. 73). Serious incidents must be reported by the provider to the supervisory authority within set deadlines. What counts as "serious" and how quickly to report is part of a process you have to build in advance.

How it lands on the product

Kompas rebuilds the recruiter UX: instead of "approve the list" — visible grounds for the score (why the candidate is in this rank), the ability to override the rank and log the override. That is meaningful oversight: the human doesn't rubber-stamp the machine's output but works with it, and the system records it.

The scoring-decision logs become an immutable trace — here the course stitches to the engineering side (the control plane and audit plane from the neighboring course ai-governance): the legal requirement (Art. 12) is realized by concrete audit architecture.

Kompas sets up a post-market monitoring process (quality and drift metrics in production) and an incident runbook with notification deadlines — before the first incident happens, not after.

Where it breaks

Oversight is easy to fake. The subtlest edge is "meaningful." An "approve" button without a real ability to understand and contest the grounds is documented automation bias, not control. Building an interface where the human can actually intervene rather than merely formally approve is a product task, not a legal one.

"Accuracy" of scoring people — a metric of what? The accuracy of a ranking (how well the order matched some "gold standard") does not equal fairness and does not guarantee the absence of discrimination. A high accuracy metric can coexist with a systematic skew against a group.

The bounds of incident reporting are blurred. What exactly a "serious incident" is for an HR system, and within what deadlines to report, is unobvious in practice, and guidance is scarce. Better to have a deliberately over-inclusive process than to miss the threshold.

What to do as engineer/product

Design human oversight as a product function — explainable grounds + a real override + a log of the override — and test that the human can actually cancel the output, not merely agree with it. Oversight that can't be used is legally worse than an honestly described absence of it.

Turn on decision logging and the monitoring/incident process before release. Post-market monitoring and the runbook are not things you write up after the first investigation.

Provocation

Human oversight is the only AI Act requirement that can't be closed with a document: it changes the product itself. And here hides a paradox — an "approve" button that the recruiter clicks without reading is legally worse than its absence. Because every such click is a documented, logged proof that there was no human control — filed by your own hands.

Read next

• Next: Value chain — Who's on the hook: provider, deployer and the value chain
• All course chapters →
• Connect the expert as an MCP →