Chapter 4. Auditability & Reproducibility: full audit trails and decision tracing

The customer's business goal

The ability to reconstruct the full reasoning chain and reproduce the context behind any Kovcheg decision. Promises to the business:

1. Everything that shaped a decision can be reconstructed from its trace_id.
2. The log is immutable — it can't be rewritten after the fact (otherwise it's useless as evidence).
3. The log's own retention and protection comply with GDPR (the log contains PII — ch. 1).

Driver: threat or regulator

• EU AI Act: Art. 12 — automatic logging for the entire lifetime of a high-risk system; Art. 13 (transparency); Art. 86 (the right to an explanation of an individual decision).
• GDPR: Art. 22 — explainability of automated decisions.
• Incident review: without a trace you can't tell a prompt bug (this chapter) from model drift (ch. 7) from an attack (ch. 3) apart — three different causes, three different responses.

Architectural pattern

Immutable AI Trace Log Pipeline built on OpenTelemetry GenAI Semantic Conventions — end-to-end tracing with standard spans, exported to immutable storage with a retention policy.

Why OTel SemConv rather than a proprietary format: the observability ecosystem is fragmented (Langfuse, Helicone, Traceloop, LangSmith — incompatible formats = vendor lock-in). GenAI Semantic Conventions (under CNCF) define a shared vocabulary: a span from a LangChain agent looks the same as one from a bare OpenAI call. An honest caveat: as of mid-2026 the conventions are still in Development status — in production, teams turn on OTEL_SEMCONV_STABILITY_OPT_IN for dual emission (legacy + new attributes) to survive the transition to stable.

Engineering stack & providers

• Tracing/standard: OpenTelemetry + GenAI Semantic Conventions (spans for LLM calls, retrieval, tool calls; conventions for MCP tools and multi-agent systems are under active development).
• Backends: Arize Phoenix, Langfuse, LangSmith (all accept OTel).
• Transport: Kafka / NATS.
• Storage: ClickHouse (analytics) + an immutable/WORM layer for legal audit.

Engineering implementation

### Step 1. An end-to-end trace_id

A single id runs the whole chain: gateway (ch. 1) → RAG + FGA (ch. 2) → input guardrails (ch. 3) → the model → tools (ch. 9) → output guardrails → response. The same id is the key into the PII mapping store (ch. 1) and the budget tracker (ch. 5).

### Step 2. What exactly gets captured in the spans

Per GenAI SemConv plus audit extensions:

• the system prompt version (a hash + a link to the version);
• the exact RAG chunks with their distance scores and source_doc_id;
• raw request/response (already with PII masked — ch. 1);
• the arguments of every called function (tool calling) and the policy gate's verdict (ch. 9);
• the model/provider version, temperature, seed;
• the guardrail verdicts (what fired, what was blocked — ch. 3).

### Step 3. Export to immutable storage

app (OTel SDK) → OTLP → collector → Kafka → ┬→ ClickHouse (analytics/dashboards)
                                            └→ WORM/append-only (legal audit, retention)

The append-only layer isn't "a database everyone agreed not to write over" — it's storage with technical impossibility of overwrite (object-lock / WORM).

### Step 4. Reproducibility

Given a trace_id, the full input is reassembled. With a fixed model version and seed, a decision can be replayed — with the caveat noted in failure modes.

### Step 5. The log's own privacy

The audit trail contains PII and secrets → encryption, role-based access, a separate TTL that balances Art. 12 (retain) against GDPR (don't over-retain).

Where it breaks

• LLM non-determinism. Even with a seed, reproducibility is incomplete: providers swap models "under the hood," sampling varies. An "explanation" is a reconstruction of the decision's context, not literal causality.
• Post-hoc rationalization. Chain-of-thought in the log is what the model wrote, not necessarily why it decided that way. Don't sell CoT as a causal mechanism.
• Volume and cost. A full trace with chunks runs into terabytes; retention under Art. 12 versus the ClickHouse budget versus GDPR minimization is a three-way conflict, resolved by sampling detail level (full trace for high-risk decisions, truncated for the rest).
• The log as an attack surface. An audit trail with PII and secrets is a prime target; its leak is worse than the original one. Protecting the log is part of the perimeter, not an afterthought.
• The standard isn't stable yet. SemConv is in Development — attributes change, hence dual emission.

Standards and mapping

• EU AI Act: Art. 12 (record-keeping), Art. 13 (transparency), Art. 86 (explanation).
• ISO/IEC 42001: logging, monitoring, incident management.
• NIST AI RMF: Measure/Manage — traceability.
• GDPR: Art. 22.

Lab and artifact

Instrument Kovcheg with OTel GenAI SemConv, export through Kafka into ClickHouse plus an object-lock layer; given a trace_id, reconstruct the full context of the decision from the introduction; configure log retention and encryption. Artifact: a trace-event schema (SemConv attributes + audit extensions) + a "replay one decision" dashboard + a retention/protection policy (evidence for ch. 6).

Maturity checklist

• L1: raw request/response and model version are logged.
• L2: an OTel trace (SemConv) with chunks, prompt version, and tool arguments in append-only storage.
• L3: decision replay by trace_id, retention aligned with the AI Act, encryption and RBAC on the audit trail, dual emission during the standard's transition period.

Sources

• [OpenTelemetry GenAI Semantic Conventions (MLflow docs)](https://mlflow.org/docs/latest/genai/tracing/opentelemetry/genai-semconv/)
• [OpenTelemetry for AI Systems: LLM & Agent Observability 2026 (Uptrace)](https://uptrace.dev/blog/opentelemetry-ai-systems)
• [How OTel traces LLM, agent reasoning, MCP tools (Greptime)](https://greptime.com/blogs/2026-05-09-opentelemetry-genai-semantic-conventions)
• [OpenTelemetry for LLMs: SRE guide 2026 (OpenObserve)](https://openobserve.ai/blog/opentelemetry-for-llms/)

How it actually works — engineering breakdowns

Standalone howto from practice, showing this control plane on real code and a working artifact.

• Sending to Kafka Is a Two-Phase CommitDelivery semantics in Kafka — transport reliability for the immutable trace log.
• Agent Builds Executable Deterministic WorkflowsThe agent builds deterministic, reproducible workflows.

Read next

• Next: FinOps — Cost under control, the system holds its SLA
• All course chapters →
• Connect the expert as an MCP →