Manual context engineering looks straightforward: find the right information, stuff it into a prompt, get an adequate answer. While your context fits in a couple of paragraphs or is assembled by hand, that model works. No questions.
The real trouble starts the moment context collection is handed to automation that parses dozens of files, databases, and third-party APIs. At that point, your context pipeline becomes a garbage chute.
Why automated collection breaks what manual collection handled
The root cause is simple: automated scripts collect data without semantic awareness. The script does not know what it is pulling — it pulls everything that matches a formal criterion. Three kinds of garbage land in a single prompt.
Different versions of truth. An instruction from 2021 documentation and a code fragment from a 2026 update that directly contradict each other. Both are valid as "sources," both enter the context.
Mutually exclusive inputs. A log entry from two days ago where the system says "user blocked," and today's database record with status "active." The script pulled both because both are about the same user.
Noise and source hallucination. Data from disparate services using identical terms for completely different things. The term matched, so the script decided it is relevant. It is not.
Feed this salad to a target LLM and it behaves predictably badly. The model oscillates between extremes, produces mutually exclusive answers to the same query, or tries to "average" things that cannot be averaged. The output is confident nonsense that takes hours to untangle. The worst part: the target model did its job honestly. Garbage in, garbage out. The LLM is not at fault.
Interlayer arbitration: an extra stage between collection and resolution
To avoid turning the target LLM into a coin-flip guess, you insert one more stage into the data chain — an arbiter model, or Consistency Evaluator.
The flow is: before the script-assembled context goes to the main model for the primary task, it runs through a separate, faster LLM tuned for logical analysis. This is not a replacement for the target model and not a more complex prompt — it is a dedicated checkpoint between collection and resolution.
[Data sources] ──> [Collection script] ──> [Arbiter model] ──┬──> (OK) ──────> [Target LLM]
└──> (Conflict) ─> [HITL flag]The arbiter's job is not to solve the user's task. Its job is to assess whether the context is fit for that task. It checks three things.
Detects logical holes. Checks whether Fragment A contradicts Fragment B.
Evaluates chronology. Filters stale inputs when fresher data on the same question exists in the context.
Classifies the conflict. Determines whether a discrepancy is a critical blocker or minor noise that can be filtered on the fly.
Classification decides what gets escalated. Not every discrepancy warrants escalation. Minor noise — identical terms from different services, a stale row when a fresh one exists — the arbiter resolves itself and passes clean context forward. A hard contradiction that cannot be resolved without losing meaning is outside its responsibility.
When the machine needs a human
Hard contradictions go to HITL — Human-in-the-Loop.
When the arbiter model sees that the context contains hard contradictions it cannot resolve without losing meaning, it does not try to guess the correct answer. This is the fundamental difference from the target LLM: the target, in the same situation, would confidently pick something and drive on. The arbiter instead raises a flag immediately — generates an alert, halts the pipeline, and sends the disputed context fragment to a human operator.
The operator confirms the correct version, which goes back into the data sources. The context is then reassembled, and if no contradictions remain — only then does the clean, consistent context reach the target model. The fix goes into the source, not into a one-off prompt: a resolved contradiction will not resurface on the next collection run for the same query.
The cost of peace of mind: +1 step
The trade-off: yes, this adds one extra step to the architecture. The pipeline gets longer, there is one more model call, and on a hard conflict — a pause for a human.
The "direct" scheme is more expensive when it fails. Spending 200 milliseconds and fractions of a cent on an evaluator model call beats getting an invalid result from an expensive LLM and sending the user nonsense generated from conflicting data. One arbiter call costs pennies and milliseconds; hours spent untangling confident nonsense cost engineering time and trust in the system.
The direct scheme — collect, stuff into prompt — works exactly as long as a human assembles the context and holds its meaning in their head. The moment collection goes to automation, someone between the script and the target model needs to own input consistency. That is either another model that can say "I won't hazard a guess here," or a human it calls in time.