An LLM is a probabilistic component. The model will err. The engineering task is ensuring that error stays an error of the model, not a product incident.
An agent sends an email with order details to the wrong recipient. Another generates a destructive SQL query against the production database — because a "delete command example" was sitting in the text of the ticket it was reading. A third pulls a fragment of an internal document through RAG, complete with a secret token, and hands it to the user.
These failure modes show up when agents get write access to email, databases, or internal knowledge bases. In each case, the model produced a syntactically valid action; the system failed to constrain its consequences. Each operated correctly — a valid tool call, a well-formed SQL query, a relevant fragment from the knowledge base. The incident came from missing authorization, validation, or data-leak controls around the model output.
Assume the model will sometimes misread context, follow injected instructions, or choose a harmful tool call. The engineering task is ensuring that error stays an error of the model, not a production incident. When you first connect to an LLM API, it feels like a finished system. It isn't. A production AI deployment needs infrastructure for authorization, validation, logging, evals, rollback, and monitoring, and it rests on two layers: guardrails and harness.
Continue reading “What AI Does While You Look Away: Guardrails and Harness”
