Chapter 5. FinOps & Resilience: cost, rate limits, and high availability

The customer's business goal

For Kovcheg: a provider outage during business hours means idle operations staff; an uncontrolled agent means a six-figure overnight bill; an exhausted rate limit means "the assistant is down" at peak hour. Promises to the business:

1. Spend is predictable and capped by department/user.
2. A provider outage or throttle doesn't bring the system down.
3. Optimizations (caching) never cross access or quality boundaries.

Driver: threat or regulator

• OWASP LLM: LLM10 (Unbounded Consumption) — cost and DoS via expensive requests.
• Vendor outage / lock-in: a single provider is a single point of failure.
• Rate limits (429): a traffic spike exhausts the quota.
• Cross-tenant leaks via cache: an optimization becoming a privacy incident.

Architectural pattern

Multi-Provider AI Gateway & Semantic Caching — a single gateway in front of all providers with semantic caching, budgets, fallback, and circuit breaking. The same layer that carries ch. 1 (PII), 3 (guardrails), 4 (tracing) — the planes don't spawn separate proxies, they live in one.

Engineering stack & providers

• AI Gateway: LiteLLM Proxy (redis-semantic / qdrant-semantic cache), Portkey, Kong AI Gateway (semantic cache plugin), Cloudflare AI Gateway, TrueFoundry.
• Cache/limits: Redis (semantic cache, rate limiting, budget counters); GPTCache as an alternative.
• Resilience: Envoy (circuit breaking), backoff-based retries.
• Observability: Prometheus + Grafana.

Engineering implementation

### Step 1. A single gateway

All model calls go through one layer. This is the same requirement as ch. 1 (bypassing it is a privacy hole).

### Step 2. Semantic caching with tenant isolation

Embed the prompt → search by cosine similarity in Redis; above the threshold, return the cached answer without calling the LLM. Lessons from the introduction's incident, baked into config:

semantic_cache:
  similarity_threshold: 0.97          # above the typical 0.88 — fewer false hits
  namespace: "{tenant_id}:{model}"    # model-level + tenant isolation, never global
  skip_if: ["contains_pii", "personalized"]  # never cache private content (ch. 1–2)
  per_request_threshold_override: true

Only cache non-personal, non-privileged content. In production, semantic caching realistically covers 20–45% of traffic — a material saving, as long as it doesn't break isolation.

### Step 3. Multi-tenant budgeting

Hard limits on tokens and cost by department and user; soft-alert → hard-stop. For agents (ch. 9), an additional per-task limit on steps/cost.

### Step 4. Fallback & circuit breaker

Auto-switch on rising latency or 5xx/429 errors; degrade to a backup provider or a local model; retries with backoff. A circuit breaker opens the circuit to a failing provider instead of hammering it into further failure.

### Step 5. Cost observability

Cost per request/tenant/feature in Grafana; a spending anomaly triggers an alert (this is where a nighttime agent burning through budget gets caught).

Where it breaks

• Semantic cache returns "almost the same." Requests close in embedding space but semantically different produce a wrong answer; in a regulated response (rate, terms) that's dangerous. Use a high threshold and a cautious scope; for critical paths, exact-match caching only.
• Cache versus privacy. A cached answer with someone else's data is a leak (the intro's case). Tenant-scoped namespaces and skipping personal content are mandatory, not optional.
• Fallback changes behavior. A different provider means different quality/format; guardrails (ch. 3) and evals (ch. 7) must cover every provider, or fallback silently degrades quality.
• Blind circuit breaking. An aggressive threshold cuts off legitimate traffic; calibrate on real latency profiles.
• Cache masks drift. If 40% of answers come from cache, degradation in fresh answers (ch. 7) is noticed later.

Standards and mapping

• OWASP LLM: LLM10 (Unbounded Consumption).
• ISO/IEC 42001: resource management, availability, continuity.
• NIST AI RMF: Manage — resilience.
• EU AI Act: robustness/continuity for high-risk (Art. 15).

Lab and artifact

Deploy LiteLLM/Portkey in front of Kovcheg: semantic cache with a tenant-scoped namespace and a 0.97 threshold, skip for PII, per-department budgets, fallback to a backup provider. Reproduce the intro's incident (global namespace + 0.88) to show the leak, then fix it via config. Simulate 429s and 5xxs — verify degradation and circuit breaking. Artifact: gateway config + a Grafana cost/latency dashboard + budget policies + a "cache hit-rate vs. cross-tenant safety" report.

Maturity checklist

• L1: a single gateway, basic limits, exact-match caching.
• L2: semantic caching with tenant isolation and skipping for private content, per-tenant budgets, fallback between providers.
• L3: circuit breaking, cost-anomaly alerts, evals across all providers, per-request threshold overrides, SLA monitoring.

Sources

• [Semantic caching thresholds and why they matter (Portkey)](https://portkey.ai/blog/semantic-caching-thresholds/)
• [Top semantic caching solutions 2026 (Maxim)](https://www.getmaxim.ai/articles/top-semantic-caching-solutions-for-ai-applications-in-2026/)
• [LLM caching strategies (NeuralTrust)](https://neuraltrust.ai/blog/llm-caching-strategies)
• [LLM Gateway guide 2026](https://noqta.tn/en/blog/llm-gateway-multi-model-routing-guide-2026)

Read next

• Next: Regulation — EU AI Act, ISO 42001 and content marking
• All course chapters →
• Connect the expert as an MCP →