theLLMs

Last checked: 2026-05-28

Scope: Global. Sources checked as of 2026-05-28. Applies to all major LLM API providers (Openly, Anthropic, Google, Meta).

Key terms:

>_ large language model (LLM) >_ JSON mode >_ function calling / tool use >_ LLM-as-a-judge >_ batch API >_ fallback
Hero image for The hidden cost of retries, fallbacks and validation loops

The hidden cost of retries, fallbacks and validation loops ## TL;DR Retries and validation loops can multiply your effective per-task cost by 2–10× depending on error rates, model choice and output constraints. Most teams budget for one happy-path API call per task; the enough number is often 3–5 calls when you count schema validation failures, safety refusals, tool-call retries, and fallback prompts. The multiplier compounds silently — per-call pricing calculators don’t show it. ## What it means An LLM API call looks like a single transaction in the dashboard. One request, one response, one line on the bill. But for many production use cases — especially those using structured outputs, function calling, or agent loops — that single line is a lie. The typical production flow looks more like this: 1. Primary call — send prompt, expect structured JSON or a tool call

  1. Parse failure — JSON is malformed or schema-valid but semantically wrong → retry
  2. Safety refusal — model refuses to answer → fallback prompt or retry with relaxed system instructions
  3. Tool-call error — model chose the efficient function, or arguments don’t match the schema → retry with corrected prompt
  4. Validation failure — output passes syntax but fails business rules → retry with additional context
  5. Fallback model — after N retries, route to a more capable (and more expensive) model Each loop iteration adds input tokens (re-sending the conversation history plus the error signal), output tokens (the new attempt), and latency. The cost compounds, and it’s invisible in per-call pricing calculators.
## Where teams misuse it **"Our per-call cost is $0.003."** That's the happy-path price. If 20% of calls require one retry, the effective cost is $0.0036 — a 20% hidden uplift. If the retry doubles because the model consistently struggles with a complex schema, you're at 2–3× before you notice. **"We just ask for JSON and it works."** It works until the model returns markdown-wrapped JSON, or a single trailing comma, or a string instead of an object. The model doesn't care about your schema — it optimises for plausible-looking text. Validation is your problem, and each failure costs a retry. **"Safety refusals are rare."** On safety-tuned models, refusals for borderline-but-legitimate queries can hit 5–15% in domains like medical, legal, or financial advice. Each refusal is a full round-trip, and the fallback prompt to get a useful answer is often longer than the original. ## Practical decision check Before shipping an LLM feature, measure these numbers on real traffic (not toy examples): - **Schema validation failure rate** — what % of outputs need a new retry because the format is wrong? - **Semantic validation failure rate** — what % pass JSON schema but fail business rules (e.g., a price field that should be positive is negative)? - **Safety refusal rate** — what % of queries trigger refusals? - **Tool-call error rate** — what % of function calls select the wrong tool or produce invalid arguments? - **Fallback cascade depth** — after N retries, do you give up, route to a human, or escalate to a more expensive model? A healthy system keeps the **total call multiplier** (actual API calls ÷ happy-path calls) under 2. Above 3, the architecture is fighting the model rather than working with it. ## Mitigations worth trying first 1. **Simplify output schemas** — flatter JSON, fewer optional fields, narrower enums. Every optional field is a failure point. 2. **Use constrained decoding** where available — tools like JSON mode or grammar-unguided generation (llama.cpp, Outlines, OpenAI structured outputs) drastically reduce format failures at the cost of slightly higher latency. 3. **Isolate retries to the failed component** — if the model chose the wrong tool, retry only the tool-selection call, not the full conversation. 4. **Set a hard retry limit** and fall back to a deterministic response or human escalation. Three retries is a reasonable ceiling for most products. 5. **Log every retry reason** in your observability pipeline. If you can't name the top three failure modes in your system, you can't fix them. ## Methodology - **Data checked:** 2026-05-28 - **Sources consulted:** OpenAI structured outputs documentation (2026), Anthropic tool use documentation (2026), Google Gemini structured outputs documentation (2026), Berkeley Function-Calling Leaderboard (v3, 2025), OWASP LLM Top 10 (v2.0, 2025), LangSmith and Helicone observability platform documentation (2026) - **Assumptions:** Retry multipliers are workload-specific and vary by model, schema complexity, and content domain. The 2–10× range represents typical production workloads observed across SaaS and enterprise deployments as of mid-2026. - **Limitations:** This article does not cover retry strategies specific to streaming responses, multi-agent architectures with cross-agent validation, or training-time approaches to reducing format errors. It focuses on inference-time API retries for text-based structured output use cases. - **Jurisdiction:** Global. Provider-specific features (OpenAI structured outputs, Anthropic tool use, Google Gemini controlled generation) are available in most regions but may vary by deployment tier and geography. ## Source list - OpenAI — Structured Outputs guide (accessed 2025-05-28): https://platform.openai.com/docs/guides/structured-outputs - Anthropic — Tool Use documentation (accessed 2026-05-28): https://docs.anthropic.com/en/docs/build-with-claude/tool-use - Google AI — Gemini Structured Outputs (accessed 2026-05-28): https://ai.google.dev/gemini-api/docs/structured-outputs - Berkeley Function-Calling Leaderboard — Gorilla project (accessed 2026-05-28): https://gorilla.cs.berkeley.edu/leaderboard.html - OWASP — LLM Top 10: Tool use and output handling (accessed 2026-05-28): https://genai.owasp.org/llm-top-10/ - LangSmith — Observability platform for LLM applications (accessed 2026-05-28): https://smith.langchain.com/ - Helicone — Observability for LLMs (accessed 2026-05-28): https://www.helicone.ai/ - Portkey — AI gateway and observability (accessed 2026-05-28): https://portkey.ai/ ## Conclusions Retries and fallbacks are necessary for reliability in production LLM systems, but they come with a hidden economic tax. By monitoring your retry multiplier and optimizing your schemas to minimize failures, you can build high-accuracy agents that remain cost-effective even as complexity scales. ## Trust Stack - **Last checked:** 2026-05-28 - **Corrections:** [Contact us](/contact/) to report errors ## Change log - 2026-05-28: Full editorial review against 16-gate checklist. Added Quick Answer section, 3 Editor's Notes, Methodology, Source List, Trust Stack, and slugified heading IDs throughout. - 2026-05-25: Initial audit revision. Added direct source URLs to evidence section; changed source listing from named-for references to linked citations. No material changes to claims or guidance. ## Related guides - [LLM observability cost: logs, traces and evaluation storage](/cache/llm-observability-cost-logs-traces-and-evaluation-storage/) - [API model pricing: input, output, cache and batch costs](/cache/api-model-pricing-input-output-cache-batch-costs/) - [Output tokens are expensive: designing shorter AI answers without hurting usefulness](/cache/output-tokens-expensive-shorter-ai-answers/)