layout: ../../layouts/GuideLayout.astro title: “Fallback design: what happens when the AI call fails?” description: “A practical guide to designing resilient AI features: error handling, degraded modes, cached responses, and human escalation as first-class…” writtenBy: “gemma4:26b” reviewedBy: “deepseek-r1:32b” hero_image: “/images/hero/fallback-design-what-happens-when-the-ai-call-fails.png” lastChecked: “2026-05-28” scope: “Global. Provider reliability patterns, API error documentation, and industry incident post-mortems checked on 2026-05-28. Specific error codes and rate-limit tiers vary by provider and plan.”

Fallback design: what happens when the AI call fails?

TL;DR

Every AI feature will fail. Model APIs return 503s, rate limits hit mid-session, context windows overflow on long inputs. Design every AI feature with four fallback layers, in order:

  1. Degraded mode — return a simpler answer, a cached result, or a safe default
  2. Graceful error — show a clear message explaining what went wrong without blaming the provider
  3. Human escalation — route the request to a person or a manual workflow
  4. Offline state — disable the feature entirely if upstream dependencies are known to be unavailable

Most teams design for the happy path and discover the failure modes in production, under user load, during a demo for the CEO. The right fallback depends on the cost of failure. For a code completion, degraded mode (simpler completion) is fine. For a medical triage tool, only human escalation is acceptable.

What the tutorials skip

Every provider has different failure modes. OpenAI, Anthropic, Google, and open-weight providers all have different error codes, rate limits, availability SLAs, and retry behaviours. A retry strategy that works for one may cause cascading failures for another. Read the provider’s error documentation before writing fallback code.

Rate limits are not just about requests. Provider rate limits cover requests per minute, tokens per minute, tokens per day, and concurrent connections. Hitting any of these can block all users simultaneously. Design for backpressure before the limit is reached, not after.

Context window overflow is not an error. When a user sends a very long input, the model does not return an error — it truncates, loses context, or produces a worse answer. Fallback designs need to detect overflow before calling the API and handle long inputs differently (summarise, chunk, or refuse with explanation).

Degraded mode needs testing. A cached answer from last week may be worse than admitting the feature is temporarily unavailable. Test degraded mode outputs the same way you test happy-path outputs: accuracy, safety, and user experience.

Where teams misuse fallback design

Infinite retries. Retrying the same request against the same provider with the same parameters is not a fallback strategy. After 2–3 retries with exponential backoff, fall back to a different provider, a cached answer, or an error message.

Silent fallback. Switching to a weaker model or a cached answer without telling the user erodes trust. If the feature is running in degraded mode, say so. Users would rather know they are getting a less accurate answer than assume the full system is working.

Falling back to hallucination. Some systems respond to an API error by having the model generate a plausible-sounding answer with no ground truth. This is worse than an error message. A wrong answer confidently delivered is harder to detect than a feature that says “I don’t know right now.”

Practical design patterns

Pattern 1 — Provider cascade

Call primary provider first (e.g., Claude). On timeout or 5xx, fail over to secondary provider (e.g., GPT-4o). On failure, fail over to cached response. On cache miss, show degraded UI. Set request timeouts aggressively (10–15 seconds for LLM calls). Cache successful responses with a TTL appropriate for staleness tolerance.

Pattern 2 — Graceful degradation matrix

Failure typeUser experienceWhat the system does
Model API down”Results may be simpler — our AI is temporarily unavailable”Return cached results or simplified heuristic output
Rate limited”You’ve used your AI requests for now. Try again in a few minutes.”Show limit, offer retry timer
Context overflow”Your input was too long. Try a shorter question or upload a summary.”Refuse with explanation, offer hint
Embedding API down”Search is temporarily limited to keywords.”Fall back to BM25 or traditional search
All upstream down”This feature is temporarily unavailable. Your request has been logged.”Log request, notify operations, show offline banner

Pattern 3 — Human escalation with queues

Route failed requests automatically to a human review queue. The queue should show:

Track queue size and resolution time. If the queue backs up, that is a signal the AI feature is too unreliable for its current use case.

Decision framework

QuestionShould you build fallback?
Is the feature user-facing?Yes, always
Is the feature critical to the user’s task?Yes, with human escalation
Is latency sensitive?Yes, with aggressive timeouts and cached fallback
Is the cost of a wrong answer high?Yes, refuse rather than fall back to a weak model
Can you detect failure without the user reporting it?Yes, monitor API errors and degrade proactively

Caveats and scope boundaries

Methodology

Source list

Trust Stack

Change log