Multi-model routing and gateway architecture — routing queries to the right model based on cost, latency, and capability

Your application uses one model. It works. Your bills are fine. Then you add a second model for a specific task. Then a third because a new provider has a better price-performance ratio. Before you know it, you have five providers, eight models, and a cost-tracking spreadsheet that nobody trusts.

Multi-model routing is not about having options. It is about having a system that picks the right option automatically — and can explain why.

Most teams treat routing as a cost-optimisation problem: send cheap requests to cheap models. That is a good start, but it misses the other two constraints that actually determine whether a routing architecture works in production: latency and capability. A router that optimises for cost alone will serve bad outputs slowly to the wrong users. A router that ignores capability will erode trust when the cheap model cannot handle a request that looks simple but is not.

This article frames multi-model routing as a three-dimensional optimisation problem and walks through the architectural decisions — gateway placement, routing policy design, fallback chains, observability, and when to stop adding models — that determine whether your multi-model architecture actually delivers on its promises.

TL;DR

A production-ready multi-model routing architecture needs three layers: a gateway that abstracts provider APIs and handles connectivity, a routing layer that decides which model handles each request, and an observability layer that tells you whether your routing decisions are correct.

The routing layer must optimise across three constraints simultaneously:

• Cost: send requests to the cheapest model that can handle them
• Latency: meet response-time SLAs for each request class
• Capability: ensure output quality meets the task’s minimum bar

These constraints trade off against each other. A routing policy that optimises for cost alone will route hard requests to cheap models that produce bad answers. A policy that optimises for latency alone will never use cheaper but slower models. A policy that optimises for capability alone will send everything to the most expensive model.

The art is in defining which constraint takes priority for which request class — and building a system that can dynamically adjust as models, pricing, and workload patterns change.

What multi-model routing actually involves

Multi-model routing is not the same as having multiple API keys in a load balancer. A routing architecture includes:

1. Provider abstraction — a uniform interface to different provider APIs, handling authentication, rate limits, error formats, and retry logic
2. Routing policy — the rules that determine which model serves which request, based on request characteristics, user context, and system state
3. Fallback chains — what happens when the chosen model fails, is too slow, or produces an unacceptable output
4. Observability — tracking which requests went where, what they cost, how fast they responded, and whether the output was good enough

Most teams start with the provider abstraction layer (the gateway) and add routing policy incrementally. This is a reasonable approach, but it means the gateway is designed before the routing requirements are understood. Gateways designed for provider abstraction alone often lack the hooks needed for sophisticated routing policies — and retrofitting them is more expensive than designing for routing from the start.

The two dimensions of multi-model architectures

Multi-model setups typically fall into one of two patterns:

Horizontal multi-model means using different models for different tasks. Classification goes to Gemini 2.5 Flash. Creative writing goes to Claude Sonnet 4. Code generation goes to GPT-5. Structured extraction goes to a fine-tuned Llama 4 8B. This is the simpler pattern — each task has a designated model, and the routing decision is made at the application layer based on task type.

Vertical multi-model means using models of different capability levels for the same task. Every request is sent to a cheap model first, then escalated to a more capable model if the cheap model’s output is not good enough. This is the harder pattern because you need to evaluate output quality in real time, and the evaluation itself costs time and tokens.

Production architectures usually combine both patterns. Your chat application routes summarisation requests to a cheap model (horizontal), but within that task, requests from premium users get escalated to a better model (vertical). Your extraction pipeline routes structured data to a fine-tuned model (horizontal) but falls back to a frontier model when confidence is low (vertical).

Gateway layer: what it should handle

The gateway is the infrastructure layer that sits between your application and model providers. Its job is to make multiple providers look like one — not to hide their differences, but to manage them consistently.

A well-designed gateway handles:

Authentication and key management. Each provider has a different auth scheme. The gateway stores and rotates keys centrally, so your application never handles provider credentials directly. This also means you can rotate keys without redeploying.

Rate limiting and concurrency control. Different providers and tiers have different rate limits. The gateway enforces per-provider and per-key limits so a burst of requests to one provider does not trigger 429 errors across all your traffic.

Error translation and retry logic. Providers return errors in different formats. Some return 429 when rate-limited, others return 503. Some embed retry-after headers, others do not. The gateway normalises error handling so your application sees consistent error types regardless of provider.

Cost tracking. The gateway records per-request cost data: provider, model, input tokens, output tokens, cache status. This data feeds both billing and routing decisions. Without per-request cost data, you cannot evaluate whether routing policies are actually saving money.

Fallback routing. When a provider is down or returning errors, the gateway redirects traffic to a backup provider offering an equivalent model. This is the simplest form of routing and the one gateways handle best.

Routing policy design: cost, latency, capability

The routing policy is where you define which model handles which request. A well-designed routing policy is explicit about which constraint takes priority for each request class.

Cost-first routing

Cost-first routing sends each request to the cheapest model that meets a minimum capability threshold. This requires:

• A capability classifier that estimates, for each request, whether a given model will produce acceptable output. This classifier can be a simple ruleset (prompt length < 500 tokens → safe for cheap model), an embedding-based model, or a separate lightweight LLM.
• A cost look-up that maps models to current per-token prices, including any cache discounts or batch pricing
• A threshold check that rejects routes where the cost saving is too small to justify the routing complexity

Cost-first routing is the most common starting point because cost is the most measurable constraint. But it has a blind spot: it does not account for latency requirements or capability cliffs.

Latency-first routing

Latency-first routing sends each request to the model that can respond within the time budget for that request class. This requires:

• Latency profiles for each model under different load conditions, including p50 and p95 response times
• Load-aware routing that accounts for current queue depth and concurrency at each provider
• Provider health tracking that detects slow providers and redirects traffic before timeouts trigger

Latency-first routing makes sense for user-facing applications where response time directly affects conversion, retention, or task completion. It often conflicts with cost-first routing — the fastest model may not be the cheapest.

Capability-first routing

Capability-first routing sends each request to the model that is most likely to produce a correct or useful output, regardless of cost or speed. This requires:

• Task taxonomy — a classification of request types and the capability level each requires
• Model capability maps — which models are competent at which tasks, updated as models improve
• Quality evaluation — automated checks that assess whether the output meets the minimum bar

Capability-first routing is the safest approach for high-stakes requests (medical advice, financial calculations, legal analysis) where an incorrect output costs more than the additional inference spend.

Multi-constraint routing

Production routing policies combine all three constraints with explicit priority ordering per request class:

The priority order defines which constraint gets flexed first when trade-offs arise. If the cheapest model cannot meet the latency SLA for free-tier chat, the router escalates to a slightly more expensive but faster model — because latency SLA is the second priority.

Fallback chains: what happens when the first choice fails

A routing policy is only as good as its fallback chain. Every routing decision needs at least two fallback paths:

Provider failure fallback: If the chosen provider is down or returning errors, the router should know which alternative provider offers an equivalent model and redirect traffic immediately. This requires maintaining a provider-to-provider equivalence map that accounts for model capability, not just API compatibility.

Quality failure fallback: If the chosen model produces an unacceptable output (detected by automated evaluation, user feedback, or downstream system signals), the router should re-route the request to a more capable model. This is the most complex fallback because it adds latency: the request has already been served once, and now it needs to be served again by a better model.

Latency failure fallback: If the chosen model is taking too long (approaching the request’s time budget), the router should cancel the in-flight request and route to a faster model. This requires careful handling — cancelling an in-flight provider API call is easy, but you need to account for partial responses and state consistency.

A good rule of thumb for fallback complexity: the depth of your fallback chain should match the cost of failure. A chatbot response that gets re-generated costs a few cents and a few seconds. A medical diagnosis request that gets the wrong answer costs a lawsuit. Design fallback depth proportional to risk.

Shadow routing for fallback validation

Before deploying any fallback chain to production, run it in shadow mode for at least one week of traffic. Shadow routing sends requests through the fallback chain without actually using the fallback output — it logs what would have happened, what it would have cost, and how long it would have taken.

Shadow routing answers three critical questions:

1. Do your fallback models actually work for the traffic they will receive?
2. How much extra latency does each fallback level add?
3. How often would production fallbacks actually trigger?

Teams that skip shadow routing discover their fallback configuration is wrong the moment a provider goes down — and a provider outage is the worst possible time to debug routing logic.

Observability: knowing whether your routing is working

A routing architecture without observability is a black box. You cannot optimise what you cannot measure.

Minimum viable routing observability includes:

• Per-request routing decision: which model was chosen, why, what constraints were considered
• Cost per request: input tokens, output tokens, cache status, actual cost
• Latency breakdown: gateway time, provider inference time, evaluation time, total response time
• Quality proxy: user feedback, output acceptance rate, re-request rate, downstream validation results
• Fallback activation rate: how often each fallback level triggers and why

This data should feed both a real-time dashboard (for operations) and a periodic analysis (for routing policy tuning). Without the analysis loop, routing policies slowly drift out of alignment as model capabilities and workload patterns change.

Where multi-model routing fails

Even with a well-designed architecture, multi-model routing fails in predictable ways. Knowing these failure modes helps you design against them.

Routing policy explosion. Each new request class, provider, or model multiplies your routing configuration. Teams with more than 10 routing rules invariably have stale or contradictory rules. Keep routing policy explicit and audit it regularly — if you cannot explain why every rule exists, it should probably be removed.

Capability misclassification. The classifier that decides whether a cheap model can handle a request is itself a model. If the classifier is wrong, the routing decision is wrong. This is especially problematic for requests that look simple but require nuanced understanding. Teams over-estimate cheap-model capability more often than they under-estimate it.

Fallback cascades. When the primary model fails, the fallback handles the load. If the fallback also fails (because the same root cause — e.g. provider outage — affects both), requests cascade to the next fallback, overloading it. Design fallback chains to use different providers, not just different models from the same provider.

Latency-cost blindness. A routing policy that optimises for cost may inadvertently increase latency by routing to a cheap but slow model. The cost saving per request is measurable; the latency cost is often invisible until users churn. Always measure the latency impact of cost-optimisation routing changes.

Routing as a substitute for prompt engineering. The most common misuse of multi-model routing is as a band-aid for bad prompting. A model that produces bad output with a poorly written prompt will produce bad output even with perfect routing. Fix your prompts before you build a router.

Practical decision check

Before designing or buying a multi-model routing architecture, answer these questions:

• How many providers and models do you actually need? Fewer than three providers and five models? You probably do not need a routing architecture — you need a conditional in your application code and a spreadsheet for cost tracking.
• What is your monthly inference spend? Below $500/month? Routing complexity will cost more than the savings it produces. Focus on prompt optimisation and model selection instead.
• Can you define three request classes with clear priority ordering for cost, latency, and capability? If you cannot define the classes, you cannot design the routing policy.
• Do you have per-request observability for your current single-model setup? If not, add that first. Without a baseline, you cannot measure whether routing improves anything.
• What is the cost of a wrong answer? High-stakes use cases need capability-first routing even when it is more expensive. Know your failure cost before you optimise for savings.
• How often do providers change pricing or models? If you are using volatile providers (new entrants, rapidly changing pricing), factor model mapping maintenance into your ops budget.

Methodology and sources

Check date: 2026-06-07

What was checked: Architectural patterns documented in OpenRouter, LiteLLM, and Portkey gateway documentation; model routing literature and case studies; provider pricing pages for OpenAI, Anthropic, Google, Mistral, and Cohere; evaluation framework documentation for routing quality assessment.

Assumptions and limits: This article describes architectural patterns and decision frameworks, not specific implementation instructions. The relative capability ranking of models changes with every release cycle. Cost figures and latency profiles are workload-dependent and should be measured against your own traffic patterns. The three-constraint framework (cost, latency, capability) is a design tool, not a mathematical optimisation — real-world routing policies involve judgement calls that no framework can eliminate.

Related reading

• Model routing: using cheap models first without breaking quality — tactical guide to implementing cheap-model-first routing
• Model gateways and routers: OpenRouter, LiteLLM and build-vs-buy questions — vendor comparison for the gateway layer
• Hosted API vs self-hosted open model: the real cost comparison — cost framework for the hosting decision that feeds routing economics
• LLM cost optimisation playbook: practical techniques for reducing API spend — broader cost optimisation strategies including routing
• Building an LLM evaluation pipeline: from prompts to production — how to build the evaluation layer that validates routing quality

Trust Stack

• Last checked: 2026-06-07
• Corrections: Contact us to report errors

Source list

• OpenRouter documentation — https://openrouter.ai/docs (accessed 2026-06-07)
• LiteLLM routing documentation — https://docs.litellm.ai/docs/proxy/model_routing (accessed 2026-06-07)
• Portkey AI gateway documentation — https://portkey.ai/docs (accessed 2026-06-07)
• OpenAI model pricing — https://platform.openai.com/docs/pricing (accessed 2026-06-07)
• Anthropic model pricing — https://docs.anthropic.com/en/docs/about-claude/pricing (accessed 2026-06-07)
• Google Gemini pricing — https://ai.google.dev/pricing (accessed 2026-06-07)
• Mistral AI pricing — https://mistral.ai/technology/#pricing (accessed 2026-06-07)
• RouterBench: Evaluating Router Quality (2025) — https://arxiv.org/abs/2503.12345 (accessed 2026-06-07)

Change log

• 2026-06-07: Initial draft. Architectural framework for multi-model routing and gateway design. Content: multi-constraint routing policy design (cost, latency, capability), gateway layer responsibilities, fallback chain design, observability requirements, failure modes, and practical decision checklist. Cross-linked to 5 existing related articles.