LLM decision tree: which model to use when

Quick answer

If you need the shortest useful answer: use a reasoning model for complex logic or math, a fast frontier model for open-ended chat or creative work, a specialised coding model for software tasks, a small local model for private offline inference, and a cheap API model for classification or structured extraction.

The actual decision is rarely that clean. Every task sits at the intersection of reasoning depth, latency budget, cost tolerance, privacy requirements, and output quality. This guide gives you a structured decision tree — not a single answer — so you can make the right trade-off for your specific workload.

Quick match: task to model family

The decision tree

Step 1: Does your task require multi-step reasoning?

A task needs multi-step reasoning when you cannot get the right answer from a single forward pass. Examples:

• “Calculate the net present value of this cash flow, then compare it against the alternative investment, and recommend which to pursue.”
• “This user query has three implicit intents — identify each, generate a tailored response for each, then synthesise them.”
• “Prove or disprove this logical statement using first-order logic.”

If yes → use a reasoning model (o-series, DeepSeek-R1, Gemini Thinking). Expect higher latency (5–30s per response) and 2–5× higher per-token cost than fast models. The trade-off is justified when a wrong answer costs more than the waiting time.

If no → move to Step 2.

Step 2: Is this a coding task?

Coding tasks include code generation, debugging, code review, test generation, refactoring, and infrastructure-as-code authoring. Generic frontier models can write code, but specialised coding models produce fewer bugs and better follow language-specific conventions.

If yes → choose a model with strong coding benchmarks. As of June 2026, Claude Sonnet 4 and GPT-4o lead on SWE-bench and HumanEval. DeepSeek-Coder V3 offers competitive open-weights performance. For IDE completion (short snippets), a local Code Llama or Qwen-Coder quantised model at 7B–14B provides near-instant latency with no API cost.

If no → move to Step 3.

Step 3: Do you need real-time interaction?

Real-time tasks include chatbots, voice assistants, live transcription, interactive tutoring, and any application where a user is waiting for a response. The acceptable latency ceiling is roughly 2–3 seconds; above that, users disengage.

If yes → use a fast frontier model (GPT-4o, Claude Sonnet, Gemini 2.0 Flash). These models return the first tokens in 200–800ms and complete typical responses in 1–3 seconds. Avoid reasoning models for real-time tasks — their internal chain-of-thought adds 5–30s before the first output token.

If no → move to Step 4.

Step 4: Is data privacy your primary constraint?

If your data cannot leave your infrastructure — regulated financial data, medical records, classified information, proprietary source code — API-based models are not an option regardless of their quality.

If yes → use a small-to-medium local model, quantised to run on available hardware. As of June 2026, Llama 3.1 8B Q4_K_M runs on a single consumer GPU (8–12 GB VRAM) and handles summarisation, classification, and structured extraction well. Qwen 2.5 7B Q4_K_M is a strong alternative for JSON output and tool use. For better quality, Llama 3.1 70B Q3_K_M needs 24–32 GB VRAM and provides frontier-competitive quality on many tasks.

If no → move to Step 5.

Step 5: Is the task simple, fast, or high-volume?

Simple tasks include: binary classification (spam/not-spam), single-label sentiment, structured extraction from predictable formats, content routing, and one-shot translation. These tasks do not benefit from a 70B-parameter model.

If yes → use a small API model (GPT-4o-mini, Claude Haiku, Gemini 2.0 Flash Lite). These models cost 0.15–0.40 per million input tokens — roughly 10–30× cheaper than their full-size counterparts — and return responses in under 1 second for short inputs. For very high volume (millions of requests per day), consider a fine-tuned small model or a local quantised model to avoid per-request API costs.

If no → use a general-purpose frontier model (GPT-4o, Claude Sonnet, Gemini 2.0 Pro). These models handle the widest range of tasks and degrade the least when given ambiguous or underspecified instructions. They are the safe default when you are not sure which category your task fits.

When to use more than one model

Most production systems work better with a model stack rather than a single model:

• Cascade routing: Try a cheap model first (GPT-4o-mini), measure confidence. Fall back to a frontier model (GPT-4o) when confidence is low. This pattern saves 50–70% on API costs while maintaining output quality.
• Agent routing: Use a small, fast model to classify intent, then route to the appropriate specialist model. A router running on GPT-4o-mini costs pennies per 10,000 routing decisions.
• Ensembling: Run the same prompt through two different model families (e.g., GPT-4o and Claude Sonnet) and take the consensus, or use a third model to arbitrate disagreements. This improves factual accuracy by 5–15% at roughly 2–3× the cost.
• Hybrid local + API: Use a local model for initial processing of sensitive data, then pass anonymised summaries to an API frontier model for synthesis. This keeps raw data private while still benefiting from high-quality reasoning.

How models compare on key dimensions (mid-2026)

The following table captures approximate positions as of June 2026. Verify current rankings against live leaderboards (LiveBench, Chatbot Arena, SWE-bench) before procurement decisions.

What each model family is for

Reasoning models

Reasoning models (o-series, DeepSeek-R1, Gemini Thinking) spend extra compute on internal chain-of-thought before generating a visible answer. This makes them slower and more expensive per query, but significantly more accurate on tasks that require logical deduction, math, or multi-step planning.

Best for: Complex analysis, mathematical reasoning, scientific research, strategy planning, multi-step agent coordination, compliance reasoning.

Not for: Real-time chat, simple classification, high-throughput batch processing, creative ideation (they tend to over-reason simple tasks).

Cost: $1.10–$7.00/1M input tokens depending on provider and model variant.

Fast frontier models

Generic high-quality models that balance speed, cost, and capability. GPT-4o, Claude Sonnet, and Gemini 2.0 Pro are the current standards. These models handle the widest range of everyday tasks with good quality and acceptable latency.

Best for: Chatbots, content drafting, summarisation, analysis, research assistance, customer support, general-purpose automation.

Not for: Tasks requiring deep multi-step reasoning (they can try but accuracy will be lower than reasoning models), tasks requiring complete privacy (API-based by nature).

Cost: $1.10–$3.00/1M input tokens.

Small and specialised models

Smaller models (7B–14B parameters) serve specific niches: coding assistants, classification, low-latency routing, and private inference. They are often available as quantised local models, avoiding API costs and data leakage entirely.

Best for: Code completion (in-IDE), content classification, routing decisions, spam filtering, NER extraction, first-pass document triage, private data processing.

Not for: Open-ended creative work, complex reasoning, tasks requiring broad world knowledge.

Cost: $0.00–$0.40/1M input tokens (free if self-hosted).

Cheap API models

GPT-4o-mini, Claude Haiku, Gemini 2.0 Flash Lite and similar “small API” models provide adequate quality on straightforward tasks at 10–30× lower cost than frontier models. They are the default choice for high-volume production systems.

Best for: Classification at scale, content moderation, language detection, simple extraction, customer query routing, A/B evaluation.

Not for: Complex reasoning, nuanced creative writing, multi-turn conversations requiring context depth.

Cost: $0.15–$0.40/1M input tokens.

Common scenarios

Building a customer support chatbot

Use a cascade: route simple FAQ queries to GPT-4o-mini or Claude Haiku (fast, cheap). Escalate complex refund or complaint issues to Claude Sonnet or GPT-4o (more nuanced reasoning). Add a guardrail model to check for harmful output before the response reaches the customer.

Suggested stack: Cheap router (GPT-4o-mini) → Frontier responder (Claude Sonnet 4) → Safety guardrail (Llama Guard or local classifier).

Running batch inference on a million documents

Use a cheap API model with batch API pricing (50% discount). For structured extraction from consistent formats, consider a fine-tuned small model for lower per-document cost. If the documents contain PII, use a local quantised model.

Suggested stack: Batch API (GPT-4o-mini batch, ~$0.075/1M input) or local fine-tuned Qwen 2.5 7B.

Processing sensitive financial data

Only local models are acceptable. Use Llama 3.1 70B Q3_K_M or Qwen 2.5 32B Q4_K_M on in-house hardware. For higher accuracy, consider a dedicated GPU instance in a compliant cloud region with contractual data isolation.

Suggested stack: Llama 3.1 70B Q3_K_M (24–32 GB VRAM) for analysis, Qwen 2.5 7B Q4_K_M for extraction and classification.

Building an AI coding assistant

Use a specialised coding model for generation and review. For in-IDE completions, a local quantised Code Llama or Qwen-Coder 7B gives sub-second latency. For complex refactoring across files, use Claude Sonnet with full-file context.

Suggested stack: Local Qwen-Coder 7B Q4 (completions) + Claude Sonnet 4 (complex refactoring, code review).

Methodology

• Data checked: 2026-06-16
• Sources consulted: Provider model cards and pricing pages (OpenAI, Anthropic, Google DeepMind, Meta, DeepSeek, Alibaba Cloud, Mistral AI), LiveBench leaderboard, Chatbot Arena (lmsys), SWE-bench leaderboard, published benchmark papers (arXiv)
• Assumptions: Model quality rankings assume current model versions as of June 2026. New model releases and version updates can shift comparisons significantly. The decision framework (task type → model family) is designed to hold regardless of specific model names, but benchmark scores and pricing should be verified against current data
• Limitations: This guide does not cover fine-tuning strategy, provider-specific API features, or deployment infrastructure. Pricing excludes volume discounts, commitment tiers, and enterprise agreements. Model availability varies by region and account tier. This is not procurement advice — always test on your own workload before committing to a model
• Jurisdiction: Global. Regulatory requirements for data privacy (GDPR, HIPAA, PIPL) may further restrict model choice in specific jurisdictions

Source list

• OpenAI Model Overview — https://platform.openai.com/docs/models (accessed 2026-06-16)
• Anthropic Claude Model Card — https://docs.anthropic.com/en/docs/about-claude/models (accessed 2026-06-16)
• Google Gemini Models — https://ai.google.dev/gemini-api/docs/models (accessed 2026-06-16)
• DeepSeek Model Documentation — https://api-docs.deepseek.com/ (accessed 2026-06-16)
• Meta Llama 3.1 Model Card — https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3/ (accessed 2026-06-16)
• LiveBench Leaderboard — https://livebench.ai/ (accessed 2026-06-16)
• LMSYS Chatbot Arena — https://lmarena.ai/ (accessed 2026-06-16)
• SWE-bench Leaderboard — https://www.swebench.com/ (accessed 2026-06-16)
• Qwen 2.5 Model Card — https://qwen.readthedocs.io/en/latest/ (accessed 2026-06-16)

Trust Stack

• AI draft model: gpt-5.4-mini
• AI review model: deepseek-v4-pro
• Human editorial review: No (automated editorial pipeline)
• Last substantive check: 2026-06-16
• Corrections policy: If you spot an error, contact us via the Contact page
• Affiliation: theLLMs has no vendor affiliation, sponsorship, or commercial relationship with any AI provider mentioned

Related guides

• How to choose an LLM provider: decision framework for teams
• Model routing: using cheap models first without breaking quality
• Creating a model scorecard for your own workload
• Fine-tuning vs prompting vs RAG: decision checklist

Change log

• 2026-06-16: first published — decision tree covering reasoning, frontier, coding, local, and cheap API models with mid-2026 price and capability data