GPU rental vs API pricing: when to self-host
The question sounds simple: should I rent a GPU and run an open model, or just pay the API provider per token?
It is not simple, because the answer flips depending on your throughput, your model choice, your utilisation, and whether you count your own ops time. But the core question — at what volume does renting a GPU become cheaper than paying API pricing — has a defensible numeric answer for most common scenarios.
This article gives you that answer in plain numbers, then shows how to adjust for your specific workload.
TL;DR
Renting a GPU for LLM inference breaks even with API pricing at roughly these daily throughput levels:
| Model class | Break-even range (tokens/day) | Suggested GPU | GPU rental (on-demand) |
|---|---|---|---|
| 7B–8B (quantised Q4) | 8–30 million tokens/day | RTX 4090 or L40S | $0.50–$1.00/hr |
| 8B–14B (quantised Q4) | 5–20 million tokens/day | L40S or A100 | $1.00–$2.50/hr |
| 70B (quantised Q4, needs 35 GB) | 3–12 million tokens/day | A100 80GB | $2.50–$3.50/hr |
| 70B (FP16, needs 140 GB) | 2–8 million tokens/day | 2× A100 or H100 | $5.00–$7.00/hr |
| Frontier-class (405B quantised, ~100 GB) | 1–4 million tokens/day | H100 80GB | $3.50–$7.00/hr |
Below the low end of the range: API is cheaper. Above the high end: GPU rental wins. Inside the range: it depends on utilisation, batching efficiency, and ops cost tolerance.
The caveat: these ranges assume on-demand cloud GPU rental at current market rates (mid-2026). Reserved instances, spot/preemptible instances, or colocated hardware shift the ranges significantly — see the spot pricing section below.
The break-even math
The calculation has two sides.
API cost per day:
tokens_per_day × API_price_per_token
For a typical frontier API (GPT-4.1, Claude Opus class): roughly $2.00/M input tokens, $8.00/M output tokens. For a cheaper API (Gemini Flash, Llama API, GPT-4.1 Mini): roughly $0.15–$0.50/M input, $0.60–$2.00/M output.
GPU cost per day:
GPU_hourly_rate × 24 × GPU_utilisation_factor + ops_labour + storage
The utilisation factor is the killer. A GPU running at 100% utilisation (continuous batching, no idle time) costs exactly the hourly rate. A GPU running at 30% utilisation effectively costs 3.3× more per token.
The break-even formula:
break_even_tokens_per_day = (GPU_daily_cost) / (API_price_per_token × utilisation_factor)Where utilisation_factor is your actual throughput compared to theoretical maximum throughput. A realistic starting point is 0.3–0.5 for most teams (30–50% of theoretical).
Worked examples
Scenario A: Small model, low throughput — 2M tokens/day
Model: Llama 3.2 8B (Q4_K_M), 2M input + 500K output tokens/day.
API route (GPT-4.1 Mini class at $0.40/M input, $1.60/M output):
- Input: 2M × 30 days × $0.40/M = $24/month
- Output: 0.5M × 30 × $1.60/M = $24/month
- Total: $48/month (~$1.60/day)
GPU route (RTX 4090 at $0.75/hr, on-demand):
- GPU: $0.75 × 24 × 30 = $540/month
- With storage/ops: ~$600/month
- Total: ~$600/month
Winner: API by a landslide. At this volume, GPU rental costs 12.5× more. You would need to process 25M+ tokens/day for the GPU to even enter the conversation.
Scenario B: Medium model, high throughput — 20M tokens/day
Model: Llama 3.1 70B (Q4_K_M, ~35 GB), 20M input + 5M output tokens/day.
API route (GPT-4.1 class at $2.00/M input, $8.00/M output):
- Input: 20M × 30 × $2.00/M = $1,200/month
- Output: 5M × 30 × $8.00/M = $1,200/month
- Total: $2,400/month (~$80/day)
GPU route (A100 80GB at $2.50/hr, on-demand):
- GPU: $2.50 × 24 × 30 = $1,800/month
- With storage/ops: ~$2,100/month
- Total: ~$2,100/month
Verdict: Borderline. The GPU route is slightly cheaper on raw numbers (~$300/month savings), but only if utilisation stays above ~60%. If utilisation drops below 40%, the API route wins again. At this throughput level, the decision turns on ops capacity and workload stability, not pure cost.
Scenario C: Large model, batch processing — 50M tokens/day
Model: Llama 3.1 70B (Q4_K_M), 50M input + 10M output tokens/day. Batchable workload (off-peak processing).
API route (GPT-4.1 class):
- Input: 50M × 30 × $2.00/M = $3,000/month
- Output: 10M × 30 × $8.00/M = $2,400/month
- Total: $5,400/month (~$180/day)
GPU route (A100 80GB at $2.50/hr, continuous batching, assume 70% utilisation):
- GPU: $2.50 × 24 × 30 = $1,800/month
- With storage/ops: ~$2,200/month
- Total: ~$2,200/month
Winner: GPU rental, significantly. The GPU saves over $3,000/month — roughly 60% less than the API. At this throughput, even with conservative utilisation estimates, the economics clearly favour self-hosting on rented GPU hardware.
Scenario D: Frontier model, moderate throughput — 5M tokens/day
Model: Llama 4 405B (Q4, ~100 GB), 5M input + 1M output tokens/day.
API route (Claude Opus class at $15/M input, $75/M output):
- Input: 5M × 30 × $15/M = $2,250/month
- Output: 1M × 30 × $75/M = $2,250/month
- Total: $4,500/month (~$150/day)
GPU route (H100 80GB at $4.50/hr, on-demand):
- GPU: $4.50 × 24 × 30 = $3,240/month
- With storage/ops: ~$3,700/month
- Total: ~$3,700/month
Verdict: GPU wins, but with a caveat. The savings (~$800/month, ~18%) are real but smaller in percentage terms than Scenario C. The H100 is expensive, and the API provider’s pricing for frontier models includes a premium for capability, not just compute cost. If the quality gap between the 405B and the API frontier model matters for your use case, the cheaper GPU option may not be the better option.
How GPU choice changes the equation
The GPU you choose determines your cost structure. Here are the current on-demand cloud rates and what each GPU makes practical:
| GPU | On-demand/hr | Models it serves | Realistic throughput (70B Q4) | Break-even API volume (70B Q4) |
|---|---|---|---|---|
| RTX 4090 | $0.50–$1.00 | ≤13B Q4 or 8B FP16 | N/A for 70B | N/A (too little VRAM) |
| L40S (48 GB) | $1.00–$2.00 | ≤30B Q4 or 13B FP16 | N/A for 70B | N/A (too little VRAM) |
| A100 80GB | $2.50–$3.50 | 70B Q4 comfortably | 30–60 tok/s sustained | 3–12M tokens/day |
| H100 80GB | $3.50–$7.00 | 70B Q4, 70B FP16 (with 2×), 405B Q4 | 50–100 tok/s sustained | 2–8M tokens/day for 70B Q4 |
For smaller models (7B–14B), the L40S and RTX 4090 offer the best cost-per-token, but they have strict VRAM limits. A 7B Q4 model fits easily on any of these GPUs; a 70B Q4 model only fits on A100 80GB or H100.
Spot/preemptible instance economics
Spot/preemptible GPU instances cost 60–80% less than on-demand. Using spot pricing:
| GPU | On-demand/hr | Spot/hr | Effective cost change |
|---|---|---|---|
| A100 80GB | $2.50 | $0.60–$1.00/hour | Break-even drops to ~1–4M tokens/day |
| H100 80GB | $4.50 | $1.20–$1.80/hour | Break-even drops to ~1–3M tokens/day |
Spot pricing changes the GPU side dramatically — for batch workloads that can tolerate interruptions, the break-even volumes drop into territory where even moderate-throughput teams can justify self-hosting.
Factors that shift your break-even
What makes GPU rental cheaper (lowers the break-even)
- Continuous batching — vLLM or TGI can process 8–64 simultaneous requests, multiplying throughput 3–10× without extra GPU time. If you batch, you need fewer GPUs.
- Headroom tolerance — if your workload can tolerate 1–5 second cold starts, you can use spot instances and low-utilisation scaling, cutting GPU cost by 50–80%.
- Cached contexts — common system prompts and few-shot examples can share KV cache. Some inference engines serve many users from the same cached prefix, effectively multiplying throughput.
- Already-owned hardware — if the GPU is already in your rack or dev machine, the marginal cost is power and cooling only (~$0.10–$0.30/hr), making self-hosting viable at much lower volumes.
What makes API pricing cheaper (raises the break-even)
- Variable traffic — if your throughput spikes 5× during business hours and drops near zero at night, the GPU sits idle for half the day. The utilisation penalty is severe.
- Ops labour — each hour your engineer spends configuring, troubleshooting or updating inference infrastructure instead of building product is a cost the API route does not have. At senior engineer rates (~$100/hr), 5 hours/month of ops time adds $500 to your GPU bill.
- Multi-model needs — if you need to switch between models hourly or expose multiple architectures, you either need more GPUs or accept throughput degradation from model swapping.
- Quality constraints — if your use case demands the best possible output quality, a self-hosted open model may not match the API frontier model regardless of cost.
Decision framework: 5-question triage
Run through these in order. Stop at the first clear answer.
Q1: Can you estimate your daily token throughput within 2×?
- Yes → continue.
- No → use API. You cannot make a GPU decision without knowing your volume within a factor of two.
Q2: Is your throughput above the lower break-even range for the model size you need?
- For 7B models: above ~8M tokens/day? Go to Q3.
- For 70B models: above ~3M tokens/day? Go to Q3.
- Below those thresholds: use API. You will not save money on GPU rental.
Q3: Can you sustain 50%+ GPU utilisation?
- Yes (batch workload, steady traffic, continuous batching configured) → Go to Q4.
- No (spiky traffic, low volume, variable hours) → use API. The utilisation penalty will erase your expected savings.
Q4: Do you have ops capacity (or tolerance) for inference infrastructure?
- Yes (dedicated platform engineer, or you are comfortable running vLLM/TGI yourself) → Go to Q5.
- No (you have one engineer doing everything, or no operations experience) → use API. The hourly cost of your time likely exceeds the GPU savings.
Q5: Do you have latency, privacy, or compliance requirements that rule out external APIs?
- Yes → GPU rental (or on-premise) is the only option, regardless of cost comparison.
- No → GPU rental likely makes sense if you passed Q2–4. You are in the cost-optimisation zone where self-hosting on rented hardware saves money.
What this page cannot tell you
This page gives you break-even ranges, not your exact break-even. The ranges are correct for mid-2026 on-demand cloud pricing with real-world utilisation assumptions. Your exact numbers depend on:
- Your specific workload token distribution (prompt-heavy vs generation-heavy)
- The inference engine you choose and how well you configure it
- The GPU provider’s current availability and regional pricing
- The rate at which API providers change their pricing (which they do frequently)
The only way to know your exact break-even is to measure your actual API spend over 30 days, set up a benchmark deployment on a comparable GPU, and compare the per-token cost.
For the broader trade-offs between open weights and hosted APIs — including quality, control, customisation and vendor lock-in — see our guide on open weights vs hosted APIs: practical trade-offs.
Methodology
- Data checked: 2026-05-30
- Sources consulted: Cloud GPU pricing from RunPod, Lambda Labs, Vast.ai, Google Cloud and AWS. Provider API pricing from OpenAI, Anthropic, Google, and Mistral. Inference engine throughput data from vLLM and TGI documentation. Model memory requirements from model cards and llama.cpp documentation.
- Worked-example assumptions: Break-even calculations assume on-demand cloud GPU rental at mid-2026 rates, 4-bit quantisation for model weights, continuous batching enabled, and 50% GPU utilisation as a default conservative estimate. Spot/preemptible scenarios called out separately.
- Limitations: Break-even ranges shift with GPU market conditions, provider pricing changes, and regional availability. The ranges assume single-GPU, single-model deployments; multi-node, multi-model scenarios have different cost dynamics. Training workloads are not covered — this article is for inference economics only.
- Jurisdiction: Global. Pricing in USD. GPU availability and pricing vary by cloud region. EU and US regions typically have the widest selection and most competitive pricing.
Source list
- RunPod GPU pricing — https://www.runpod.io/pricing (accessed 2026-05-30)
- Lambda Labs GPU cloud — https://lambdalabs.com/service/gpu-cloud/pricing (accessed 2026-05-30)
- Vast.ai GPU marketplace — https://vast.ai/ (accessed 2026-05-30)
- vLLM documentation — https://docs.vllm.ai/en/stable/ (accessed 2026-05-30)
- Hugging Face TGI — https://huggingface.co/docs/text-generation-inference/en/index (accessed 2026-05-30)
- OpenAI API pricing — https://openai.com/api/pricing/ (accessed 2026-05-30)
- Anthropic pricing — https://www.anthropic.com/pricing (accessed 2026-05-30)
Trust Stack
- Last checked: 2026-05-30
- Corrections: Contact us to report errors
Change log
- 2026-05-30: Editorial review against 16-gate checklist. Fixed reviewedBy model label, shortened description to 150 chars, added slugified IDs to 4 H2s and 6 H3s, corrected Trust Stack model labels.
- 2026-05-30: Initial draft. Break-even ranges based on mid-2026 cloud GPU pricing and provider API rates.