RAG costs: vector database, embeddings, reranking and generation

TL;DR

RAG cost is not just LLM tokens. A real estimate covers four layers: ingestion (document parsing, chunking, embedding, indexing), storage (vector database, metadata, backups), serving (retrieval, optional reranking, generation), and operations (logging, evaluation runs, human review of flagged answers).

The cheapest-looking prototype can become expensive when documents churn or usage grows. A policy handbook that changes once a year costs very little to re-index. A product catalogue that updates daily costs substantially more. The embedding and storage bills are visible on your cloud dashboard; the generation and reranking costs per query are what catch teams off guard.

Where the costs live

Ingestion: the one-off that isn’t

Every document entering a RAG system gets parsed, split into chunks, embedded into vectors, and stored. If your knowledge base is 10,000 chunks at 768 dimensions each, the initial embedding cost with a model like text-embedding-3-small ($0.02/1M tokens) is measured in cents.

The problem is re-ingestion. If 20% of your documents change each month — common for product catalogues, support tickets, or policy updates — you pay that embedding cost again on the changed chunks. You also pay to delete or update the old vectors. Over a year, re-ingestion can exceed the initial indexing cost by 5× or more.

Vector storage: cheap until it isn’t

Most managed vector databases charge by stored vector count or by compute. Pinecone’s free tier covers roughly 100,000 vectors at 768 dimensions. Qdrant Cloud starts at around $25/month for 1M vectors. pgvector costs whatever your Postgres instance costs, which can be near-zero on an existing server.

Storage becomes expensive when teams keep multiple versions of embeddings, store full document copies alongside vectors, or hold vectors at higher dimensions (3,072-dimension embeddings use 4× the storage of 768-dimension ones). The storage line item is rarely the largest cost, but it grows steadily and silently.

Retrieval and reranking

Retrieval — finding the top-N most relevant chunks for a user query — is typically included in the vector database price. The cost is in latency and compute, not a separate line item.

Reranking is different. A reranker re-scores the retrieved candidates using a more precise (and more expensive) model. Services like Cohere Rerank charge per search request — roughly $1–2 per 1,000 searches. If your app handles 100,000 queries a month, reranking adds $100–200/month on top of everything else.

The question is whether reranking improves answer quality enough to justify the cost. For a legal Q&A bot where precision matters, yes. For a general FAQ bot where top-3 retrieval is usually good enough, probably not. The right answer depends on your use case, not on a blog post.

Generation: the sleeper cost

This is where most RAG bills live. Every user query sends retrieved chunks — typically 3–10 chunks at 200–500 tokens each — to the LLM alongside the user’s question and any system instructions.

At GPT-4.1 pricing ($2.00/M input tokens, $8.00/M output tokens), a query that sends 2,500 tokens of retrieved context + 200 tokens of user input costs $0.0054 in input alone. At 100,000 queries/month, that is $540/month before the model writes a single word of output. If the model generates 500 tokens per answer, add another $400/month. The total: roughly $940/month for generation alone.

Observability and evaluation

Logging traces, storing evaluation datasets, and running periodic quality checks add operational cost. Tools like Langfuse or Helicone charge by event volume; at 100,000 queries/month, expect $50–200/month for observability alone. This cost is easy to skip in a prototype and painful to add later when something breaks and you have no logs.

Worked example: monthly cost for a policy Q&A bot

A policy Q&A bot with 10,000 document chunks, updated weekly (5% churn), serving 50,000 queries/month:

If you drop reranking (acceptable for this use case) and trim context to 1,200 tokens by sending only the top 2 chunks, the generation cost drops to ~$240/month and the total falls to ~$335/month — roughly half.

Numbers use provider pricing as of May 2026. GPT-4.1: $2.00/M input, $8.00/M output. Cohere Rerank: $2/1K searches. Actual costs vary with usage patterns, caching, and contract pricing.

Practical ways to control RAG cost

• Chunk with purpose. Smaller chunks produce more precise retrieval and cost less per query. If your chunks average 200 tokens instead of 500, you send less context to the LLM on every query.
• Limit retrieved candidates. Retrieving 10 chunks and reranking to 5 costs more than retrieving 5 and sending 3. Start small and add more only when retrieval quality demands it.
• Cache frequent queries. If 30% of user questions are variations on the same topic, cache the generated answer and skip the full pipeline for those queries.
• Re-index incrementally. Do not re-embed your entire knowledge base when one document changes. Update only the affected chunks.
• Measure before adding reranking. Run a side-by-side comparison with and without reranking on 50–100 real queries. If answer quality does not measurably improve, skip it.
• Use cheaper models for straightforward queries. A simple fact lookup (“what is the refund policy?”) does not need GPT-4.1. Route simpler queries to a cheaper model and reserve the expensive one for complex reasoning.

When RAG is the wrong answer

RAG is not always cheaper or better than the alternatives:

• Small, static knowledge bases under 50 pages — a long system prompt may be simpler, faster, and cheaper than building a full RAG pipeline.
• Highly structured data — if your data lives in a database with clear columns (prices, dates, SKUs), direct SQL queries are more reliable than semantic search.
• Real-time requirements — RAG with embedding lookup + retrieval + generation can take 2–5 seconds end-to-end. If users expect sub-second responses, consider pre-computed answers or a simpler architecture.

Methodology

• Data checked: 2026-05-28
• Sources consulted: Provider pricing pages (OpenAI, Cohere, Pinecone, Qdrant), API documentation on token and embedding billing, vector database pricing tiers, observability tool pricing pages
• Assumptions: Worked example uses GPT-4.1 and Cohere Rerank pricing as of May 2026. Query volumes and chunk counts are illustrative, not measured from production traffic. No caching, batch, or enterprise discount applied.
• Limitations: This article covers cost estimation, not retrieval quality measurement, latency optimisation, or security architecture. Provider pricing changes frequently — re-check before budgeting.
• Jurisdiction: Global. Pricing in USD. Providers referenced are available internationally; regional pricing and data residency may vary.

Source list

• OpenAI pricing — https://openai.com/api/pricing/ (accessed 2026-05-28)
• Cohere Rerank pricing — https://cohere.com/pricing (accessed 2026-05-28)
• Pinecone pricing — https://www.pinecone.io/pricing/ (accessed 2026-05-28)
• Qdrant Cloud pricing — https://qdrant.tech/pricing/ (accessed 2026-05-28)
• Langfuse pricing — https://langfuse.com/pricing (accessed 2026-05-28)

Related guides

• Embeddings explained for business search and RAG
• API model pricing: input, output, cache and batch costs
• Chunking documents for RAG: size, overlap and metadata choices
• Rerankers explained: the quiet quality layer in RAG systems

Trust Stack

• Last checked: 2026-05-28
• Corrections: Contact us to report errors

Change log

• 2026-05-28: Full editorial review against 16-gate checklist: added 3 Editor’s Notes, slugified all H2/H3 IDs, added Methodology and Trust Stack sections, added concrete worked example with pricing table, added source access dates, removed editorial meta-commentary, and expanded description to meet length requirement.
• 2026-05-27: Added direct source URLs to all named providers and services; added Change Log section.
• 2026-05-25: First published.