RAG Evaluation: Measuring Retrieval Quality in Production

TL;DR

Production RAG systems silently degrade over time as schemas shift, indexes update, and new data sources are added — making continuous retrieval evaluation essential, not optional. This article covers the complete production evaluation stack: five core retrieval metrics (recall@K, MRR, nDCG, context precision, context recall), frameworks that separate retrieval failures from generation failures (RAGAS, DeepEval, TruLens, Phoenix), production observability platforms (Langfuse, Arize Phoenix, MLflow), and CI/CD gates that catch regressions before they reach users. The key insight is that fixing the wrong component — retrieval vs. generation — is the most common cause of wasted engineering effort in RAG debugging, and proper separation is the single most impactful step operators can take.

Retrieval-Specific Metrics: The Core Scorecard

Retrieval quality is the foundation upon which every downstream RAG decision rests — a weak retrieval signal cannot be rescued by a stronger generation model. The standard scorecard for retrieval consists of five complementary metrics, each capturing a different facet of how well a retrieval system surfaces relevant context.

Recall@K measures the fraction of truly relevant documents that appear among the top-K retrieved results. It is the simplest and most widely reported metric: if a query has three ground-truth relevant documents and two of them appear in the top ten, recall@10 is 0.67. Its primary weakness is that it depends heavily on ground-truth coverage — if the evaluation corpus does not contain all relevant documents, recall@K will systematically underestimate retrieval quality. In production, this means recall numbers are a lower bound until the corpus is exhaustively annotated.

Mean Reciprocal Rank (MRR) focuses on the position of the first relevant result rather than the total count. For a query where the first relevant document appears at rank 3, the reciprocal rank is 1/3 ≈ 0.33. MRR is ideal for single-answer queries — the kind of question where a user wants one right answer at the top, not a bibliography. It is sensitive to ranking order but blind to what happens after the first hit.

nDCG (normalized Discounted Cumulative Gain) is the gold standard for multi-answer RAG scenarios. It assigns graded relevance to each retrieved document, applies an exponential decay by position (documents at rank 1 contribute more than those at rank 10), and normalizes against an ideal ranking. nDCG rewards systems that correctly rank multiple relevant documents in order of relevance, making it the most informative single metric for complex, multi-document queries.

Context Precision evaluates how well a retriever ranks relevant documents above irrelevant ones. It is computed as the mean precision@K across all relevant chunks in the retrieved set — essentially, it measures whether the relevant documents are clustered toward the top of the results rather than scattered among noise. When context precision is low, the generation model must work harder to ignore irrelevant passages that have been injected into the prompt window.

Context Recall measures the inverse: what fraction of the ground-truth facts needed to answer a query are recoverable from the retrieved context. If a query requires information from five documents and the retrieval step surfaces only three of them, context recall is 0.6. Together, context precision and context recall give a complete picture of whether the retrieved context is both sufficient and focused.

These metrics should be computed on a representative production query set — not a synthetic benchmark — and re-run after every index or schema change to ensure retrieval quality does not silently degrade.

Separating Retrieval from Generation Quality

RAG output quality conflates two distinct failure modes: retrieval may be poor (wrong documents surfaced) or generation may be poor (correct documents but bad synthesis). Isolating retrieval signals is essential for targeted debugging — you cannot tell whether to fix the index or tune the prompt without first decoupling the two.

RAGAS (Retrieval-Augmented Generation Assessment) is the most widely adopted framework for this separation. It computes context precision and context recall independently of the generated answer, scores faithfulness (whether the answer is grounded in the retrieved context), and measures answer relevance against ground truth. RAGAS supports custom grounding annotations and runs as a lightweight Python library or managed service.

DeepEval takes a similar approach but integrates more tightly with CI pipelines. It provides retrieval-specific tests that check context relevance and faithfulness, making it straightforward to block merges that regress retrieval quality. Its test suite structure maps directly to pytest, which lowers the barrier for teams already using Python testing infrastructure.

TruLens adds retrieval-aware feedback functions that attribute quality drops to specific RAG stages. Instead of reporting a single end-to-end score, TruLens lets you write custom feedback functions that evaluate retrieval separately from generation, then aggregate results into a per-stage quality profile.

Phoenix (Arize) provides end-to-end tracing that separates retrieval and generation spans within a single trace. It automatically scores context quality and detects drift between retrieval runs, enabling per-component metric dashboards. Phoenix’s Python-native design makes it easy to embed in existing evaluation pipelines.

The practical takeaway: proper separation lets operators answer the question “is this a retrieval problem or a generation problem?” before investing engineering time. Fixing the wrong component is the most common cause of wasted effort in RAG debugging.

Production Observability Platforms

Retrieval metrics computed on static test sets tell only part of the story. Production RAG systems encounter query distributions, schema changes, and data source additions that no offline test set fully captures. Production observability platforms close this gap by ingesting live traces, computing metrics continuously, and surfacing anomalies before they become user-facing regressions.

Langfuse is an open-source LLM observability platform with built-in RAG trace support. It logs retrieval events, context windows, and generation outputs, and provides custom evaluation metric definitions that can run against production traffic. Langfuse’s real-time dashboards and alerting make it practical to set SLA thresholds for retrieval metrics and get notified when they breach. It integrates with LangChain, LlamaIndex, and other common RAG frameworks.

Arize Phoenix is a lightweight, Python-native observability tool with built-in RAG evaluation. It automatically ingests traces, computes context quality scores, and detects drift between retrieval runs. Phoenix’s emphasis on lightweight deployment — it can run on a laptop or a single container — makes it attractive for teams that need evaluation without heavy infrastructure.

MLflow is an experiment tracking platform that has added LLM evaluation capabilities. It supports custom metrics, model registry integration, and full pipeline versioning, making it suitable for teams that need to version their evaluation data alongside their retrieval pipeline. MLflow’s strength is reproducibility: every evaluation run is tied to the exact schema, data source version, and query set that produced it.

When selecting a platform, evaluate four capabilities: automatic trace ingestion (can it connect to your existing RAG stack?), metric customizability (can you define your own retrieval metrics?), alerting and SLA enforcement (can you set thresholds and get notified?), and CI/CD integration (can it run evaluations as part of your deployment pipeline?).

Building an Evaluation Pipeline

A production evaluation pipeline turns retrieval metrics from ad-hoc experiments into a repeatable, automated process. Here is how to build one from scratch.

Step 1: Build a golden dataset. Collect 50–200 representative queries from production logs — real user queries, not synthetic ones. Annotate each with ground-truth answers and the set of relevant documents. This dataset becomes your evaluation anchor. Quality matters more than quantity: 100 well-annotated production queries are more valuable than 1,000 synthetic ones.

Step 2: Automate metric computation. Configure your evaluation pipeline to run recall@K, MRR, nDCG, context precision, and context recall against the golden set on every code change, index update, or schema modification. The pipeline should output a structured report with per-metric scores and a pass/fail decision.

Step 3: Gate your CI pipeline. Set minimum thresholds for each metric — for example, recall@20 ≥ 0.85, MRR ≥ 0.70, context precision ≥ 0.60 — and block merges that regress below these thresholds. Thresholds should be calibrated against your current production performance; they should catch meaningful degradation, not statistical noise.

Step 4: Version everything. Use MLflow or a similar tool to version your evaluation data, metrics, and retrieval pipeline code. Every evaluation run should be reproducible: given the same golden set, pipeline version, and index snapshot, you should get the same metrics. This is essential for diagnosing regressions and comparing A/B configurations.

Step 5: Schedule periodic re-evaluation. Set up a daily or weekly evaluation against a rolling window of anonymized production queries. This catches drift between schema changes — queries that were fine yesterday may behave differently after a data source update, and only continuous re-evaluation will surface these changes.

CI/CD Gates and Automated Regression Detection

A retrieval evaluation pipeline is only valuable if it actually prevents regressions from reaching users. CI/CD gates and automated regression detection close the feedback loop between evaluation and deployment.

Pre-merge gates run the full evaluation suite on every pull request that touches retrieval code, vector schema, or data connectors. The gate should fail fast: if recall@K drops below the threshold on a PR, the merge is blocked and the author sees exactly which metric regressed and by how much. This prevents retrieval degradation from ever reaching the main branch.

Post-deployment alerting monitors live production metrics and triggers alerts when they breach SLA thresholds. Platforms like Langfuse and Phoenix support custom dashboards with threshold-based alerts that can hook into PagerDuty, Slack, or email. The key is to monitor the same metrics you evaluate on offline — recall@K, MRR, context precision — so production alerting is directly comparable to pre-merge evaluation.

Shadow evaluation routes a percentage of live traffic through a parallel evaluation pipeline that scores queries without affecting user experience. The shadow pipeline computes retrieval metrics in real time, comparing the production retrieval against the golden dataset. Because the results never reach the user, shadow evaluation lets you measure the impact of a new retrieval configuration before promoting it to production.

Regression rollbacks combine metric thresholds with automated rollback triggers. If a production deployment causes a retrieval metric to breach its threshold, the system should automatically roll back to the previous configuration. The goal is detection-to-reversal in minutes, not hours.

The end state is a fully automated retrieval quality loop: every change is evaluated, every production query is monitored, and every regression is caught before users notice. This transforms retrieval evaluation from a periodic manual exercise into a continuous, automated quality assurance system.

Methodology

Data checked: 2026-07-27
Sources consulted: RAGAS documentation, DeepEval documentation, TruLens documentation, Arize Phoenix documentation, Langfuse documentation, MLflow documentation, MTEB leaderboard
Assumptions: Tool capabilities and documentation reflect public information as of July 2026; threshold values (recall@20 ≥ 0.85, MRR ≥ 0.70, context precision ≥ 0.60) are illustrative and should be calibrated to each team’s production baseline.
Limitations: This guide does not cover prompt engineering, chunking strategies, or embedding model selection — those are separate domains that affect retrieval quality but are not the focus of this article. It also does not provide step-by-step implementation code for any of the mentioned platforms.
Jurisdiction: Global.

Source list

RAGAS — https://docs.ragas.io/ (accessed 2026-07-27)
DeepEval — https://docs.confident-ai.com/ (accessed 2026-07-27)
TruLens — https://trulens.org/ (accessed 2026-07-27)
Arize Phoenix — https://docs.arize.com/phoenix/ (accessed 2026-07-27)
Langfuse — https://langfuse.com/ (accessed 2026-07-27)
MLflow — https://mlflow.org/ (accessed 2026-07-27)
MTEB Leaderboard — https://huggingface.co/spaces/mteb/leaderboard (accessed 2026-07-27)

Trust Stack

Last substantive check: 2026-07-27
Corrections policy: If you spot an error, contact us via the Contact page
Affiliation: theLLMs has no vendor affiliation, sponsorship, or commercial relationship with any AI provider mentioned

Change log

2026-07-27: first published