Evaluation dataset design: creating high-quality test sets for RAG, agents, and LLM products

Most teams treat the evaluation dataset as a one-off artefact. They write a few dozen test cases before launch, run evaluations during development, and then leave the dataset to rot. Six months later, the product has changed, user behaviour has shifted, and the eval set is measuring something that no longer exists.

A well-designed evaluation dataset is a living asset. It surfaces regressions, captures production failures as test cases, and tells you whether your model or prompt is actually improving — not just whether the same stale examples still pass.

This guide covers the design decisions that determine whether your eval dataset produces useful signal or comfortable noise: golden data strategy, annotation methods, version control, contamination avoidance, and the specific challenges of evaluating RAG systems and agent workflows.

Before reading this, make sure you have read the site’s building an evaluation pipeline guide — that covers the overall pipeline structure that this dataset design feeds into.

TL;DR

Design your evaluation dataset as three layered collections: a small golden set of hand-verified canonical examples (30–100 cases), a larger automated regression set (200–500 cases that cover known edge cases and past failures), and a living held-out sample of production traffic for drift monitoring. Version each collection independently, label every case with its source and creation date, and run a contamination check whenever a dataset overlaps with your training data or fine-tuning set.

What makes a good eval dataset

An evaluation dataset is not a random sample of user inputs. It is a curated collection designed to test specific failure modes. A good dataset surfaces differences between prompt or model versions. A bad dataset produces the same pass rate regardless of what you change.

The three signals a dataset must produce

If your dataset fails on any of these three, the scores it produces are not reliable enough to make product decisions.

Structuring your dataset

A single monolithic eval set is too inflexible. Split it into three tiers that serve different purposes.

Tier 1: Golden dataset (30–100 cases)

The golden set is your ground truth. Every case is hand-written or hand-reviewed, annotated with the expected behaviour, and frozen until a deliberate decision is made to change it.

What belongs here:

• Canonical examples of the most common user request patterns
• Known edge cases that must not regress
• Safety boundary examples that should always be refused or handled with specific care
• A few examples that the current model handles poorly — these are your improvement targets

How to maintain it:

• Version the golden set like software (v1.0, v1.1). Every change requires a documented reason.
• Run the golden set against every release. A regression here is a deployment blocker.
• Review the golden set quarterly. Remove cases that no longer match the product scope. Add cases for new critical behaviours.

What does not belong here:

• Randomly sampled traffic without review — those go in Tier 3.
• Cases that are too easy and never fail — they inflate scores without improving signal.

Tier 2: Regression set (200–500 cases)

The regression set is your early warning system. It should cover all known failure modes and edge cases that the team has encountered in production, plus automatically generated cases from your generation pipeline.

What belongs here:

• Every production incident converted into a test case (with redaction)
• Synthetic cases that have been manually reviewed and accepted
• Cases from the golden set’s generalisation — variations on the canonical patterns
• Known model weaknesses surfaced by periodic manual audits

How to maintain it:

• Add cases aggressively. Every production bug, user complaint, or manual audit finding becomes a test case.
• Remove cases sparingly. A case that has passed for six months may be stale, but removing it without analysis risks missing a regression that the case was uniquely positioned to catch.
• Track pass rates by case source: golden vs synthetic vs production. A divergence in pass rates between sources is a diagnostic signal.

Tier 3: Held-out production sample (variable size)

This is not a curated dataset. It is a statistically representative sample of real user traffic that you run through evaluation periodically to check for drift that your curated sets might miss.

What belongs here:

• A random sample of production inputs, stratified by source, length, and user segment
• No manual review before evaluation — you evaluate blind and review outliers afterwards
• Rotated regularly so the sample does not become stale

How to maintain it:

• Evaluate against this sample weekly or monthly, not per-release.
• Compare pass rate distributions between the curated sets and the production sample. If the production sample scores significantly lower, your curated sets have coverage gaps.
• Do not add production sample cases to your curated sets without manual review. Direct transfer introduces the distributional bias of your current traffic pattern without the quality check.

Annotation strategies

Every test case needs an annotation: what is being tested, what the expected behaviour is, and which evaluation dimensions apply. The quality of this annotation determines whether your automated evaluation can actually use the dataset.

What every test case should include

{
  "id": "eval-golden-0042",
  "input": "What is my current balance?",
  "context": "User has called customer support. Agent has authenticated them. The account is in good standing.",
  "expected_behaviour": "The output should state the current balance in pounds and pence. If the balance includes pending transactions, state that clearly.",
  "dimensions": ["factuality", "usefulness"],
  "pass_condition": "Output contains a numeric balance that matches the provided account data. Output does not show pending transactions as settled unless explicitly noted.",
  "source": "human-written",
  "created": "2026-05-01",
  "creator": "sme-dom"
}

Each field serves a specific purpose:

• id — unique and stable. Do not reuse IDs when replacing cases.
• input — the user-facing prompt or query. This is what gets sent to the model.
• context — any system information, retrieved documents, or conversation history that the model receives alongside the input. Critical for RAG and agent test cases.
• expected_behaviour — what a correct output looks like, described functionally rather than as an exact string. This allows for variation in phrasing while testing intent.
• dimensions — which evaluation dimensions this case tests. A single case can test multiple dimensions.
• pass_condition — the specific criteria that determine whether the output passes for each dimension. This is the machine-readable or reviewer-readable rule.
• source — how the case was created: human-written, synthetic, production-converted, etc.
• created — creation date, not last-modified date.
• creator — who or what created the case. Useful for spotting drift in synthetic generation quality over time.

Common annotation mistakes

Over-specifying the expected output. If the annotation includes the exact expected text, the eval becomes a string-matching exercise. Models that produce correct but differently-phrased outputs fail for the wrong reasons. Describe the expected behaviour, not the expected wording.

Under-specifying the context. For RAG systems, the same input can produce very different outputs depending on what was retrieved. A test case without context documents tests the model-in-the-abstract, not your RAG system. Always include the retrieved context the model actually receives.

Missing the pass condition. Without a pass condition, human reviewers guess what “good” means and produce inconsistent scores. Without a machine-readable pass condition, automated evaluation cannot use the case.

Version control for eval datasets

Your eval dataset should be versioned alongside your prompts and model configurations — not stored separately in a spreadsheet that only one person knows how to update.

Store in version control, not a database

Each test case is a file. Each tier is a directory. The dataset version is the git commit hash.

evals/
  golden/
  golden-0001.yaml
  golden-0002.yaml
  regression/
  regression-0001.yaml
  regression-0002.yaml
  schemas/
  test-case-v1.yaml

File-based storage gives you:

• History — every change is visible in git log.
• Review — changes to eval cases go through pull requests, just like code.
• Blame — you can see who added a case and why.
• Reproducibility — a given git commit produces a deterministic eval run.

Track changes explicitly

When a test case changes, the commit message should say why:

feat(eval): add production regression for balance-query-after-account-closed

Incident #4392: model returned "Current balance: £0.00" for a closed account
with outstanding charges. The correct behaviour is to indicate the account
is closed and direct the user to billing. Added as golden-0042.

When a test case is removed, the commit message should say why it is safe to remove:

chore(eval): retire golden-0018 — product no longer supports bulk queries

Added 2025-11 for the bulk-import feature that was removed in Q2 2026.
All other bulk-query edge cases are covered by regression-0022 through
regression-0027.

Link datasets to model or prompt versions

Your eval config should specify exactly which dataset version to use for a given evaluation run:

evaluation:
  dataset: "evals/at-commit/a1b2c3d"
  model: "claude-sonnet-4-20260501"
  prompt: "prompts/production-v12.yaml"

This makes every evaluation run reproducible. Six months later, you can re-run the exact same evaluation and compare results, as long as the dataset and prompt configs are still valid.

Avoiding contamination in team-built eval sets

The contamination problem everyone knows about is benchmark data leaking into model training data. There is a second contamination problem that is equally damaging: your own evaluation examples leaking into your fine-tuning or prompt-crafting feedback loop.

What contamination looks like for your own eval set

Test cases used in few-shot prompts. A team includes evaluation examples in prompt few-shot examples to improve accuracy. The model now sees eval data during inference. The eval pass rate jumps because the model has seen the questions before — but real user requests, which were not in the few-shot set, produce worse results.

Test data mixed into fine-tuning datasets. A team curates a high-quality training set for fine-tuning, and includes eval-golden cases “because they are well-written examples.” The fine-tuned model scores higher on eval because it was trained on the eval data, not because it learned the underlying capability.

Prompt engineering guided by eval scores. A team iterates on prompts using eval scores as the sole success metric. The prompts eventually optimise for the eval set — producing outputs that match the expected_behaviour annotations but fail on real user inputs that differ even slightly. This is the eval equivalent of overfitting.

Practical prevention

1. Isolate eval data from training and prompt data. Store evaluation datasets in a separate repository or directory tree that never feeds into training pipelines or prompt templates. Automate a cross-check that warns if any eval ID appears in training or prompt data.

2. Version-lock eval data before training starts. Once a fine-tuning run begins, freeze the eval dataset version. Any changes during training invalidate the eval results.

3. Hold back a secret sample. Before starting active development, set aside 10–20 hand-crafted eval cases that nobody on the team sees — not the prompt engineers, not the fine-tuning pipeline. Reveal them only for final evaluation. If the team’s eval score is high but the secret-sample score is low, you have contamination or overfitting.

4. Use temporal separation. Create evaluation examples after the training data freeze date. If your training data includes everything up to 2026-04-01, create eval examples from 2026-04-02 onward. This guarantees no training-data contamination.

5. Track n-gram overlap. Before running an evaluation, check whether any eval input shares more than 80% of its bigrams with any training example. Flagged cases should be reviewed for contamination risk.

Special considerations for RAG eval datasets

RAG evaluation has unique challenges because the model’s output depends on what is retrieved — and retrieval quality varies independently of generation quality.

What a RAG test case must include

A RAG test case is not just a query and an expected answer. It also includes the source documents or retrieval context that the model will receive:

id: "rag-regression-0031"
query: "What is the return policy for opened electronics?"
retrieved_chunks:
  - "Returns for unopened items within 30 days with receipt."
  - "Opened electronics must be returned within 14 days and may incur a restocking fee of up to 15%."
  - "Items without a receipt can only be exchanged for store credit."
expected_behaviour: "The output should cite the 14-day window and restocking fee for opened electronics. It should distinguish this from the unopened-items policy."
dimensions: ["factuality", "citation_quality"]

Why this matters: If you test a RAG system without providing the retrieval context, you are testing the model’s inherent knowledge — not your RAG system’s ability to answer from retrieved documents. A high score in that scenario tells you the model knows the answer from training, which is not the same as your RAG pipeline working correctly.

Testing retrieval and generation separately

A RAG eval dataset should have two evaluation modes:

1. Oracle retrieval eval — provide the correct documents directly, bypassing the retrieval step. This tests whether the generation component can correctly use the provided context.
2. Full pipeline eval — run the actual retrieval step. This tests end-to-end quality.

If oracle-mode scores are high but full-pipeline scores are low, the problem is in retrieval. If both are low, the problem is in generation or the dataset itself.

Coverage requirements for RAG

RAG test cases should cover:

• Single-chunk answers — the needed information is in one retrieved chunk.
• Multi-chunk synthesis — the answer requires combining information from multiple chunks.
• Conflicting chunks — different retrieved chunks give different information, and the model must handle the conflict.
• Missing information — no retrieved chunk contains the answer. The model should say so, not hallucinate.
• Out-of-date context — retrieved chunks contain information that was accurate at publication but has since changed.

Special considerations for agent eval datasets

Agent evaluation is harder than RAG evaluation because the output is not a single response but a sequence of tool calls, observations, and decisions.

What an agent test case must include

An agent test case captures the initial user request, the available tools, the expected tool call sequence, and success criteria for each step:

id: "agent-regression-0012"
task: "Book a return flight from London to Edinburgh for next Tuesday, departing after 2pm and returning Thursday evening."
available_tools:
  - "search_flights(origin, destination, date)"
  - "book_flight(flight_id, passenger_details)"
  - "cancel_booking(booking_reference)"
  - "check_availability(flight_id)"
expected_tool_sequence:
  - step: 1
  tool: "search_flights"
  expected_params: {"origin": "LHR", "destination": "EDI", "date": "next Tuesday"}
  - step: 2
  tool: "search_flights"
  expected_params: {"origin": "EDI", "destination": "LHR", "date": "next Thursday"}
  - step: 3
  tool: "book_flight"
  expected_params: {"flight_id": "any valid flight_id for the Tuesday outbound"}
success_criteria:
  - "Both flights are booked before the task is marked complete."
  - "The user is informed of the booking reference for each flight."
end_state: "Flights are confirmed. User has booking references."

Evaluation dimensions unique to agents

Ground-truth dataset for agents

Agent eval datasets are more expensive to build because each test case requires defining the expected tool call sequences — which can branch based on intermediate results. Start with 10–20 hand-written agent scenarios and grow only after you have validated that the evaluation format captures meaningful differences. A badly-designed agent eval set can produce the same scores for a perfectly-functioning agent and a broken one because the expected sequences are too rigid.

Methodology

• Data checked: 2026-05-30
• Sources consulted: Evaluation methodology documentation (Promptfoo, DeepEval, OpenAI Evals), academic literature on benchmark contamination (Sainz et al. 2023, Jiang et al. 2024), RAG evaluation frameworks (Chen et al. 2024, Es et al. 2024), and agent evaluation patterns documented by LangChain, Anthropic, and OpenAI.
• Assumptions: Dataset design quality depends on team investment in annotation and version control. A team with no dedicated evaluation resource may need to start with a minimal golden set and build infrastructure incrementally. The three-tier structure assumes a team with at least one person responsible for evaluation quality.
• Limitations: This guide covers general evaluation dataset design principles and contamination prevention for team-built eval sets. It does not address benchmark-level decontamination or large-scale academic eval dataset design. Specific annotation formats and tool choices should be adapted to the team’s existing infrastructure.
• Jurisdiction: Global. Evaluation dataset design methodology is jurisdiction-agnostic; specific requirements for evaluation rigour, human oversight, and dataset representativeness may apply under the EU AI Act, UK Online Safety Act, or sector-specific rules for medical, legal, or financial applications.

Source list

• Promptfoo eval dataset documentation — https://www.promptfoo.dev/docs/guides/eval/ (accessed 2026-05-30)
• DeepEval synthetic data — https://docs.confident-ai.com/docs/synthetic-data (accessed 2026-05-30)
• OpenAI Evals generation patterns — https://platform.openai.com/docs/guides/evals (accessed 2026-05-30)
• LangChain agent evaluation guide — https://docs.langchain.com/docs/guides/evaluation/ (accessed 2026-05-30)
• Anthropic evaluation strategies — https://docs.anthropic.com/en/docs/evaluate (accessed 2026-05-30)
• Sainz et al., “NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for Each Benchmark,” EMNLP 2023 — https://arxiv.org/abs/2310.18018 (accessed 2026-05-30)
• Jiang et al., “Contamination: A Critical Analysis of Benchmark Contamination in Large Language Models,” 2024 — https://arxiv.org/abs/2402.06467 (accessed 2026-05-30)
• Chen et al., “RAGAS: Automated Evaluation of Retrieval Augmented Generation,” 2024 — https://arxiv.org/abs/2309.15217 (accessed 2026-05-30)
• Es et al., “Correctness and Faithfulness Evaluation for RAG Systems,” 2024 — https://arxiv.org/abs/2401.10062 (accessed 2026-05-30)

Trust Stack

• Last checked: 2026-05-30
• Corrections: Contact us to report errors

Change log

• 2026-05-30: First published. Covers evaluation dataset design for RAG, agents, and LLM products: three-tier dataset structure, annotation strategies, version control, contamination prevention, and domain-specific considerations for RAG and agent eval sets.

Related guides

• Building an LLM evaluation pipeline: from prompts to production
• LLM-as-a-judge: when automated grading helps and when it lies
• Human evaluation for LLMs: rubrics that editors and SMEs can actually use
• Synthetic eval datasets: useful shortcut or false confidence?
• Contamination and leakage: why benchmark scores can be too good
• LLM-as-Judge vs human evaluation: cost, accuracy, bias trade-offs