Building an LLM evaluation pipeline: from prompts to production

Most teams skip evaluation until something breaks. A prompt change quietly degrades output quality, a model update shifts response style, or a feature launch causes a spike in user complaints that nobody caught in testing. By then, the cost of fixing it includes lost users, rework, and trust.

An evaluation pipeline does not eliminate all failures. It makes them visible before users feel them.

This guide walks through the full pipeline: defining what “good” means, building test datasets, choosing metrics, running eval harnesses, and tracking regressions over time. Each section includes practical choices and trade-offs so you can build what fits your team, not a generic template.

TL;DR

Start with a few dozen golden test cases, not thousands. Define 3–5 evaluation dimensions (factuality, usefulness, safety, citation quality) and run automated checks on every prompt or model change. Add human review on a sample for the dimensions automation cannot measure. Track per-dimension pass rates over time and set regression alerts. Expand the dataset gradually as you learn where your model actually fails.

What an evaluation pipeline actually does

An evaluation pipeline is not a test suite in the software engineering sense. There is no single pass/fail check that tells you whether an LLM output is correct. Instead, the pipeline layers several types of checks that together give you a signal: is this version of the prompt or model better, worse, or about the same as the last one?

The core loop is:

Define criteria — what dimensions of output quality matter for your use case.
Build a test dataset — a set of inputs with expected behaviours.
Run automated checks — assertion-based tests, LLM-as-judge scoring, or both.
Sample human review — the parts automation cannot measure.
Track results over time — compare scores across versions, spot drift, set regression thresholds.
Iterate the dataset — add examples from production failures and remove stale ones.

The pipeline should catch regressions before deployment, not after. That means it needs to run in minutes, not hours, and it needs to produce a signal that a product manager or non-technical stakeholder can interpret without a data science degree.

Step 1: Define what good looks like

The most common failure in evaluation is not defining bad criteria — it is using criteria that do not match the real product behaviour.

For a customer support chatbot, “good” means the user gets an accurate answer that helps them resolve their issue quickly. For a code generation tool, “good” means the output compiles, is secure, and achieves the stated goal. These are different concepts of quality, and they need different metrics.

A minimal set of dimensions

Most products can start with 3–5 evaluation dimensions. Fewer than three misses important failure modes. More than five creates cognitive load and measurement noise.

Dimension	What it measures	Typical checks
Factuality	Are claims supported by sources or established facts?	Assertion-based tests for known facts, LLM-as-judge for grounded-ness
Usefulness	Does the output directly address the user’s stated need?	Human review on sample, LLM-as-judge as screening
Safety	Does the output refuse appropriately and avoid harm?	Pattern-based blocking, LLM-as-judge for borderline cases
Citation quality	Are sources cited accurately and relevantly?	Regex/citation extraction checks
Style/tone	Does the output match the expected voice?	Human review on sample

The exact dimensions depend on your domain. A medical summarisation tool needs factuality and safety weighted heavily. A creative writing assistant weights tone and usefulness higher. Start with defaults and refine as you learn where the model underperforms.

Setting thresholds

Define three tiers per dimension:

Pass — output meets the standard. No action needed.
Borderline — acceptable but flagged. Review a sample weekly.
Fail — output does not meet the standard. Block the release until fixed.

Do not set thresholds based on guesswork. Run a baseline evaluation on your current prompt or model, measure the distribution of scores, and set thresholds at meaningful breakpoints — for example, “fail if factuality drops below 80% of the baseline pass rate.”

Step 2: Build a test dataset

The quality of your evaluation is bounded by the quality of your test dataset. A bad dataset produces confident but wrong signals. A good dataset surfaces regressions before they reach users.

Start small, grow deliberately

Begin with 30–50 hand-written test cases representing the most common user patterns. These should cover:

Happy path — typical user requests where the expected output is well-understood.
Edge cases — unusual phrasing, contradictory instructions, very long or very short inputs.
Known failure modes — inputs that caused problems in the past or that the current model handles poorly.
Safety boundaries — requests that should be refused or handled with caution.

Each test case includes: the input, the expected behaviour (not always the exact output), and which dimensions it tests. A test case for factuality might have the input, a known-correct reference answer, and a pass condition (the output matches the reference on key claims). A test case for safety might define the expected refusal pattern.

Add synthetic data with caution

Synthetic generation — using an LLM to produce additional test cases — can expand coverage quickly, but every synthetic example introduces distributional bias. The generation model produces examples that look like its own training data, not necessarily like real user inputs.

If you use synthetic data:

Generate 3× the target size to allow for pruning.
Review every synthetic example manually before adding it to the dataset.
Label synthetic examples separately so you can track pass rates by source.
Maintain at least 20% human-written examples in every dimension.

The site’s guide on synthetic eval datasets covers this in detail.

Version control and provenance

Treat your test dataset like production code. Store it in version control. Each test case has a unique ID, creation date, source (human-written or synthetic generation prompt), and the model/prompt version it was created against. When a test case becomes stale — because the product changed, the domain evolved, or the model’s capabilities shifted — retire it with a note, not a deletion.

Step 3: Run automated checks

Automated evaluation is where the pipeline earns its keep. A well-designed set of automated checks catches most common regressions in minutes without human intervention.

Assertion-based tests

The most reliable automated checks are narrow, verifiable, and unambiguous. They look for specific patterns:

Does the output contain a phone number or email in a PII-masking task?
Does the output contain the word “cannot” when the prompt required a refusal?
Does the output parse as valid JSON for a structured output task?
Does the output include citations from the provided source document?

Assertion-based tests have no ambiguity. They pass or fail deterministically. Use them wherever possible and fall back to LLM-as-judge only for dimensions that require semantic understanding.

LLM-as-judge for broad quality checks

For dimensions like factuality or tone, assertion-based tests are often too narrow. LLM-as-judge — using one model to score another’s outputs — fills the gap but comes with known biases: position bias, verbosity bias, self-enhancement, and rubric neglect.

Use LLM-as-judge for:

Relative comparisons — “did the pass rate drop compared to last week?”
Screening — flag outputs that need human review, not certifying them as good.
Trend tracking — chart scores over time and watch for drift.

Do not use LLM-as-judge for:

Absolute quality certification — “this model is production-ready because the judge gave it 92%.”
Safety-critical decisions — human review is required for regulated domains.
Scores without human validation — calibrate the judge against human review before trusting its outputs.

The site’s guide on LLM-as-a-judge covers the bias patterns and mitigation strategies in depth.

Running the harness

Tools like Promptfoo, DeepEval, LM Eval Harness, and bespoke CI scripts all serve the same purpose: take a dataset, run it through the model or prompt under test, apply the configured checks, and produce a report.

Your harness should:

Run in under 10 minutes for a standard eval set (50–100 test cases).
Output a per-dimension score and a per-test-case breakdown.
Support diffing against a baseline run — “this prompt version changed factuality by -3% compared to last week.”
Integrate into CI so evaluation runs automatically on every prompt or model change.

Step 4: Add human review on a sample

Automated evaluation cannot measure every dimension. It cannot catch tone issues, contextual appropriateness, or subtle factual errors that look plausible but are wrong. Human review fills these gaps.

Sample, do not review everything

Reviewing every output defeats the purpose of automation. Sample 5–10% of outputs per release, stratified by output type and complexity. For high-risk domains (medical, legal, financial), increase the sample rate to 20–30%.

Use structured rubrics

A human without a rubric produces inconsistent, unrepeatable judgements. Use the same five-category rubric from Step 1: factuality, usefulness, safety, citation quality, and tone. Each output gets one score per category (fail/borderline/pass) and a free-text note for the reviewer’s context.

The site’s guide on human evaluation for LLMs provides a complete rubric template and calibration guidance.

Track inter-rater reliability

Two reviewers scoring the same output should agree at least 70% of the time. Measure agreement per category and re-calibrate when it drops below that threshold. Calibration sessions — where reviewers score a shared set of examples and discuss disagreements — are the highest-leverage activity for improving human evaluation quality.

Step 5: Track results over time

Evaluation becomes useful when it has history. A single pass rate tells you almost nothing. A chart of pass rates over releases tells you whether quality is improving, degrading, or flat.

What to track

Per-dimension pass rate — not an aggregate score. An aggregate can hide a growing tail of failures in one dimension.
Baseline comparison — compare each run against a frozen baseline prompt or model version.
Regression count — number of test cases that went from pass to fail in this run.
Judge calibration score — agreement between LLM-as-judge and human review, tracked over time to detect drift in the judge’s behaviour.

Set alerts, not dashboards

A dashboard that nobody checks is decoration. Set alerts that trigger when:

Any dimension drops below the fail threshold.
Human review agreement drops below 70%.
Judge calibration drifts by more than 5% from the previous check.

Alerts should go to the team’s communication channel, not a buried email. A CI failure that sends a Slack message gets attention. One that lives in a dashboard gets ignored.

Step 6: Iterate the dataset

The test dataset is not a fixed artefact. It degrades as the product, model, and user behaviour change.

Add production failures

Every production incident where the model gave a wrong or harmful output should become a test case (with appropriate redaction). Adding regression tests for real failures is how the pipeline learns. Without this feedback loop, the dataset stays frozen while the real failure patterns evolve.

Remove stale cases

A test case that has passed every run for six months is no longer testing anything useful. Retire it with a note about why it was originally added and why it is no longer relevant. Keeping it inflates pass rates and hides true regression detection.

Refresh synthetic data

Synthetic examples should be regenerated and re-reviewed periodically — at least quarterly. The generation model’s distribution drifts over time, and old synthetic data encodes patterns that may not match current model behaviour.

Where teams misuse evaluation pipelines

Building the pipeline and never using it. A team spends weeks designing a perfect evaluation framework, writes 1,000 test cases, configures a CI job — and then nobody looks at the results. The pipeline exists but does not produce decisions. Evaluation without a feedback loop is theatre.

Using evaluation as a once-off audit. A team evaluates the model before launch, gets a pass, and never runs evaluation again. Three months later, a model update degrades quality by 15% and nobody notices because the evaluation pipeline was never integrated into the development cycle.

Chasing a single aggregate score. A team optimises for an overall “quality score” of 95% and ignores per-dimension degradation. They tweak prompts to increase the aggregate number while factuality quietly drops from 92% to 85% — a dangerous loss hidden by a flat average.

Treating human review as optional. A team builds an entirely automated pipeline, declares it production-ready, and discontinues human review. When the LLM-as-judge model drifts and starts scoring badly-written but plausible-sounding outputs higher than accurate but concise ones, the pass rate stays high while real quality drops.

Practical decision check

Before deploying your evaluation pipeline:

Are the evaluation dimensions derived from what users need, not from what is easy to measure?
Does the test dataset include real user examples or is it entirely synthetic?
Is evaluation integrated into CI so it runs on every prompt or model change?
Are per-dimension scores tracked separately, not aggregated into one number?
Is there a monthly calibration check for the LLM-as-judge model?
Is there a process for converting production failures into new test cases?
Are alerts configured for regression thresholds, or does someone have to remember to check the dashboard?

Methodology

Data checked: 2026-05-29
Sources consulted: Evaluation tool documentation (Promptfoo, DeepEval, OpenAI Evals, LM Eval Harness), academic literature on LLM evaluation methodology (Ribeiro et al. 2020, Gao et al. 2021), Anthropic and OpenAI evaluation guides, and known bias studies (Zheng et al. 2023, Wang et al. 2023).
Assumptions: This pipeline approach assumes a team with access to evaluation tools and the ability to run CI jobs. For teams without CI infrastructure, a manual pipeline with periodic runs provides most of the same benefits. The specific tool choices and dataset sizes should be adapted to the product’s risk profile and team resources.
Limitations: This guide covers general pipeline design principles. It does not provide a production-ready configuration for any specific tool, framework, or deployment environment. The evaluation dimensions and thresholds must be adapted to the specific domain, user base, and regulatory context.
Jurisdiction: Global. Evaluation methodology is jurisdiction-agnostic; specific regulatory requirements for evaluation rigour and human oversight may apply under the EU AI Act, UK Online Safety Act, or sector-specific rules (medical, legal, financial).

Source list

Promptfoo evaluation documentation — https://www.promptfoo.dev/docs/guides/eval/ (accessed 2026-05-29)
DeepEval documentation — https://docs.confident-ai.com/docs (accessed 2026-05-29)
OpenAI Evals guide — https://platform.openai.com/docs/guides/evals (accessed 2026-05-29)
LM Eval Harness repository — https://github.com/EleutherAI/lm-evaluation-harness (accessed 2026-05-29)
Anthropic evaluation guide — https://docs.anthropic.com/en/docs/evaluate (accessed 2026-05-29)
Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” NeurIPS 2023 — https://arxiv.org/abs/2306.05685 (accessed 2026-05-29)
Wang et al., “Large Language Models are Not Fair Evaluators,” arXiv:2305.17926, 2023 — https://arxiv.org/abs/2305.17926 (accessed 2026-05-29)
Ribeiro et al., “Beyond Accuracy: Behavioral Testing of NLP Models with CheckList,” ACL 2020 — https://aclanthology.org/2020.acl-main.442/ (accessed 2026-05-29)
Gao et al., “A Framework for Few-Shot Language Model Evaluation,” 2021 — https://github.com/EleutherAI/lm-evaluation-harness/blob/main/README.md (accessed 2026-05-29)

Trust Stack

Last checked: 2026-05-29
Corrections: Contact us to report errors

Change log

2026-05-29: first draft