The- most- common- failure- mode- for- eval- CI- is- not- that- the- tests- are- wrong- —- it- is- that- they- are- too- slow.- If- your- eval- suite- takes- more- than- 5- minutes,- developers- will- skip- it.- A- fast,- representative- subset- of- tens- of- examples- that- catches- 90%- of- regressions- is- worth- more- than- a- comprehensive- suite- that- nobody- runs.

##- What- this- means

The- point- of- eval- CI- is- not- perfect- scientific- measurement.- The- point- is- to- make- quality- drift- visible- before- users- find- it.- That- means- the- suite- has- to- be- repeatable,- fast- and- tied- to- the- release- cadence.

A- minimal- example:- a- GitHub- Actions- workflow- that- runs- promptfoo- eval- against- a- small- regression- set- (tens- of- examples,- not- thousands)- on- every- pull- request- that- touches- a- prompt- file.- If- the- pass- rate- drops- below- an- agreed- threshold- —- say- 80%- on- the- core- task- —- the- workflow- returns- a- non-zero- exit- code- and- the- PR- cannot- merge.- Be- cautious- with- synthetic- datasets:- they- are- fast- to- generate- but- can- create- false- confidence- if- the- synthetic- distribution- does- not- match- real- user- inputs.- Promptfoo’s- docs- cover- this- exact- pattern,- and- DeepEval- offers- a- similar- integration- with- its- own- assertion- framework.- Neither- requires- a- dedicated- server;- both- run- as- a- single- CI- step.

#- .github/workflows/eval-ci.yml- —- minimal- example
name:- prompt-eval
on:
- - pull_request:
- - paths:- ['prompts/**']
jobs:
- - eval:
- - runs-on:- ubuntu-latest
- - steps:
- - uses:- actions/checkout@v4
- - run:- npm- install
- - run:- npx- promptfoo- eval- --config- promptfooconfig.yaml
- - run:- npx- promptfoo- assert- --threshold- 0.8

The- suite- should- cover- the- product’s- most- common- failure- modes:- hallucination- on- known- facts,- refusal- on- safe- queries,- instruction-following- drift.- OpenAI- Evals- and- Anthropic’s- evaluation- guide- both- cover- how- to- design- these- small- regression- sets- —- pick- the- scenarios- that- hurt- most- when- they- break.

##- Where- teams- get- it- wrong

Running- evals- after- release- instead- of- before.- A- team- ships- a- prompt- update- on- Friday,- runs- the- eval- suite- on- Monday- as- a- “quality- check,”- finds- a- regression- that- affects- 5%- of- responses,- and- has- no- clean- rollback- path.- By- then- the- bad- responses- have- enough- already- reached- users,- support- tickets- are- already- filed,- and- the- fix- requires- another- deploy- cycle.- The- eval- suite- was- useful- only- as- a- post-mortem- —- it- told- them- what- went- wrong- but- not- before- the- damage- happened.- Eval- CI- means- the- gate- fires- before- merge,- not- after- deploy.

Using- a- huge- suite- that- is- too- slow- for- everyday- work.- An- engineering- team- builds- a- comprehensive- eval- suite- with- hundreds- of- test- cases- covering- every- known- edge- case.- Each- run- takes- 45- minutes.- Developers- start- skipping- it- before- merging- quick- fixes- —- “it’s- just- a- wording- change”- —- and- the- eval- effectively- becomes- an- occasional- batch- job- rather- than- a- release- gate.- The- suite- was- correct- in- coverage- but- wrong- in- design:- a- fast,- representative- subset- that- runs- in- under- 5- minutes- catches- 90%- of- regressions- and- actually- gets- used.- The- full- suite- can- run- nightly- for- deeper- analysis.

Treating- one- score- as- proof- that- the- whole- feature- is- safe.- A- team- watches- the- overall- pass- rate- hover- at- 92%- across- releases- and- calls- it- good- enough.- But- the- 8%- failure- rate- is- concentrated- in- one- category- —- say,- queries- about- financial- advice- —- which- means- a- specific- safety-critical- failure- is- recurring- silently.- An- aggregate- score- hides- distribution.- The- useful- signal- is- per-category- tracking:- if- the- “financial- advice”- category- drops- from- 95%- to- 80%- while- the- headline- score- barely- moves,- that- is- a- real- regression- the- combined- number- would- miss.