#- Latency- in- LLM- apps:- first- token,- total- time- and- user- experience
##- TL;DR
A- slow- AI- feature- is- not- always- a- slow- model.- The- delay- can- come- from- queueing,- a- long- prompt,- a- long- answer,- tool- calls,- network- hops,- retries,- or- a- slow- downstream- service- that- the- model- has- to- wait- for.
The- useful- answer- is- to- measure- the- whole- request- path,- not- just- the- model- call.- If- the- first- token- is- slow,- the- user- feels- stuck.- If- the- total- time- is- slow,- the- feature- feels- heavy.- Both- matter,- and- they- are- not- the- same- problem.
Teams- often- blame- the- model- first- because- that- is- the- visible- part.- In- practice,- the- slowest- bit- is- often- the- plumbing- around- it- —- bloated- prompts,- oversized- conversation- histories- sent- on- every- turn,- synchronous- retrieval- that- blocks- streaming,- or- validation- loops- that- retry- before- anything- reaches- the- user.- A- model- that- benchmarks- as- “fast”- can- still- feel- slow- if- the- prompt- is- bloated,- the- output- is- too- long,- or- the- system- waits- on- tools- before- streaming- anything- useful.
##- TL;DR
If- the- feature- feels- slow,- measure- three- things- separately:- time- to- first- token,- total- completion- time,- and- time- spent- outside- the- model- call.
If- first- token- is- slow,- look- at- queueing,- prompt- size,- routing- and- any- pre-processing- the- app- does- before- streaming- starts.- If- total- time- is- slow,- look- at- output- length,- tool- calls,- retries- and- slow- post-processing.- If- only- some- users- see- the- issue,- check- region,- account- tier,- concurrency- or- downstream- service- load.
Do- not- jump- straight- to- a- bigger- model- or- a- new- vendor- until- you- know- where- the- delay- actually- lives.
Time- to- first- token- (TTFT)- is- the- metric- that- most- closely- tracks- user- frustration- —- more- than- total- completion- time.- A- user- staring- at- a- blank- screen- for- 3- seconds- will- perceive- the- feature- as- "broken,"- even- if- the- full- answer- arrives- in- 4- seconds- total.- If- you- can- only- instrument- one- latency- metric,- start- with- TTFT.- Streaming- a- placeholder- or- a- typing- indicator- before- the- model- starts- generating- can- buy- you- an- extra- second- of- perceived- responsiveness.
##- What- latency- means- in- practice
“Latency”- in- an- LLM- app- is- a- bundle- of- timings,- not- one- number.
The- main- parts- are:
queueing- time:- the- request- waits- before- a- worker- starts; prompt- processing- time:- the- model- reads- the- input- tokens; time- to- first- token:- the- delay- before- the- first- streamed- output- appears; generation- time:- the- model- finishes- the- rest- of- the- answer; tool- time:- external- API- calls,- database- queries- or- function- calls; network- time:- round-trip- delay- between- user,- app- and- provider; ##- What- usually- causes- the- delay
Common- causes- are- boring,- which- is- why- they- recur:
1.- The- prompt- is- larger- than- it- needs- to- be. 2.- The- app- sends- too- much- history- back- into- the- model. 3.- The- model- waits- on- retrieval- or- tool- calls- before- it- can- stream- anything. 4.- The- response- is- longer- than- the- product- really- needs. 5.- The- app- retries- automatically- after- a- validation- failure. 6.- A- downstream- API- is- slow,- so- the- model- sits- idle. 7.- The- user- is- on- a- busy- region- or- account- tier- with- more- contention.
None- of- that- proves- the- provider- is- slow.- It- only- proves- the- request- path- is- doing- work.
##- A- worked- example
A- customer- support- bot- was- handling- small-business- IT- queries.- Users- reported- the- bot- felt- “laggy”- —- responses- appeared- 4–5- seconds- after- hitting- send.
Before:- The- app- sent- the- full- 40-message- chat- history- on- every- turn.- The- prompt- included- the- complete- conversation- plus- a- 600-word- system- instruction.- The- model- took- ~1.2- seconds- to- process- the- prompt,- the- streaming- connection- was- held- open- while- the- app- ran- a- synchronous- ticket-ID- lookup,- and- the- first- token- appeared- at- 3.8- seconds.- Total- response- time- averaged- 6.2- seconds.
After:- The- team- trimmed- conversation- history- to- the- last- 8- messages- (the- only- ones- relevant- for- context),- moved- the- ticket-ID- lookup- to- an- async- prefetch- that- runs- in- parallel- with- streaming,- and- reduced- the- system- instruction- to- 200- words- by- removing- redundant- examples.- First- token- dropped- to- 0.7- seconds.- Total- response- time- dropped- to- 1.9- seconds.
The- model- did- not- change.- The- provider- did- not- change.- The- fix- was- plumbing.
The- single- highest-ROI- latency- fix- we- see- across- production- LLM- apps- is- trimming- conversation- history.- Most- apps- send- the- full- chat- log- on- every- turn,- and- most- turns- only- need- the- last- 6–10- messages.- This- is- not- a- model- problem- —- it- is- a- prompt-design- problem.- Before- you- optimise- anything- else,- check- how- many- tokens- you- are- sending- that- the- model- does- not- actually- need.
##- What- to- measure
Before- changing- models,- record- a- simple- timing- breakdown:
request- received; request- queued; model- call- started; first- token- streamed; tool- calls- started- and- finished; final- token- received; response- rendered- to- the- user.
That- lets- you- answer- the- only- question- that- enough- of- your- plumbing- is- optimized.
##- What- to- change- first
If- the- first- token- is- slow,- try- these- in- order:
trim- the- prompt;
shorten- the- conversation- history- you- resend;
avoid- needless- retrieval- before- the- first- streamed- token;
stream- earlier- if- the- workflow- allows- it- (send- stream:- true- and- process- SSE- events- as- they- arrive,- as- recommended- by- OpenAI’s- streaming- docs);
reduce- any- synchronous- pre-checks;
cap- output- length- where- a- shorter- answer- is- enough.
If- total- time- is- slow,- look- at:
output- limits; tool-call- count; validation- loops; retries; slow- database- or- API- calls; post-processing- that- could- happen- after- the- user- already- has- the- answer.
A- smaller- model- can- help,- but- only- if- it- removes- the- actual- bottleneck.
##- Practical- decision- check
Use- this- check- before- you- rewrite- the- stack:
Does- the- user- need- the- full- answer- immediately,- or- just- a- visible- start? Is- the- app- waiting- on- retrieval,- tools- or- validation- before- it- can- stream? Are- you- sending- more- context- than- the- task- needs? Is- the- output- length- under- your- control? Are- retries- hiding- the- true- time- cost? Are- a- few- users- seeing- a- region- or- tenant-specific- slowdown?
If- you- cannot- answer- those- questions,- the- measurement- layer- is- too- thin.
##- What- this- page- cannot- tell- you
This- page- cannot- tell- you- which- provider- is- fastest- for- your- workload.
It- cannot- tell- you:
your- exact- queueing- delay; your- model’s- real- throughput- under- load; whether- a- tool- call- or- a- rerank- step- is- the- real- blocker; whether- the- delay- sits- in- your- app,- your- network- or- the- vendor; whether- a- user-perceived- slowdown- is- caused- by- one- long- request- or- many- small- ones.
It- can- only- show- you- how- to- stop- guessing.
##- What- would- change- the- advice
The- guidance- above- assumes- the- bottleneck- is- in- the- request- path- and- can- be- fixed- without- adding- systemic- risk.- That- assumption- breaks- down- in- three- cases:
When- latency- optimisation- adds- complexity- without- reducing- user-perceived- delay.- Adding- streaming- proxies,- cache- layers- or- multi-region- routing- introduces- new- failure- modes- —- dropped- connections,- stale- cache- entries,- routing- misconfiguration.- If- the- user- is- already- reading- the- response- faster- than- the- model- produces- it,- optimising- further- does- not- improve- the- experience. When- the- bottleneck- is- user- reading- speed,- not- model- speed.- A- 300-word- answer- produced- in- 0.8- seconds- feels- instant.- Trying- to- squeeze- that- to- 0.4- seconds- by- switching- providers- or- adding- infrastructure- is- engineering- theatre- —- no- user- notices- the- difference. When- a- provider- ships- a- hardware- generation- that- changes- the- latency- baseline.- If- a- provider- deploys- new- inference- hardware- (next-generation- GPUs,- custom- ASICs)- or- changes- its- routing- layer,- the- assumptions- that- guided- your- optimisation- may- no- longer- hold.- Re-check- latency- distributions- quarterly,- not- just- when- something- feels- slow.
If- any- of- these- apply,- stop- optimising- and- measure- again.
Quarterly- latency- re-benchmarking- is- not- paranoia- —- it- is- operational- hygiene.- Provider- routing- changes,- model- version- swaps,- and- silent- infrastructure- migrations- happen- without- announcement.- A- feature- that- was- fast- in- March- can- be- slow- in- June- with- no- code- changes- on- your- side.- Set- a- calendar- reminder.- Better- yet,- automate- it:- a- simple- cron- job- that- calls- your- production- endpoint- and- logs- TTFT- and- total- time- gives- you- a- trail- when- users- start- complaining.
If- any- of- these- apply,- stop- optimising- and- measure- again.
##- Regional- caveats
Latency- optimisation- advice- is- universal,- but- the- specific- levers- vary- by- region:
UK/Europe:- Provider- endpoint- location- matters.- A- request- routed- to- a- US-west- data- centre- adds- 80–150- ms- of- round-trip- time- compared- to- a- London- or- Frankfurt- endpoint.- Several- providers- offer- European- inference- endpoints- with- data-sovereignty- guarantees;- check- whether- your- account- is- configured- to- prefer- them. Asia-Pacific- /- South- America:- Provider- coverage- is- thinner.- Fewer- regional- inference- endpoints- mean- higher- baseline- network- latency.- This- makes- prompt-size- optimisation- proportionally- more- valuable- —- every- unnecessary- token- costs- extra- round-trip- time. Global- /- multi-region- deployments:- If- your- users- are- distributed- across- regions,- a- single- provider- endpoint- will- serve- some- well- and- others- poorly.- Consider- multi-region- routing- or- at- minimum- measure- your- actual- latency- distribution- by- user- region- before- declaring- a- provider- “slow.”
The- useful- caution- is- universal:- latency- should- be- measured- on- the- live- request- path,- not- inferred- from- a- model- page- or- a- marketing- claim.- But- the- baseline- you- are- optimising- against- depends- on- where- you- and- your- users- are.ion- before- declaring- a- provider- “slow.”
The- useful- caution- is- universal:- latency- should- be- measured- on- the- live- request- path,- not- inferred- from- a- model- page- or- a- marketing- claim.- But- the- baseline- you- are- optimising- against- depends- on- where- you- and- your- users- are.
##- Related- guides- guides
What- is- a- token,- and- why- does- it- affect- AI- cost? Context- windows- explained:- why- bigger- is- not- always- better API- model- pricing:- input,- output,- cache- and- batch- costs Rate- limits- explained:- requests,- tokens,- tiers- and- hidden- launch- risks
##- Methodology
Data- checked:- 2026-05-28 Sources- consulted:- OpenAI- streaming- documentation,- Anthropic- streaming- documentation,- Google- Gemini- streaming- documentation,- OpenTelemetry- tracing- specification Assumptions:- No- hands-on- latency- benchmark- is- claimed.- Provider- load- changes- over- time.- Timing- numbers- vary- by- region,- tenant- and- workload.- User-perceived- speed- is- not- the- same- as- raw- API- completion- time. Limitations:- This- article- does- not- benchmark- specific- providers,- does- not- provide- legal- or- SLA- advice,- and- does- not- cover- hardware-level- latency- optimisation.- Worked- example- numbers- are- illustrative,- drawn- from- documented- team- experiences,- not- from- a- controlled- experiment. Jurisdiction:- Global.- Regional- caveats- cover- UK/Europe,- Asia-Pacific/South- America,- and- multi-region- deployments.
##- Source- list
OpenAI- streaming- documentation- —- https://platform.openai.com/docs/guides/streaming- (accessed- 2026-05-28) Anthropic- streaming- documentation- —- https://docs.anthropic.com/en/docs/build-with-claude/streaming- (accessed- 2026-05-28) Google- Gemini- streaming- documentation- —- https://ai.google.dev/gemini-api/docs/streaming- (accessed- 2026-05-28) OpenTelemetry- documentation- —- https://opentelemetry.io/docs/- (accessed- 2026-05-28)
##- Trust- Stack
Last- checked:- 2026-05-28 Corrections:- Contact- us- to- report- errors
##- Change- log
2026-05-28:- Full- editorial- review- against- 16-gate- checklist.- Added- 3- Editor’s- Note- cards,- slugified- heading- IDs,- standardised- Methodology- and- Source- List- formats,- added- Trust- Stack- section- with- corrections- policy- and- affiliation- declaration,- removed- inline- citation- numbers,- removed- workflow- leak- reference,- corrected- frontmatter- writtenBy- label. 2026-05-25:- Integrated- Editor’s- Notes,- added- inline- citations,- fixed- related-guide- links- to- production- routes,- added- worked- example- with- concrete- numbers,- replaced- Global- applicability- with- regional- caveats,- added- What- would- change- the- advice- section. 2026-05-22:- First- published.