#— LLM— monitoring— and— alerting:— tracking— performance,— cost,— and— reliability— in— production
##— TL;DR
Build— your— LLM— alerting— pipeline— around— four— measurable— failure— modes— —— cost— spikes,— latency— regressions,— error— rate— changes,— and— quality— score— drops— —— with— separate— thresholds— for— each— rather— than— a— single— “something— is— wrong”— alarm.— Start— with— metadata-level— alerts— (no— prompt— content— needed),— add— sampled— quality— monitoring— at— weekly— cadence,— and— escalate— only— when— multiple— signals— confirm— the— same— incident.
Most— teams— either— fire— too— many— alerts— (every— latency— blip,— every— small— cost— increase)— or— too— few— (no— quality— regression— detection,— no— drift— awareness).— The— right— target— is— one— actionable— alert— per— 4— 72— hours— of— normal— operation,— with— tiered— escalation— when— two— or— more— signals— trigger— simultaneously.
There-- is-- no-- off-the-shelf-- "LLM-- alerting"-- product-- that-- covers-- all-- four-- failure-- modes-- today.-- Every-- observability-- platform-- covers-- different-- subsets.-- Teams-- that-- skip-- the-- design-- phase-- and-- just-- turn-- on-- every-- default-- alert-- in-- their-- chosen-- tool-- end-- up-- with-- alert-- fatigue-- within-- two-- weeks.-- The-- framework-- below-- is-- designed-- to-- be-- implemented-- gradually-- —-- start-- with-- cost-- and-- error-- rate-- alerts,-- add-- latency-- only-- when-- users-- complain-- about-- performance,-- and-- add-- quality-- monitoring-- last.
##— What— alerting— means— for— LLM— applications
Alerting— for— LLM— apps— is— not— fundamentally— different— from— alerting— for— any— enough— other— production— system— —— you— measure— signals,— set— thresholds,— notify— people,— and— review— whether— the— alarm— was— useful.— Four— things— make— it— harder:
1.— Normal— behaviour— shifts— constantly.— Model— providers— update— their— APIs— without— notice.— A— latency— jump— may— mean— the— provider— deployed— a— new— inference— kernel,— not— that— your— application— broke.— Distinguishing— drift— from— failure— requires— baseline— tracking. 2.— Cost— is— a— real-time— signal— that— most— teams— ignore.— An— LLM— cost— spike— often— precedes— a— reliability— problem— —— runaway— retries,— prompt— injection,— or— an— infinite— loop— in— an— agent— workflow— —— before— the— application— visibly— fails. 3.— Quality— cannot— be— measured— from— dashboards— alone.— Latency,— error— rate,— and— token— counts— tell— you— whether— the— system— ran.— They— do— not— tell— you— whether— the— output— was— correct.— Quality— regression— detection— requires— sampled— content— review— or— automated— evaluation. 4.— Most— alerting— platforms— lack— LLM-specific— primitives.— Standard— APM— tools— understand— status— codes— and— response— times— but— not— token— counts,— cost-per-request,— prompt— lengths,— or— eval— scores.— You— need— to— define— these— custom— metrics— yourself.
###— The— four— alerting— categories
|— Category— |— What— it— catches— |— Typical— threshold— |— Escalation— trigger— | | | -| | | |— Cost— spikes— |— Runaway— retries,— long— prompts,— model— routing— errors— |— >50%— increase— in— cost/hour— vs— 7-day— rolling— average— |— Combined— with— error— rate— or— latency— increase— | |— Latency— regressions— |— Degraded— provider— performance,— prompt— bloat,— saturation— |— P95— latency— >3x— 7-day— baseline— for— >5— minutes— |— Sustained— >5x— baseline— | |— Error— rate— changes— |— Provider— outages,— auth— failures,— rate— limits,— timeouts— |— >5%— error— rate— sustained— >2— minutes— |— >10%— at— any— duration— | |— Quality— score— drops— |— Model— drift,— prompt— degradation,— bad— eval— configurations— |— LLM-as-judge— score— drop— >3x— baseline— variance— |— Sustained— over— >2— hours— (sampling-dependent)— |
##— Where— teams— misuse— alerting
|--- Alerting— on— every— latency— measurement— separately.— A— team— sets— P50,— P95,— and— P99— latency— alerts— at— 500ms,— 2s,— and— 5s— respectively.— Every— late-afternoon— traffic— spike— triggers— the— P95— alert.— The— team— ignores— it— after— three— days,— missing— the— real— P99— regression— when— a— model— provider— changes— its— inference— engine— and— pushes— median— latency— from— 400ms— to— 1.2s.— Use— P95— as— your— single— latency— alert— metric.— Add— P99— only— as— an— escalation— condition.
|--- Measuring— only— cost— or— only— latency,— never— both.— A— team— spots— a— cost— spike— and— investigates.— It— turns— out— the— agent— workflow— entered— a— retry— loop,— consuming— 15x— normal— tokens— per— request.— The— latency— dashboard— was— normal— because— each— individual— request— was— fast— —— just— repeated— uselessly— many— enough— times.— Cost— and— latency— alerts— need— to— be— read— together:— a— cost— spike— without— a— latency— change— suggests— a— loop— or— prompt-length— problem.— A— cost— spike— with— a— latency— spike— suggests— a— model— routing— error— or— provider— degradation.
|--- Setting— static— thresholds— for— a— non-stationary— signal.— A— team— configures— Datadog— to— alert— when— P95— latency— exceeds— 3— seconds.— For— the— first— month,— baseline— P95— is— 800ms,— and— 3s— catches— genuine— problems.— Three— months— later,— the— team— switched— to— a— different— model— that— naturally— runs— at— P95— 2.5s.— The— 3s— threshold— now— catches— nothing— useful,— but— no— one— remembers— to— update— it.— Use— rolling-window— thresholds— (e.g.,— 3x— the— 7-day— average)— rather— than— absolute— numbers.
|--- Skipping— the— alert— post-mortem.— A— team— builds— a— comprehensive— alerting— setup,— but— never— reviews— whether— the— alerts— were— useful.— After— six— months,— 80%— of— alerts— are— ignored— because— they— do— not— actually— indicate— actionable— problems.— The— team— has— alert— fatigue— and— a— false— sense— of— security.— A— monthly— alert— review— is— as— important— as— the— initial— setup.
Cost-aware— alerting— is— the— single— most— underused— practice— in— production— LLM— operations.— Many— teams— do— not— expose— per-request— cost— as— a— real-time— metric— at— all.— They— discover— runaway— costs— at— the— end— of— the— month— when— the— bill— arrives.— Deploy— a— per-minute— token-count— dashboard— before— any— other— monitoring— investment— —— it— pays— for— itself— in— the— first— incident— it— catches.
--- Setting— static— thresholds— for— a— non-stationary— signal.— A— team— configures— Datadog— to— alert— when— P95— latency— exceeds— 3— seconds.— For— the— first— month,— baseline— P95— is— 800ms,— and— 3s— catches— genuine— problems.— Three— months— later,— the— team— switched— to— a— different— model— that— naturally— runs— at— P95— 2.5s.— The— 3s— threshold— now— catches— nothing— useful,— but— no— one— remembers— to— update— it.— Use— rolling-window— thresholds— (e.g.,— 3x— the— 7-day— average)— rather— than— absolute— numbers.
--- Skipping— the— alert— post-mortem.— A— team— builds— a— comprehensive— alerting— setup,— but— never— reviews— whether— the— alerts— were— useful.— After— six— months,— 80%— of— alerts— are— ignored— because— they— do— not— actually— indicate— actionable— problems.— The— team— has— alert— fatigue— and— a— false— sense— of— security.— A— monthly— alert— review— is— as— important— as— the— initial— setup.
The-- alert-- that-- fires-- and-- is-- ignored-- is-- worse-- than-- no-- alert-- —-- it-- trains-- your-- team-- to-- treat-- every-- alarm-- as-- noise.-- During-- your-- monthly-- review,-- disable-- any-- alert-- that-- fired-- more-- than-- 3-- times-- without-- producing-- a-- documented-- action.-- If-- you-- cannot-- explain-- what-- someone-- should-- do-- when-- an-- alert-- fires,-- you-- should-- not-- have-- that-- alert.
##— Practical— decision— check:— questions— to— ask— before— building
Before— configuring— any— alert,— answer— these— questions:
--- What— would— a— user— notice— before— my— alerts— fire?— If— users— can— tell— something— is— wrong— before— your— monitoring— system— fires,— your— alert— thresholds— are— too— loose.— For— LLM— apps,— users— notice:— slow— responses— (latency— >5s),— obviously— wrong— answers— (quality— regression),— or— error— messages— (timeouts,— rate— limits).— Tune— thresholds— so— the— alert— fires— at— least— one— response— cycle— before— users— reliably— notice. --- What— signal— combination— means— “this— is— real”?— Single-signal— alerts— are— noisy.— Latency— increases— every— time— you— ship— a— new— feature— with— more— context.— A— simultaneous— cost— spike— +— latency— increase— +— error— rate— increase— means— the— provider— is— degraded.— Design— escalation— so— multi-signal— events— page— the— on-call— engineer— and— single-signal— events— go— to— a— daily— digest. --- Do— I— have— a— baseline— to— compare— against?— You— cannot— meaningfully— alert— on— a— metric— you— have— not— measured— for— at— least— a— week.— Deploy— telemetry— first,— collect— seven— days— of— data,— then— set— thresholds.— Without— a— baseline,— every— threshold— is— a— guess. --- Who— gets— paged— at— 3— AM— and— what— can— they— actually— fix?— If— the— on-call— engineer— can— only— escalate— to— the— provider— (which— will— not— answer— for— hours),— do— not— page— them.— Send— an— email— instead.— Page— only— when— there— is— a— service-side— intervention:— restart— a— gateway,— roll— back— a— prompt— change,— flip— a— model— routing— rule,— or— rotate— API— keys. --- Which— alerts— have— I— disabled— or— ignored— this— month?— If— the— answer— is— “most— of— them,”— you— have— too— many— alerts.— Remove— or— re-threshold— anything— that— fires— but— never— leads— to— an— action.— An— active— but— ignored— alert— is— worse— than— no— alert— —— it— creates— a— trained— habit— of— ignoring— alarms.
##— Evidence— and— caveats
|--- Provider— monitoring.— Major— LLM— providers— (OpenAI— status.openai.com,— Anthropic— status.highly-reliable-service.com,— Google— health.status.cloud.google.com)— publish— status— pages— but— do— not— give— per-customer— latency— SLAs— above— availability— commitments— in— standard— tiers.— External— monitoring— (e.g.,— a— synthetics— check— hitting— your— own— endpoint)— is— the— only— reliable— way— to— detect— provider-side— degradation— affecting— your— specific— use— case. |--- Cost— baseline— drift.— Model— prices— change.— OpenAI— lowered— GPT-4o— prices— by— roughly— 50%— in— mid-2025;— Anthropic— and— Google— matched— similar— reductions— within— months.— Cost-per-request— baselines— shift— with— pricing— changes,— model— upgrades,— and— routing— optimisation.— Re-baseline— after— every— provider— pricing— announcement— or— model-switch— deployment. |--- Quality— score— limitations.— LLM-as-judge— quality— scoring— adds— latency— (one— extra— model— call— per— evaluated— output),— costs— money,— and— introduces— its— own— biases.— Google’s— Gemini— 2.5— Pro— research— (2025)— and— the— HELM— benchmark— from— Stanford— CRFM— (2023–2026)— both— document— metric— alignment— and— skew— risks.— Quality— alerts— based— on— automated— evaluation— are— indicators,— not— ground— truth.— Pair— them— with— manual— spot-checks— on— a— 2%— random— sample— at— weekly— cadence. |--- Tooling— maturity.— The— alerting— features— in— LLM— observability— tools— are— less— mature— than— the— tracing— and— logging— features.— LangSmith— alerting— is— webhook-only— (no— built-in— SMS/pager/email).— Datadog— has— the— most— mature— alerting— engine— because— you— can— define— arbitrary— metric— thresholds.— Helicone— has— basic— cost-threshold— alerts.— Plan— for— a— custom— alerting— pipeline— (Prometheus— +— Alertmanager— +— your— preferred— notification— channel)— if— you— need— production-grade— escalation. |--- This— is— not— a— scalability— guide.— Coverage— of— GPU— utilisation— alerts,— container— health— monitoring,— and— infrastructure-level— reliability— is— intentionally— excluded.— Those— are— separate— operational— disciplines— with— well-documented— patterns— —— Prometheus— +— Grafana— +— PagerDuty— for— infrastructure;— Nvidia— DCGM— for— GPU— health;— Datadog— or— Grafana— Cloud— for— APM.— This— guide— covers— the— LLM— application— layer— only.
##— Methodology
|--- Data— checked:— 2026-06-09 |--- Sources— consulted:— OpenAI— and— Anthropic— status— pages— and— documentation,— LangSmith— observability— documentation,— Datadog— APM— documentation,— Helicone— documentation,— Prometheus— and— Alertmanager— documentation,— Google— Gemini— 2.5— Pro— research— on— LLM-as-judge— bias— (2025),— Stanford— HELM— benchmark— documentation |--- Assumptions:— Alerting— thresholds— are— starting— recommendations— and— assume— a— baseline— collection— period— of— at— least— 7— days.— The— four-category— alerting— model— assumes— a— production— LLM— application— with— measurable— traffic;— pre-launch— teams— should— focus— on— cost— and— error— rate— only. |--- Limitations:— This— guide— does— not— cover— infrastructure-level— monitoring— (GPU— utilisation,— container— health,— network— telemetry),— security-specific— alerting— (prompt— injection— detection,— PII— leakage),— or— regulatory— compliance— monitoring— (EU— AI— Act— logging— requirements).— It— focuses— on— application-layer— LLM— alerting— for— performance,— cost,— and— quality. |--- Jurisdiction:— Global.— No— jurisdiction-specific— regulatory— guidance— is— included.
##— Source— list
|--- OpenAI— Platform— Documentation— —— https://platform.openai.com/docs— (accessed— 2026-06-09) |--- Anthropic— Documentation— —— https://docs.anthropic.com— (accessed— 2026-06-09) |--- LangSmith— Documentation— —— https://docs.smith.langchain.com— (accessed— 2026-06-09) |--- Datadog— APM— Documentation— —— https://docs.datadoghq.com/tracing/— (accessed— 2026-06-09) |--- Helicone— Documentation— —— https://docs.helicone.ai— (accessed— 2026-06-09) |--- Prometheus— Alertmanager— —— https://prometheus.io/docs/alerting/latest/alertmanager/— (accessed— 2026-06-09) |--- Stanford— HELM— Benchmark— —— https://crfm.stanford.edu/helm/— (accessed— 2026-06-09)
##— Trust— Stack
--- Last— checked:— 2026-06-09 --- Corrections:— Contact— us— to— report— errors
##— Change— log
--- 2026-06-09:— Editorial— review— against— 16-gate— checklist— —— added— Source— list,— Trust— Stack,— Change— Log,— Methodology— section;— added— 1— Editor’s— Note— (alert— review— discipline);— added— slugified— H2/H3— IDs;— reduced— and— renamed— related— guides;— trimmed— description— to— ≤155— chars;— updated— reviewedBy— and— lastChecked.— Core— content— preserved. --- 2026-06-09:— Initial— draft— published.
##— Related— guides— guides
--- LLM— observability— stack:— logging,— tracing,— monitoring,— and— cost— tracking --- AI— output— monitoring:— what— to— log,— sample— and— review --- AI— incident— response:— what— to— do— when— a— model— gives— harmful— or— wrong— advice --- Model— drift— without— training:— why— API— behavior— changes— over— time