LLM monitoring and alerting: tracking performance, cost, and reliability in production

TL;DR

Build your LLM alerting pipeline around four measurable failure modes — cost spikes, latency regressions, error rate changes, and quality score drops — with separate thresholds for each rather than a single “something is wrong” alarm. Start with metadata-level alerts (no prompt content needed), add sampled quality monitoring at weekly cadence, and escalate only when multiple signals confirm the same incident.

Most teams either fire too many alerts (every latency blip, every small cost increase) or too few (no quality regression detection, no drift awareness). The right target is one actionable alert per 4 72 hours of normal operation, with tiered escalation when two or more signals trigger simultaneously.

What alerting means for LLM applications

Alerting for LLM apps is not fundamentally different from alerting for any enough other production system — you measure signals, set thresholds, notify people, and review whether the alarm was useful. Four things make it harder:

1. Normal behaviour shifts constantly. Model providers update their APIs without notice. A latency jump may mean the provider deployed a new inference kernel, not that your application broke. Distinguishing drift from failure requires baseline tracking.
2. Cost is a real-time signal that most teams ignore. An LLM cost spike often precedes a reliability problem — runaway retries, prompt injection, or an infinite loop in an agent workflow — before the application visibly fails.
3. Quality cannot be measured from dashboards alone. Latency, error rate, and token counts tell you whether the system ran. They do not tell you whether the output was correct. Quality regression detection requires sampled content review or automated evaluation.
4. Most alerting platforms lack LLM-specific primitives. Standard APM tools understand status codes and response times but not token counts, cost-per-request, prompt lengths, or eval scores. You need to define these custom metrics yourself.

The four alerting categories

Where teams misuse alerting

|- Alerting on every latency measurement separately. A team sets P50, P95, and P99 latency alerts at 500ms, 2s, and 5s respectively. Every late-afternoon traffic spike triggers the P95 alert. The team ignores it after three days, missing the real P99 regression when a model provider changes its inference engine and pushes median latency from 400ms to 1.2s. Use P95 as your single latency alert metric. Add P99 only as an escalation condition. |- Measuring only cost or only latency, never both. A team spots a cost spike and investigates. It turns out the agent workflow entered a retry loop, consuming 15x normal tokens per request. The latency dashboard was normal because each individual request was fast — just repeated uselessly many enough times. Cost and latency alerts need to be read together: a cost spike without a latency change suggests a loop or prompt-length problem. A cost spike with a latency spike suggests a model routing error or provider degradation. |- Setting static thresholds for a non-stationary signal. A team configures Datadog to alert when P95 latency exceeds 3 seconds. For the first month, baseline P95 is 800ms, and 3s catches genuine problems. Three months later, the team switched to a different model that naturally runs at P95 2.5s. The 3s threshold now catches nothing useful, but no one remembers to update it. Use rolling-window thresholds (e.g., 3x the 7-day average) rather than absolute numbers. |- Skipping the alert post-mortem. A team builds a comprehensive alerting setup, but never reviews whether the alerts were useful. After six months, 80% of alerts are ignored because they do not actually indicate actionable problems. The team has alert fatigue and a false sense of security. A monthly alert review is as important as the initial setup.

• Setting static thresholds for a non-stationary signal. A team configures Datadog to alert when P95 latency exceeds 3 seconds. For the first month, baseline P95 is 800ms, and 3s catches genuine problems. Three months later, the team switched to a different model that naturally runs at P95 2.5s. The 3s threshold now catches nothing useful, but no one remembers to update it. Use rolling-window thresholds (e.g., 3x the 7-day average) rather than absolute numbers.

• Skipping the alert post-mortem. A team builds a comprehensive alerting setup, but never reviews whether the alerts were useful. After six months, 80% of alerts are ignored because they do not actually indicate actionable problems. The team has alert fatigue and a false sense of security. A monthly alert review is as important as the initial setup.

Practical decision check: questions to ask before building

Before configuring any alert, answer these questions:

• What would a user notice before my alerts fire? If users can tell something is wrong before your monitoring system fires, your alert thresholds are too loose. For LLM apps, users notice: slow responses (latency >5s), obviously wrong answers (quality regression), or error messages (timeouts, rate limits). Tune thresholds so the alert fires at least one response cycle before users reliably notice.
• What signal combination means “this is real”? Single-signal alerts are noisy. Latency increases every time you ship a new feature with more context. A simultaneous cost spike + latency increase + error rate increase means the provider is degraded. Design escalation so multi-signal events page the on-call engineer and single-signal events go to a daily digest.
• Do I have a baseline to compare against? You cannot meaningfully alert on a metric you have not measured for at least a week. Deploy telemetry first, collect seven days of data, then set thresholds. Without a baseline, every threshold is a guess.
• Who gets paged at 3 AM and what can they actually fix? If the on-call engineer can only escalate to the provider (which will not answer for hours), do not page them. Send an email instead. Page only when there is a service-side intervention: restart a gateway, roll back a prompt change, flip a model routing rule, or rotate API keys.
• Which alerts have I disabled or ignored this month? If the answer is “most of them,” you have too many alerts. Remove or re-threshold anything that fires but never leads to an action. An active but ignored alert is worse than no alert — it creates a trained habit of ignoring alarms.

Evidence and caveats

|- Provider monitoring. Major LLM providers (OpenAI status.openai.com, Anthropic status.highly-reliable-service.com, Google health.status.cloud.google.com) publish status pages but do not give per-customer latency SLAs above availability commitments in standard tiers. External monitoring (e.g., a synthetics check hitting your own endpoint) is the only reliable way to detect provider-side degradation affecting your specific use case. |- Cost baseline drift. Model prices change. OpenAI lowered GPT-4o prices by roughly 50% in mid-2025; Anthropic and Google matched similar reductions within months. Cost-per-request baselines shift with pricing changes, model upgrades, and routing optimisation. Re-baseline after every provider pricing announcement or model-switch deployment. |- Quality score limitations. LLM-as-judge quality scoring adds latency (one extra model call per evaluated output), costs money, and introduces its own biases. Google’s Gemini 2.5 Pro research (2025) and the HELM benchmark from Stanford CRFM (2023–2026) both document metric alignment and skew risks. Quality alerts based on automated evaluation are indicators, not ground truth. Pair them with manual spot-checks on a 2% random sample at weekly cadence. |- Tooling maturity. The alerting features in LLM observability tools are less mature than the tracing and logging features. LangSmith alerting is webhook-only (no built-in SMS/pager/email). Datadog has the most mature alerting engine because you can define arbitrary metric thresholds. Helicone has basic cost-threshold alerts. Plan for a custom alerting pipeline (Prometheus + Alertmanager + your preferred notification channel) if you need production-grade escalation. |- This is not a scalability guide. Coverage of GPU utilisation alerts, container health monitoring, and infrastructure-level reliability is intentionally excluded. Those are separate operational disciplines with well-documented patterns — Prometheus + Grafana + PagerDuty for infrastructure; Nvidia DCGM for GPU health; Datadog or Grafana Cloud for APM. This guide covers the LLM application layer only.

Methodology

|- Data checked: 2026-06-09 |- Sources consulted: OpenAI and Anthropic status pages and documentation, LangSmith observability documentation, Datadog APM documentation, Helicone documentation, Prometheus and Alertmanager documentation, Google Gemini 2.5 Pro research on LLM-as-judge bias (2025), Stanford HELM benchmark documentation |- Assumptions: Alerting thresholds are starting recommendations and assume a baseline collection period of at least 7 days. The four-category alerting model assumes a production LLM application with measurable traffic; pre-launch teams should focus on cost and error rate only. |- Limitations: This guide does not cover infrastructure-level monitoring (GPU utilisation, container health, network telemetry), security-specific alerting (prompt injection detection, PII leakage), or regulatory compliance monitoring (EU AI Act logging requirements). It focuses on application-layer LLM alerting for performance, cost, and quality. |- Jurisdiction: Global. No jurisdiction-specific regulatory guidance is included.

Source list

|- OpenAI Platform Documentation — https://platform.openai.com/docs (accessed 2026-06-09) |- Anthropic Documentation — https://docs.anthropic.com (accessed 2026-06-09) |- LangSmith Documentation — https://docs.smith.langchain.com (accessed 2026-06-09) |- Datadog APM Documentation — https://docs.datadoghq.com/tracing/ (accessed 2026-06-09) |- Helicone Documentation — https://docs.helicone.ai (accessed 2026-06-09) |- Prometheus Alertmanager — https://prometheus.io/docs/alerting/latest/alertmanager/ (accessed 2026-06-09) |- Stanford HELM Benchmark — https://crfm.stanford.edu/helm/ (accessed 2026-06-09)

Trust Stack

• Last checked: 2026-06-09
• Corrections: Contact us to report errors

Change log

• 2026-06-09: Editorial review against 16-gate checklist — added Source list, Trust Stack, Change Log, Methodology section; added 1 Editor’s Note (alert review discipline); added slugified H2/H3 IDs; reduced and renamed related guides; trimmed description to ≤155 chars; updated reviewedBy and lastChecked. Core content preserved.
• 2026-06-09: Initial draft published.

Related guides

• LLM observability stack: logging, tracing, monitoring, and cost tracking
• AI output monitoring: what to log, sample and review
• AI incident response: what to do when a model gives harmful or wrong advice
• Model drift without training: why API behavior changes over time