LLM observability stack: logging, tracing, monitoring, and cost tracking

TL;DR: If you are choosing an LLM observability tool today, LangSmith fits best for LangChain-heavy stacks, Arize for teams that already monitor ML models, Helicone for lightweight token-and-latency dashboards, Weights & Biases for research-track evaluation, and Datadog for enterprises that want to consolidate infrastructure and LLM monitoring in one platform. You do not need more than one, and you should pick before your first production deployment, not after.

The problem with LLM observability tools

LLM observability is in its awkward teenage phase. Every vendor says they do the same thing — traces, prompts, evals, cost tracking — and they all look similar in a demo. The differences show up in how they handle scale, how they integrate with your existing stack, and what they do with the data after they collect it.

The five tools covered here occupy different niches:

• LangSmith — tightly coupled to the LangChain ecosystem, best if you already use LangChain/LangGraph
• Arize AI — built for teams that already monitor ML models and want unified observability
• Helicone — simple, focused, cheap; good for small teams that just want to know what their LLM calls cost
• Weights & Biases — strong for research and evaluation; weaker for real-time production monitoring
• Datadog LLM Observability — full enterprise observability; expensive but comprehensive if you already use Datadog

None of them is universally better. The right choice depends on your stack, team size, and what you already pay for.

What a real observability stack covers

Before comparing tools, decide what you actually need to observe. Every observability setup should capture four data layers:

The tools below all cover logging and tracing. The differences are in monitoring depth, evaluation integration, and cost-accounting precision.

Five tools compared

LangSmith

Best for: Teams already using LangChain, LangGraph, or the LangChain ecosystem. If you are not using LangChain, LangSmith is probably overkill.

LangSmith started as a debugging UI for LangChain traces and grew into a full observability platform. It offers tracing for LangChain chains and agents, prompt versioning and management, evaluation runs against test datasets, a feedback API for user ratings, and cost tracking by model and run.

Typical setup:

# Using the LangChain Python SDK
from langsmith import Client
import os

os.environ["LANGSMITH_API_KEY"] = "lsv2_..."
os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_PROJECT"] = "my-production-app"

# Traces are captured automatically for all LangChain calls
# To log custom traces:
client = Client()
with client.trace(
  "my-custom-run",
  inputs={"query": user_input},
  project_name="my-production-app",
) as run:
  response = my_model.invoke(user_input)
  run.end(outputs={"response": response})

When to choose LangSmith: You use LangChain or LangGraph. You need evaluation-runs with dataset management. You want prompt versioning alongside tracing.

When not to: You are not using LangChain. You need infrastructure-level monitoring. You want a single-vendor everything platform.

Arize AI

Best for: Teams that already monitor ML model performance and want to extend that to LLM observability. Arize is particularly strong for data-centric observability — understanding how input distributions affect output quality.

Arize (now Arize AX) started as a traditional ML observability platform and added LLM monitoring. Its differentiator is the data-science focus: embedding drift detection, input/output quality dashboards, and correlation analysis between input changes and quality changes.

Typical setup:

from arize.experimental import ArizeClient
from openinference.instrumentation.openai import OpenAIInstrumentor

# Auto-instrument OpenAI calls
OpenAIInstrumentor().instrument()

# Custom trace with schema
client = ArizeClient(api_key="...", space_key="...")
spans = [
  {
  "name": "completion",
  "attributes": {
  "openinference.span.kind": "LLM",
  "openinference.llm.model_name": "gpt-4o",
  "openinference.llm.token_count.prompt": 150,
  "openinference.llm.token_count.completion": 42,
  }
  }
]
client.log_spans(project_id="my-app", spans=spans)

When to choose Arize: You already monitor traditional ML models with Arize. You need embedding drift detection and input-distribution correlation. You want unified ML + LLM observability.

When not to: You only need API cost tracking. You want zero-setup instrumentation. You do not have ML monitoring needs beyond LLMs.

Helicone

Best for: Small teams and solo operators who want a straightforward cost-and-latency dashboard with minimal setup. Helicone is the simplest tool in this comparison — it acts as a proxy between your app and the LLM provider.

Helicone works differently from the others: instead of an SDK you install, you point your API calls through the Helicone proxy URL. This makes it trivially easy to set up for OpenAI, Anthropic, and other standard API providers.

Typical setup:

import openai

# Just prefix the base URL — no SDK needed
client = openai.OpenAI(
  api_key="sk-...",
  base_url="https://oai.hconeai.com/v1",  # Helicone proxy
)
# Add Helicone headers for user/feature tracking
response = client.chat.completions.create(
  model="gpt-4o",
  messages=[{"role": "user", "content": user_input}],
  extra_headers={
  "Helicone-User-Id": user_id,
  "Helicone-Property-App": "chat-feature",
  }
)

When to choose Helicone: You want the simplest possible setup. You need cost tracking and latency dashboards. You are a small team or solo builder. You want an open-source option you can self-host.

When not to: You need quality evaluation. You need span-level tracing for complex agent chains. You need infrastructure or GPU monitoring.

Weights & Biases (W&B Prompts)

Best for: Research and development teams that already use W&B for experiment tracking. W&B Prompts extends the experiment-tracking paradigm to LLM observability — it is stronger for eval runs and dataset iteration than for real-time production monitoring.

W&B Prompts captures traces, manages evaluation datasets, and lets you compare prompt versions side-by-side. It integrates naturally with W&B’s artifact system and model registry.

Typical setup:

import wandb

# Initialize a W&B run
run = wandb.init(project="my-llm-app", job_type="prompts")

# Trace an LLM call
from wandb.integration.openai import autolog
autolog({"project": "my-llm-app"})

# Or trace manually
response = openai.chat.completions.create(...)
wandb.log({
  "tokens": {"prompt": ..., "completion": ...},
  "response": response.choices[0].message.content,
})

When to choose W&B: You already use W&B for experiment tracking. You need evaluation-dataset management with side-by-side comparison. You iterate on prompts in a research or dev setting.

When not to: You need real-time production alerting. You want automatic cost tracking. You need infrastructure monitoring alongside LLM data.

Datadog LLM Observability

Best for: Enterprise teams already invested in the Datadog ecosystem. Datadog LLM Observability is not a standalone LLM tool — it is an extension of Datadog APM that adds LLM-specific spans to your existing traces.

This is the most powerful option for teams that already monitor infrastructure, application performance, and logs in Datadog, because it correlates LLM traces with the underlying infrastructure traces in a single dashboard.

Typical setup:

from ddtrace import patch, config

# Auto-instrument supported LLM providers
patch(openai=True)

# Or use the LLM observability SDK directly
from ddtrace.llmobs import LLMObs

LLMObs.enable(
  ml_app="my-llm-app",
  api_key="...",
  site="datadoghq.eu",
)

with LLMObs.llm(
  model_name="gpt-4o",
  name="completion",
  model_provider="openai",
  input_messages=[{"content": user_input}],
  output_message={"content": response},
):
  response = openai.chat.completions.create(...)

When to choose Datadog: You already use Datadog for APM, infrastructure monitoring, and logging. You need to correlate LLM traces with service performance. You have enterprise compliance requirements for monitoring and audit trails.

When not to: You are not on Datadog already. You are a small team or solo builder. You only need simple cost tracking.

Decision framework

|| Your situation | Recommended tool | Rationale | |---------------|-----------------|-----------| | Using LangChain/LangGraph in production | LangSmith | Tightest integration; trace depth unmatched for LangChain chains; includes evaluation | | Already monitor ML models | Arize AI | Unified ML + LLM observability; embedding drift; correlation analysis | | Small team, just want cost tracking | Helicone | Trivial setup; per-request cost; open-source self-host option | | Research/dev, iterate on prompts | Weights & Biases | Evaluation dataset management; side-by-side comparison; natural for research workflows | | Full Datadog shop, enterprise | Datadog LLM Obs | Correlate LLM traces with infra; existing alerting and dashboards | | Switching approaches frequently | Arize or W&B | Most flexible for combining evaluation + traces without framework lock-in |

How to set up observability before production

Regardless of which tool you choose, the setup pattern is the same:

1. Instrument before the first user hits your feature. Adding observability after a problem surfaces means you have a blind spot on the data that caused the problem. Instrument at the prototype stage.

2. Set sampling rates from day one. Start with 1–5% of requests sampled for full prompt-and-response logging. Log 100% of errors, safety refusals, and high-latency requests. Adjust based on traffic volume.

3. Define retention tiers. Full traces for 7 days. Aggregated metrics forever. Raw prompts for a maximum of 30 days unless compliance requires longer. Review retention at least quarterly.

4. Add evaluation alongside tracing, not after. Run LLM-as-judge on a sampled subset (1–5%) from the start. Retroactively building an evaluation pipeline means you lose the comparison baseline.

5. Pick your cost tracking level. At minimum, track cost per model per day. At a deeper level, track cost per user-per-feature per model. Helicone is the easiest way to get the first level; LangSmith and Arize support the second with more setup.

Where teams misuse observability tools

“More tools means more observability.” Running LangSmith, Arize, and Helicone simultaneously doubles your observability cost without doubling your insight. Each tool captures overlapping data. Pick one and learn it deeply.

“Observability is a DevOps concern, not a product concern.” If the product team does not look at the observability dashboards, the data is decoration. Assign a named owner to review traces and evaluation scores weekly.

“We will add observability after launch.” Adding observability to a system that already handles production traffic means deploying instrumentation, backfilling comparison baselines, and configuring retention — all while the system runs. Teams that add observability later often skip sampling configuration and end up with runaway data costs.

Related guides

|- LLM observability basics: traces, prompts, evals and feedback loops — the conceptual framework this tool comparison builds on |- LLM observability cost: logs, traces and evaluation storage — budgeting guidance for storage and evaluation compute |- AI output monitoring: what to log, sample and review — privacy-aware logging patterns |- LLM-as-a-judge: when automated grading helps and when it lies — evaluation methodology for LLM outputs |- Eval CI for AI apps: testing prompts before every release — continuous evaluation in deployment pipelines

Methodology

• Data checked: 2026-05-30
• Sources consulted: LangSmith documentation (smith.langchain.com), Arize AI documentation (docs.arize.com), Helicone documentation (docs.helicone.ai), Weights & Biases Prompts documentation (docs.wandb.ai), Datadog LLM Observability documentation (docs.datadoghq.com), Portkey documentation (portkey.ai), LangFuse documentation (langfuse.com), OpenTelemetry semantic conventions for LLM (opentelemetry.io)
• Assumptions: Vendor pricing and feature sets are as documented in May 2026. All five tools change their pricing and feature sets frequently — the comparison is a snapshot. Tool recommendations assume the reader operates in a standard cloud environment (AWS, GCP, Azure) and uses major LLM providers (OpenAI, Anthropic, Google).
• Limitations: This article does not benchmark any tool on real workloads. Feature assessments are based on vendor documentation and community knowledge, not hands-on deployment testing. The article does not cover infrastructure-level monitoring (GPU utilisation, container health, network I/O). Pricing is indicative — check current vendor pricing for accurate figures. The article does not cover legal or compliance requirements for observability data retention.
• Jurisdiction: Global. GDPR and UK DPA referenced where applicable for data retention guidance.

Source list

• LangSmith observability platform — https://docs.smith.langchain.com/ (accessed 2026-05-30)
• Arize AI (Arize AX) LLM observability — https://docs.arize.com/ (accessed 2026-05-30)
• Helicone LLM observability — https://docs.helicone.ai/ (accessed 2026-05-30)
• Weights & Biases Prompts — https://docs.wandb.ai/guides/prompts (accessed 2026-05-30)
• Datadog LLM Observability — https://docs.datadoghq.com/llm_observability/ (accessed 2026-05-30)
• Portkey AI gateway and observability — https://portkey.ai/ (accessed 2026-05-30)
• LangFuse open-source LLM observability — https://langfuse.com/ (accessed 2026-05-30)
• OpenTelemetry LLM semantic conventions — https://opentelemetry.io/docs/specs/semconv/llm/ (accessed 2026-05-30)
• ICO guidance on AI data retention — https://ico.org.uk/for-organisations/ai-and-data-protection/ (accessed 2026-05-30)

Trust Stack

• Last checked: 2026-05-30
• Corrections: Contact us to report errors

Change log

• 2026-05-30: First draft. Covers LangSmith, Arize AI, Helicone, Weights & Biases, and Datadog LLM Observability with comparison table, decision framework, and per-tool setup guidance.