Updated 6 Oct 2026 • 5 mins read
LLM Observability
LLM observability is the practice of tracing every model call an application makes, capturing the prompt, model, tokens, cost, latency, and quality, so teams can debug, evaluate, and attribute spend. This guide explains why conventional monitoring falls short, the four signals to capture, the OpenTelemetry standard, the tools, and the cost connection.
Traditional monitoring answers a question that LLM applications make almost irrelevant: is the service up? A request to a language model can return in 800 milliseconds with a 200 status code, consume forty thousand tokens, cost a dollar, and give the user a confident, fluent, completely wrong answer. Every conventional health check passes. Nothing in the dashboard changes. And the bill, which is accruing per request rather than per server, grows in a way the dashboard was never designed to show.
LLM observability is the discipline that fills that gap: recording, for every model call, what was asked, what was answered, what it cost, how long it took, and whether it was any good, and tying those records to the feature, user, and agent run that caused them. This guide explains why monitoring alone falls short, lays out the four signals that matter and what to capture for each, points to the open standard for instrumenting them, surveys the tools, and makes the connection most observability guides skip: that per-call tracing is also the only way to know what any of this costs.
The short answer: LLM observability is the practice of instrumenting applications that call large language models so that every call is recorded with its prompt, model, tokens consumed, cost, latency, tool calls, and response, grouped into traces that represent a complete user action or agent run. It differs from conventional monitoring because LLM calls are non-deterministic (success is not a status code), are billed per request (cost must be traced to its cause), and change behavior when prompts, models, or retrieved context change. The four signals to capture are traces, quality, latency, and cost. The OpenTelemetry GenAI semantic conventions standardize how to record them. Tools range from open-source Langfuse and Arize Phoenix to LangSmith, Braintrust, Portkey, Helicone, and Datadog. Beyond debugging, this instrumentation is the prerequisite for attributing AI spend to the teams and features that generate it.
Key Takeaways
- LLM observability records every model call with prompt, model, tokens, cost, latency, and response, grouped into traces for each user action or agent run.
- It exists because LLM calls break monitoring's assumptions: a 200 response can be wrong, cost is per request not per server, and behavior shifts with prompt and model changes.
- Capture four signals: traces (what happened), quality (was it right), latency (how fast, including time to first token), and cost (what it consumed, attributed to a cause).
- Instrument against the OpenTelemetry GenAI semantic conventions so traces are portable across tools.
- Per-call observability is also the foundation of AI cost management: without it, token spend cannot be attributed to features, teams, or customers.
Why Monitoring Falls Short for LLM Applications
Application performance monitoring was built for deterministic software, and three properties of LLM calls break its assumptions.
- Non-determinism. The same prompt can yield different outputs, and a syntactically perfect response can be factually wrong, off-topic, or unsafe. Status codes and error rates measure whether the call completed, not whether it succeeded in any sense the user cares about. Quality has to be measured separately, and it has to be measured on the output.
- Per-request cost. A server costs the same whether it handles one request or a million. An LLM call costs in proportion to the tokens it reads and writes, so cost is a property of each request and of the prompt that shaped it. Without recording tokens and cost per call, the bill is a total with no explanation.
- Behavioral drift without deploys. A prompt edit, a model version upgrade, a change in the documents a retrieval system returns, or a provider-side model update can change behavior and cost overnight with no code deploy to blame. Observability has to capture prompt and model versions so a regression can be traced to its cause.
These are the same properties that make AI spend hard to govern, which we discuss in our guide to FinOps for AI: the cost is set by code and traffic rather than by provisioning, and there is nothing to tag. Observability is how the tag gets attached.
The Four Signals to Capture
1. Traces: what actually happened
A trace is the full record of one user action or agent task, composed of spans for each step: every model call, every tool invocation, every retrieval, every retry. For a simple chatbot a trace is one span; for an agent it may be forty. The trace is the unit that makes everything else interpretable, because it is the level at which a user perceives one action and at which cost and quality should be judged. Each span should record the model and provider, the prompt and system instructions, the retrieved context, the response, token counts, and the tool calls made.
2. Quality: was it any good
Because a completed call is not a correct one, quality must be measured on outputs. In practice that means a mix of automated evaluations (an LLM or heuristic grading responses against criteria such as faithfulness to retrieved context, relevance, format compliance, and safety), user feedback signals (thumbs up or down, edits, abandonment), and periodic human review of sampled traces. Evaluation scores attached to traces turn a prompt change from a guess into an experiment.
3. Latency: how fast, and where the time went
LLM latency has two numbers that matter: time to first token, which governs perceived responsiveness in streaming interfaces, and total duration. Both vary with model, prompt length, and provider load. In agentic systems the trace shows where the time accumulates, often in sequential tool calls rather than in the model itself, which is a different fix.
4. Cost: what it consumed, and who caused it
Every span should carry input tokens, output tokens, and the cost computed from the model's current rates, including reasoning tokens that providers bill as output and cache reads billed at a discount. Traces then carry the total cost of a user action, and metadata on each trace (feature, team, tenant, environment) makes that cost attributable. This is the signal most observability guides mention last and the one that matters most to anyone paying the bill. Our guide to LLM token costs by model covers the rates that turn token counts into dollars.
What to Record on Every Call
| Attribute | Why it matters |
|---|---|
| Model and provider (including version) | Cost rate, behavior, and regression tracing across model updates |
| Prompt, system instructions, and prompt version | Reproduce issues; attribute behavior and cost changes to prompt edits |
| Retrieved context (or a reference to it) | Diagnose hallucinations and faithfulness failures |
| Response | Evaluate quality; audit what users were told |
| Input, output, and reasoning token counts | Cost; detect bloated prompts and verbose outputs |
| Cache hit indicators | Verify caching is working; cache reads cost a fraction of input |
| Latency and time to first token | User experience; locate slow steps in agents |
| Tool calls and their results | Agent debugging; count calls per action |
| Errors, retries, and fallbacks | Reliability; retries bill tokens every attempt |
| Trace ID, feature, team, tenant, environment | Group spans into actions; attribute cost and quality to owners |
Instrument Against the Open Standard
The OpenTelemetry GenAI semantic conventions define standard attribute names for recording generative AI operations as spans: the provider and model, request parameters, token usage, prompts and completions, and tool calls. Instrumenting against them has one decisive benefit: your traces are portable. The same data can flow to an open-source backend today and a commercial platform next year without re-instrumenting the application, and most of the tools below either emit or ingest this format. For teams already running OpenTelemetry for conventional tracing, LLM spans become part of the same trace as the HTTP request and database query that surround them, which is where they belong.
The Tools
The market has consolidated and repositioned quickly through 2026; verify current positioning before selecting.
| Tool | Model | Known for |
|---|---|---|
| Langfuse | Open source, self-host or cloud | Tracing, prompt management, evaluations; widely adopted OSS default |
| Arize Phoenix | Open source | Tracing and evaluation, OpenTelemetry-native, strong on RAG analysis |
| OpenLLMetry (Traceloop) | Open source | OpenTelemetry instrumentation for LLM frameworks and providers |
| LangSmith | Commercial (LangChain) | Tracing, evals, and prompt tooling, deepest for LangChain and LangGraph |
| Braintrust | Commercial | Evaluation-centric platform with tracing and prompt playgrounds |
| Weights & Biases Weave | Commercial | Tracing and evaluation integrated with ML experiment tracking |
| Portkey | Commercial gateway | AI gateway with routing, guardrails, caching, and observability in the request path |
| Helicone | Commercial gateway (acquired by Mintlify in 2026) | Proxy-based logging and cost tracking with minimal integration |
| Datadog LLM Observability | Commercial, part of Datadog | LLM traces alongside infrastructure and APM in one platform |
A useful distinction when choosing: gateways such as Portkey and Helicone sit in the request path and can act (route, cache, block) as well as observe, while tracing platforms such as Langfuse, Phoenix, and LangSmith sit alongside the application and observe deeply. Many teams run one of each. For the cost-specific end of this landscape, our guide to the best AI cost observability tools goes further, and the AI cost visibility piece covers what the cost layer should show once the traces exist.
A Worked Example: What Tracing Reveals About an Agent
A research-assistant feature lets users ask a question and get a cited summary. Without observability, the team knows it costs about $14,000 a month and that users like it. With traces, the picture changes. A typical user action turns out to be 34 model calls: a planning call, eleven search-and-read steps each with a tool call and a summarization call, several retries caused by a flaky search integration returning malformed results, and a final synthesis on a frontier model. The retries alone are 9 of the 34 calls. The eleven summarization steps run on the frontier model because the prototype did, though a mid-tier model scores identically on the team's faithfulness evaluation.
Three changes follow directly from what the trace shows and from nothing else: fix the search integration to remove most retries, route the summarization steps to the mid-tier model, and cache the shared system prompt. Cost per action falls from about $0.47 to about $0.11, the monthly bill from $14,000 to roughly $3,300, and the evaluation scores do not move. None of those changes were visible in the total; all of them were obvious in the trace.
Observability as the Foundation of AI Cost Management
This is the connection the observability conversation and the cost conversation usually miss about each other. AI cost allocation is impossible without per-call data, because provider invoices record only the API key, model, and token count; which feature, team, or customer caused the spend exists only in your application at the moment of the call. LLM observability is the system that captures it. Every trace with feature, team, and tenant metadata is a cost record as much as a debugging record, and summing those records is how an undifferentiated token bill becomes cost per feature and per customer, the method in our guide to AI unit economics. The Tokenomics Foundation's 2026 survey found that the organizations able to prove AI value to their CFO were the ones whose spend was metered and attributable per workload, which is precisely what this instrumentation provides. Feeding those traces into a cost platform that sits alongside the rest of the cloud bill is what Opslyft's cost visibility does, so the same data that debugs the agent also explains its line on the invoice.
How to Start
- Instrument every model call through one shared client wrapper, emitting OpenTelemetry GenAI spans with model, tokens, cost, latency, and your own feature, team, and tenant attributes.
- Group calls into traces per user action or agent run with a shared trace ID, so cost and quality are judged at the level users experience.
- Add automated evaluations on sampled traces for the two or three quality criteria that matter most to your product, and attach user feedback where you can.
- Version prompts and record the version on every span, so regressions trace to the edit that caused them.
- Pick a backend, open-source first if you want to defer the vendor decision, and connect the cost fields to whoever owns the AI budget.
- Review traces, not dashboards, when something changes: the trace is where the answer is.
The Bottom Line
LLM observability exists because language models broke the assumptions monitoring was built on: a completed call is not a correct one, cost belongs to the request rather than the server, and behavior changes without a deploy. Capturing traces, quality, latency, and cost for every call, against the open standard and with the metadata that names a cause, restores the ability to debug, to evaluate, and to explain. It also happens to be the only way to know what an AI feature costs and whether it is worth it, which makes it less an engineering nicety than the foundation of the tokenomics discipline that enterprises are now building. Instrument first; everything else about managing AI, from quality to cost, depends on it.
FAQs
LLM observability is the practice of instrumenting applications that call large language models so you can see what each call did: the prompt and model used, the tokens consumed and their cost, the latency, the response, and whether it was any good. It extends traditional application monitoring with signals that do not exist for ordinary software, because LLM calls are non-deterministic, billed per token, and can fail by returning a plausible wrong answer rather than an error.
Three reasons. LLM calls are non-deterministic, so the same input can produce different outputs and a 200 status code says nothing about whether the answer was correct. Cost is incurred per request rather than per server, so spend has to be traced to the prompt, feature, and user that caused it. And behavior changes when prompts, models, or retrieved context change, none of which a conventional deploy pipeline tracks. Monitoring tells you the service is up; LLM observability tells you what it said, what it cost, and whether it was right.
For every model call: the model and provider, the prompt and any retrieved context, the response, input and output token counts, cost, latency including time to first token, the tool calls made, errors and retries, and metadata linking the call to a feature, a user or tenant, and the parent request or agent run. For agents, calls are grouped into traces so a single user action spanning dozens of calls can be seen as one unit with one cost.
Open-source options include Langfuse, Arize Phoenix, and OpenLLMetry from Traceloop. Commercial platforms include LangSmith from LangChain, Braintrust, Weights and Biases Weave, Portkey, Helicone, and Datadog's LLM Observability within its wider platform. Most support the OpenTelemetry GenAI semantic conventions, which standardize how LLM calls are recorded so tooling can be swapped.
A standard, published by the OpenTelemetry project, for how to record LLM and other generative AI operations as spans: standardized attribute names for the model, provider, token counts, prompts, responses, and tool calls. Instrumenting against them means your traces are portable across observability backends rather than tied to one vendor's format.
It is the mechanism that turns an undifferentiated token bill into cost per feature, per team, and per customer. Because cost is recorded per call with metadata, you can see which features are expensive, which agents are making too many calls, where retries are burning tokens, and which model each request actually ran on. Without per-call observability, AI cost allocation is impossible; with it, routing, caching, and prompt optimization become measurable.