LLM Observability: Monitoring LLMs in Production

LLM Observability is the practice of instrumenting, collecting, and analyzing operational data from language models in production: latency, tokens consumed, error rate, response quality, and cost per call. The goal is not observing for its own sake. It is having enough visibility to decide when to switch models, adjust prompts, or reroute traffic before the user notices degradation. For more on model selection, see our LLM benchmarking guide.

What is LLM Observability?

LLM Observability extends the three classic pillars of observability (logs, metrics, tracing) to the specific context of language models. The core difference is in the object being monitored: not a deterministic microservice, but a probabilistic model that answers the same question differently, consumes tokens variably, hallucinates under certain conditions, and degrades in ways a CPU graph cannot capture.

The term gained traction in 2024 with the launch of specialized tools like LangSmith, Arize Phoenix, and Helicone. But the problem started earlier: engineering teams that put GPT-3.5 and GPT-4 into production in 2023 discovered their traditional APM dashboards were blind to what actually mattered.

Why doesn't traditional monitoring work for LLMs?

APM (Application Performance Monitoring) tools like Datadog, New Relic, and Grafana were designed for deterministic software. A successful HTTP request always returns status 200 with a predictable payload. For LLMs, status 200 can mean a correct answer, a convincing hallucination, or a technically accurate response that is completely irrelevant to the user.

Four fundamental differences make traditional APM insufficient:

Non-determinism. The same prompt, the same model, the same temperature parameter generates different responses. A binary error-rate dashboard (success/failure) is blind to qualitative degradation.

Variable cost per call. A REST request consumes predictable CPU and memory resources. An LLM call can consume anywhere from 50 to 50,000 tokens in the same operation, depending on the prompt, the context, and the model's verbosity at that specific moment.

Composite latency. Response time is not just inference time. It includes tokenization, streaming, post-processing, and, when function calling or RAG is involved, multiple chained calls. A latency spike can originate in the model, the retrieval step, or the prompt engineering.

Silent degradation. A degraded database returns errors or timeouts. A degraded LLM keeps responding, but the answers lose precision, context, or relevance. The system looks healthy on traditional dashboards while delivering zero value to the user.

Which metrics actually matter in production?

The industry has converged on four categories of metrics. Separating signal from noise here is what distinguishes teams running LLMs at scale from teams that only build prototypes.

Operational metrics

The metrics any production service demands, adapted to the LLM context:

• Latency: time to first token (TTFT) and total response time. TTFT below 200ms is the threshold of perceived fluidity for the user. Streaming models mask latency; measuring both times is mandatory.
• Throughput: tokens per second (output and total). Output throughput below 20 tokens/s degrades the reading experience.
• Error rate: API errors (rate limiting, timeout, context exceeded) and application errors (invalid response, parse failure in structured output).
• Availability: model endpoint uptime, including partial degradation (fallback activated counts as available if the user didn't notice).

Cost metrics

Cost is the variable that surprises teams the most when moving from prototype to production. In a prototype, cost is invisible. In production with hundreds of thousands of daily calls, it becomes the largest line in the infrastructure budget.

• Cost per call: monetary value per request, accounting for input and output tokens with model-specific pricing.
• Cost per session: when the product involves multiple calls per user interaction (chat, multi-step RAG, agents).
• Cost per active user: a business metric connecting engineering to product. If cost per user exceeds revenue per user, the product is unsustainable regardless of technical quality.
• Cost trend: day-over-day percentage variation. Spikes of 30% or more in a single day without a corresponding traffic increase indicate prompt regression or model behavior change.

A real case: Intercom's support platform reported in 2024 that moving certain queries from GPT-4 to GPT-4o-mini cut cost per ticket by 80% with no measurable loss in customer satisfaction. Without per-call cost visibility, this optimization would have gone undiscovered for months.

Quality metrics

Quality is the hardest dimension to measure and the most important. Without it, the team is optimizing latency and cost blindfolded.

• Human evaluation (human eval): sampling of responses reviewed by domain experts. Expensive and slow, but irreplaceable for calibrating automated metrics.
• LLM-as-judge: a stronger model evaluates the production model's responses. Common metrics: relevance, factuality, completeness, toxicity. Requires careful calibration because the judge also makes mistakes.
• Similarity metrics: BLEU, ROUGE, BERTScore. Useful for detecting drift (the model's response today differs from yesterday's response to the same prompt) but weak for measuring absolute quality.
• Hallucination rate: detected via factuality verification (grounding check), NLI (natural language inference), or inconsistent citation detection when RAG is in use.
• User rejection rate: the most underestimated metric. If the user reformulated the question, ignored the response, or abandoned the session, the system failed, regardless of what the technical metrics say.

Security and compliance metrics

• PII leak rate: personal data (SSN, email, phone number) appearing in responses or being sent to the model.
• Toxicity and bias: toxicity score per response, segmented by language and user profile.
• Jailbreak attempts: attempts to bypass system instructions, detected and blocked.

How does LLM call tracing work?

Tracing is the observability pillar that connects an LLM call to its full context: the prompt sent, parameters, response received, tokens consumed, latency of each step, tools called (function calling), and documents retrieved (RAG).

In LLM-based systems, a single user interaction often triggers a chain of 3 to 15 calls. An agent searches a knowledge base, decides whether it needs more data, calls an external API, and then formulates the final response. Without tracing, debugging this chain is impossible: you know the final result was bad, but you don't know which step broke.

The emerging industry standard is OpenTelemetry adapted for LLMs, with extensions that capture:

• Nested spans: each LLM call, retrieval, and function call generates a span with timestamps, tokens, and model metadata.
• LLM-specific attributes: llm.model_name, llm.prompt_template_version, llm.temperature, llm.total_tokens, llm.output.content.
• Session correlation: all calls within a user interaction share a common trace_id.

Tools like Langfuse and Arize Phoenix implement this standard with SDKs that automatically instrument LangChain, LlamaIndex, and direct calls to the OpenAI API. The build vs. buy decision here depends on volume: below 10,000 calls/day, a ready-made solution is cheaper than the engineering cost of maintaining a custom stack. Above 100,000 calls/day, the external tool's cost competes with the model cost itself, and the equation flips.

Comparison table: observability instrumentation approaches

The choice is not mutually exclusive. A mature stack combines routing (base collection) with a specialized platform (analysis and alerts). The router captures everything passing through it without requiring instrumentation of every service. The platform adds dashboards, anomaly detection, and quality evaluation.

Why is the router the ideal instrumentation point?

Teams operating multiple models in production face a dispersion problem: each provider (OpenAI, Anthropic, Google, Groq) has its own log format, its own latency profile, and its own error mechanism. Instrumenting each integration separately duplicates effort and fragments visibility.

The LLM router solves this by definition. It is the only component that sees all traffic, regardless of the target model. Instrumenting observability at the routing layer means:

Full coverage with a single collection point. Any model added to the roster inherits tracing, latency metrics, and token counting automatically. There is no uninstrumented code path.

Visible routing decisions. Tracing captures not just the call to the selected model, but also the logic that selected it: why was GPT-4o chosen over Claude 3.5 Sonnet for this specific request? That information is critical for auditing and optimizing the routing strategy.

Observable failover. When a model fails and the router redirects to the fallback, tracing records the full event: original model, error, fallback model, additional latency. Without this visibility, the engineering team discovers the failover through cost increase (the fallback is usually a more expensive model), not through monitoring.

Consolidated cost by model, by tenant, by feature. The router aggregates spending from all providers into a single view. Teams using 4 or 5 different models without routing often discover their actual cost when the provider's bill closes, 30 days later.

Nexforce Router implements this model: native tracing and metrics at the routing layer, exposed via OpenTelemetry for integration with any existing observability stack. This eliminates the need to instrument every service that consumes LLMs.

Implementation: 5 steps to instrument LLM observability

Order matters. Teams that start with beautiful dashboards before having reliable collection waste weeks and abandon the initiative.

1. Define the 5 metrics that pay the bill

Before installing any SDK, answer this: which 5 metrics, if monitored, justify the investment in observability? The answers vary by product. A support chatbot prioritizes hallucination rate and user satisfaction. A document extraction API prioritizes latency and parse failure rate. An autonomous agent prioritizes cost per completed task and completion rate.

Choosing 5 metrics forces prioritization. Choosing 20 guarantees none will actually be tracked.

2. Instrument the routing layer first

If your stack has an LLM router, instrument there. If it doesn't, place a reverse proxy (or a router like Nexforce Router) between the application and the providers. The immediate gain: tracing of all calls without changing a single line of code in existing services.

Routing instrumentation captures the minimum viable start: model called, input/output tokens, latency, response status, estimated cost. This set covers 80% of production incidents with LLMs.

3. Add quality evaluation via sampling

Don't instrument 100% of responses with LLM-as-judge from day 1. Evaluation cost competes with inference cost. Start with 5% sampling of responses, evaluated by a cheaper and faster model (GPT-4o-mini as judge for GPT-4o, for example). Increase sampling when the cost of an undetected error exceeds the cost of evaluation.

4. Set up alerts on thresholds that break the product

Generic alerts create fatigue. Specific alerts prevent incidents. Examples of thresholds that justify immediate alerting:

• P95 latency above 5 seconds for more than 10 minutes
• Daily cost 40% above the 7-day moving average
• Error rate above 2% (models returning 5xx)
• Hallucination rate above 5% in the evaluated sample
• Primary model availability below 99% (indicating excessive failover)

5. Close the loop with weekly review

Observability without action is an expense, not an investment. Every week, the engineering team reviews: which models have rising latency? Where is cost growing faster than traffic? Does any prompt need adjustment? Is the most expensive model being used for queries the cheaper model handles fine?

Teams that close this loop consistently reduce LLM cost by 30 to 60% in the first three months, without degrading quality.

Costs and trade-offs of LLM observability

Observability is not free. The costs fall into three categories:

Infrastructure cost. Storing LLM call traces and logs consumes significant volume. Each call generates 2 to 50 KB of tracing data, depending on prompt and response size. At 1 million calls per month, that's 2 to 50 GB of observability data. SaaS tools charge by ingested volume or by tracked call. Self-hosted requires a columnar database (ClickHouse, Pinot) for efficient time-series queries.

Evaluation cost. LLM-as-judge consumes tokens. Evaluating 5% of 1 million calls with a cheap model costs between USD 50 and USD 200 per month, depending on average response size and evaluation prompt. Money well spent if it prevents a quality incident. Waste if the sampling threshold is never reviewed.

Engineering cost. Instrumenting, configuring dashboards, tuning alerts, and conducting weekly reviews consumes 5 to 15 hours of engineering per week in the first month, dropping to 2 to 5 hours after stabilization. Teams that underestimate this cost install tools no one ever consults.

The central trade-off: insufficient observability = operating blind. Excessive observability = paying more to observe than to infer. The sweet spot is when the monthly observability cost sits between 5% and 10% of the monthly inference cost.

Common mistakes when monitoring LLMs in production

Monitoring only latency and ignoring cost. The most frequent mistake. The team celebrates P99 at 800ms while spending USD 15,000 per month on GPT-4o for queries GPT-4o-mini would handle for USD 2,000. Low latency bought with model overprovisioning is not a technical win.

Treating the LLM as a complete black box. Some teams give up on instrumentation because "models are too unpredictable." They are unpredictable, but they generate rich telemetry. Tokens, temperature, prompt template version, and model used are deterministic variables that explain most of the variation in quality and cost.

Alerting on absolute thresholds without a baseline. A "latency above 3 seconds" alert fires every week when the heavier model is used legitimately. The correct threshold is relative: latency 50% above the baseline for the same model at the same time of day.

Using similarity metrics as a quality proxy. BLEU and ROUGE measure lexical overlap, not quality. A response can score high on BLEU by repeating parts of the prompt and still be completely useless. Similarity detects drift, not quality.

Instrumenting everything before having a product in production. Teams that spend two weeks configuring perfect tracing before launching anything are optimizing something that doesn't exist. Start with basic logging. Add tracing when call volume makes log-based debugging infeasible. Add LLM-as-judge when the cost of an error justifies the cost of evaluation.

Ignoring context cost. Input token cost often exceeds output token cost in RAG applications, where each call carries dozens of documents in the context. A trace showing only total tokens without separating input from output hides the fact that 80% of the cost is in the context, not the response.

FAQ

What's the difference between observability and monitoring for LLMs?

Monitoring answers "is the system working?" (latency, error rate, availability). Observability answers "why is the system behaving this way?" (which step in the chain degraded, which prompt caused a hallucination, which model is costing more than expected). For LLMs, monitoring without observability is a green dashboard while the user gets a wrong answer.

Do I need a specialized tool or can I use Datadog/Grafana?

Datadog and Grafana handle operational metrics (latency, error rate). They do not handle LLM chain tracing, response quality evaluation, or cost per model. The typical stack combines both: traditional APM for infrastructure, a specialized tool (Langfuse, Arize, Helicone) for the LLM layer, and a router for unified collection.

How much does it cost to implement LLM observability?

It depends on volume. At 100,000 calls per month, SaaS tools cost between USD 50 and USD 300 per month. Self-hosted (Langfuse open source on your own instance) costs USD 50 to USD 150 in infrastructure plus maintenance engineering. The real cost is not in the tool: it's in the engineering hours to instrument, configure alerts, and review metrics.

Is it worth instrumenting if I use only one model?

Yes. Even with a single model, tracing reveals prompt degradation, latency increases, and cost variation that traditional logs don't capture. The difference is that with one model the urgency is lower: without routing, you aren't making decisions about which model to use, so tracing primarily serves debugging and cost optimization.

Does LLM Observability replace offline evaluation?

No. Observability detects problems in production. Offline evaluation (benchmarks, test sets, eval sets) detects problems before deployment. The two practices are complementary. A model that passes every benchmark can degrade in production because of poorly constructed prompts, insufficient context, or users using the system in unexpected ways.

How do I choose between build and buy for observability?

Below 50,000 calls per month: a SaaS solution is cheaper than the engineering cost of maintaining a custom stack. Between 50,000 and 500,000: self-hosted open source (Langfuse, Phoenix) reduces licensing cost but requires 2 to 4 hours per week of maintenance. Above 500,000: the volume justifies a dedicated team and a custom stack on ClickHouse or Pinot, with a router providing the base collection.

Where to go deeper

LLM observability is a rapidly consolidating domain. The practices described here cover the state of the art as of mid-2025. Three directions to continue:

• If your stack uses RAG, read about retrieval evaluation (NDCG, MRR, hit rate) and how to integrate retrieval metrics into LLM tracing.
• If your stack uses autonomous agents, investigate multi-step agent tracing: the complexity of debugging an agent making 20 chained calls demands specific tooling.
• If you manage costs across multiple providers, consult our practical guide to LLM benchmarks and study cost-based dynamic routing: the router selects the cheapest model that meets the quality threshold for each query.

Nexforce Router implements all three cases: native metrics and tracing collection at the routing layer, exposure via OpenTelemetry for any observability stack, and dynamic routing that weighs latency, cost, and quality when deciding which model handles each request. The instrumentation that would normally require weeks of SDK integration per service is concentrated at a single point in the architecture.

References and Further Reading

• OpenTelemetry
• Langfuse - Open Source LLM Engineering Platform
• Arize Phoenix - AI Observability
• Helicone - LLM Observability
• Intercom - How we cut LLM costs without sacrificing quality