AI Agent Monitoring & Observability: Production Guide 2026

An AI agent that works in a demo fails differently in production: it fails softly, expensively, and often invisibly. Monitoring an AI agent means tracking three things at once — technical health (latency, errors), output quality (accuracy, task completion), and unit economics (token cost per resolved task) — because a degradation in any one of them looks like success in the other two. This guide covers what to measure, how to instrument it with open standards, what the tooling costs, and where teams usually get it wrong.

Why agent monitoring is different from application monitoring

Traditional APM assumes deterministic code: the same request produces the same response, so an error rate and a latency histogram tell you most of what you need. Agents break that assumption in four ways:

Non-deterministic outputs: the same input can produce different responses, so "200 OK" tells you nothing about correctness.
Soft failures: an agent can return a fluent, confident, wrong answer. No exception is thrown; nothing in your error dashboard moves.
Multi-step execution: a single user request may fan out into a dozen LLM calls, tool invocations, and retrieval steps. You need traces, not just logs, to see where a run went wrong.
Marginal cost per request: every call burns tokens. A prompt regression can double your inference bill overnight without any alert firing.

The practical consequence: you monitor agents on three layers — performance, quality, and cost — and you need trace-level visibility into each run, not just aggregate metrics.

The metrics that actually matter

Performance and cost

Instrument every LLM call with latency, token counts, model name, and finish reason. The OpenTelemetry GenAI semantic conventions standardize the attribute names: gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.response.finish_reasons. The conventions are still marked "Development" stability, so attribute names may shift, but adopting them now means your instrumentation stays compatible with Langfuse, Datadog, and every other backend that ingests OTLP.

The short list of metrics worth charting:

Latency: P50/P95/P99 end-to-end, plus time to first token for streaming interfaces.
Token usage: input and output tokens per call, labeled by model and by agent step. Output tokens cost 3 to 5 times more than input tokens on most models, so track them separately.
Cost per task: not cost per call. A support resolution might take 8 LLM calls; that is the unit that matters.
Error rate: provider errors, timeouts, and tool-call failures, labeled by type.
Retry amplification: how often retries multiply token spend. See our companion piece on AI agent error handling and retry strategies for the other half of this problem.

Quality

Quality metrics are what separate agent monitoring from ordinary APM. Track:

Task completion rate: did the agent actually resolve the request, measured against a defined success condition (ticket closed, booking confirmed, escalation avoided)?
Evaluation scores: sampled responses scored by an LLM-as-judge or a deterministic checker.
User feedback: thumbs up/down or explicit corrections, tied back to the trace.
Escalation rate: percentage of sessions handed to a human, and why.

A word of caution on hallucination detection: there is no reliable real-time "hallucination score" you can stream into a gauge. The honest version is sampling — score 1 to 5 percent of production traces with a judge model and track the trend, not an absolute number.

Business

Automation rate: sessions resolved without human intervention.
Cost per resolved task: total token plus infrastructure spend divided by successful resolutions.
Time saved: agent handling time versus the human baseline for the same task.

These are the numbers your CFO will ask about, and the ones that justify (or kill) the deployment.

A worked cost example

Suppose a support agent handles 10,000 conversations a month, each averaging 6 LLM calls of 2,000 input tokens and 400 output tokens on Claude Sonnet 4.5, priced at $3 per million input tokens and $15 per million output tokens per Anthropic's published pricing:

Monthly input tokens: 10,000 × 6 × 2,000 = 120M tokens × $3/MTok = $360
Monthly output tokens: 10,000 × 6 × 400 = 24M tokens × $15/MTok = $360
Total inference: about $720 a month, or $0.072 per conversation

Now the monitoring payoff: a prompt change adds a verbose 1,500-token system reminder to every call. Input cost rises 75 percent to $630 and nobody notices for three weeks because there is no per-call token alert — that is $270 a month in pure waste, plus degraded latency. A dashboard with cost per conversation and a daily anomaly check catches this on day one. Prompt caching (cache reads are billed at roughly a tenth of the input rate on Anthropic) is the standard fix for repeated context, which is another reason to track input tokens split by cached and uncached.

Instrumentation: traces first, then metrics

For agents, start with distributed tracing. Each user request becomes a trace; each LLM call, retrieval, and tool invocation becomes a span. With OpenTelemetry:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

async def handle_request(query: str):
    with tracer.start_as_current_span("support_agent") as span:
        span.set_attribute("gen_ai.operation.name", "invoke_agent")

        with tracer.start_as_current_span("chat claude-sonnet-4-5") as llm:
            llm.set_attribute("gen_ai.provider.name", "anthropic")
            llm.set_attribute("gen_ai.request.model", "claude-sonnet-4-5")
            response = await client.messages.create(...)
            llm.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
            llm.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)

One deliberate decision: the OTel conventions say prompts and completions should not be captured by default — recording them is opt-in via gen_ai.input.messages and gen_ai.output.messages, because full content creates storage cost and privacy exposure. Capture content on a sampled basis or for sessions flagged for review, and redact PII before export. If you operate in a regulated industry, this is where monitoring design meets AI governance requirements — retention, access control, and audit trails on the telemetry itself.

Choosing tooling

You have three realistic options:

Langfuse — open source (MIT for the core), self-hostable with all core features free, or cloud-hosted: Hobby tier free with 50k units a month, Core at $29/month, Pro at $199/month, with usage billed per unit (a unit is any trace, observation, or score) per the Langfuse pricing page. It ingests OTLP natively, so the instrumentation above flows straight in. Self-hosting means running ClickHouse, PostgreSQL, and Redis yourself — budget the operational time.
LangSmith — the natural fit if you build on LangChain or LangGraph. Developer plan is free with 5k base traces a month; Plus is $39 per seat per month with 10k traces included, then pay-as-you-go, per LangChain's pricing. Closed source; self-hosting is an Enterprise option.
General APM (Datadog and similar) — Datadog's LLM observability product (recently folded into "Agent Observability") starts around $160/month on annual commitment for 100k spans with 15-day retention, according to a November 2026 comparison of observability tools. Worth it when you want agent telemetry in the same pane as infrastructure metrics; expensive when agents are the main workload.

For a deeper signal-by-signal breakdown of what to emit, see our guide to LLM agent telemetry signals and monitoring best practices.

Our honest recommendation for most teams: start with self-hosted Langfuse or its free cloud tier, instrument with the OTel conventions, and only pay for a hosted platform when trace volume or compliance requirements justify it.

Alerting without the fatigue

Static thresholds page you for noise. A workable alerting setup for agents:

Error rate: page above 5 percent of runs over 15 minutes; investigate above 2 percent.
Cost anomaly: alert when hourly token spend exceeds twice the trailing 7-day average for the same hour. Traffic-driven spikes are normal; cost-per-task spikes are not.
Quality drift: alert when the sampled judge score drops more than half a point week over week, not on single bad responses.
Latency: alert on P95 relative to the agent's own baseline (a 10-step research agent is legitimately slower than a classifier), not an absolute number.

Every alert needs a runbook. The most common ones: prompt regression (check recent prompt or model changes first), provider degradation (fail over or queue), and cost spike (find the trace cohort responsible before changing anything).

Common mistakes

Monitoring only the LLM call: the agent's tools, retrieval, and orchestration logic fail too. Trace the whole run.
Aggregate-only quality metrics: a 4.2 average judge score hides the 3 percent of responses that are dangerously wrong. Track the tail.
No baseline period: anomaly detection needs two to four weeks of normal operation data before it can flag anything sensibly.
Logging everything forever: full prompt and completion logging at scale is a storage bill and a privacy liability. Sample, redact, and set retention windows.
Dashboards nobody acts on: every chart should map to an alert threshold or a decision. Delete the rest.

Frequently asked questions

What is the difference between AI agent monitoring and observability?

Monitoring tracks known failure modes against predefined metrics and thresholds. Observability is the ability to investigate failures you did not predict, using traces, logs, and recorded runs. You need both: monitoring to catch regressions quickly, observability to debug the weird ones.

Which tool should I start with for LLM observability?

Self-hosted Langfuse is free under MIT with all core features, and its cloud Hobby tier gives you 50k units a month at no cost. Because it accepts standard OpenTelemetry traces, you are not locked in if you later move to LangSmith, Datadog, or an in-house stack.

How much does it cost to monitor AI agents in production?

The platform cost ranges from free (self-hosted Langfuse, LangSmith Developer) to $39 per seat per month (LangSmith Plus) or $199 per month (Langfuse Pro). The often-larger cost is the observability data itself: tracing roughly doubles your instrumentation overhead, and judge-model evaluations add inference spend. Budget 5 to 15 percent of your inference bill for evaluation and tracing.

How do you detect hallucinations in production?

You sample. Pick 1 to 5 percent of traces, score them with a judge model against the retrieved context or source data, and track the trend. Real-time hallucination blocking is unreliable; trend detection plus human review of flagged samples is what works.

Do I need monitoring before I have production traffic?

Yes, but keep it simple. Structured logs and basic tracing from day one cost almost nothing and give you the baseline data you need later. Skip the elaborate dashboards and anomaly detection until you have real volume.

Where to go from here

If you are running agents in production — or planning to — the monitoring layer is not an afterthought; it is what makes the deployment defensible to your finance team and your risk team. We build and operate always-on agents with this telemetry built in from the first day. Take a look at our AI employees service to see how a production agent is run end to end.

AI Agent Monitoring and Observability: Production Guide for 2026