AI agent observability tools record what an agent did on every run: each model call, tool call and retrieval step, with its inputs, outputs, latency, token cost and a quality score. That lets you find the bad runs and see why they went wrong. The main options are Langfuse, LangSmith, Arize Phoenix, Helicone, Braintrust, W&B Weave and Datadog LLM Observability, plus the OpenTelemetry GenAI semantic conventions as the open standard for the data itself. Below is what a setup must capture, a neutral comparison with prices read on each vendor's pricing page on 23 October 2026, and how to choose by team situation. Gigabit is not a partner of any of these vendors and earns nothing from your choice.
Why agents need more than ordinary monitoring
Uptime and error-rate dashboards stay green while an agent gets worse, because a wrong answer returns HTTP 200 like a right one. Quality drifts when a vendor changes a model, a policy document goes stale or traffic shifts, and nobody notices for weeks; why AI systems break after launch covers the five ways it happens. Agents add a second problem: a run is a chain of steps, and small per-step error rates multiply. When a multi-step AI agent fails in production does the arithmetic. Observability exists so you can see inside the chain.
What agent observability must capture
Use this as a requirements list when you evaluate any tool, including one you build yourself.
| Signal | What to record | Why it matters |
|---|---|---|
| Traces | One trace per request, with a nested span for each model call, tool call, retrieval and sub-agent, in order | You can replay what the agent did, step by step |
| Inputs and outputs | Prompt and its version, retrieved context, tool arguments and results, final answer | You cannot debug what you did not keep |
| Latency | Per span and end to end, watched at the median and the slow tail | A slow tool looks like a slow agent until you split it out |
| Token usage and cost | Input and output tokens per model call; cost per run and per resolved case | Retry loops burn money quietly, and cost per outcome is the number finance asks for |
| Eval scores | Offline scores on a fixed test set for every release; online scores on a sample of live runs | Turns "it feels worse" into a number with a date |
| User feedback | Ratings, CSAT, escalations and human edits to drafts, joined to the trace ID | Real users find failures your test set missed |
| Versions | Model, prompt, tool and retrieval index version on every trace | Links a quality drop to the change that caused it |
| PII handling | Redaction before storage, retention limits, access control, data location | Traces hold customer messages word for word |
| Alerts | Cost spikes, latency, tool errors, eval score drops, escalation rate | Someone hears about a problem before a customer does |
A quick test for any setup: pick one run from yesterday and answer these five questions in under five minutes. What did the user ask? What did the agent retrieve, and which tools did it call, in what order, with what arguments? What did each step return? What did the run cost and how long did it take? Which model and prompt version produced it? If any answer needs a developer and a log search, the setup is not finished. Our AI operations runbook explains how to write the trace-reading section around that test.
On scores, two posts go deeper than this one: what to measure in an AI agent eval suite for which metrics should block a deploy, and calibrating LLM-as-judge for automated scoring of live runs you can trust.
The open standard: OpenTelemetry GenAI conventions
OpenTelemetry is the vendor-neutral standard most infrastructure monitoring already uses. Its GenAI semantic conventions define spans, metrics and events for generative AI: model call spans, token usage metrics and agent spans, with operation names such as invoke_agent and execute_tool. Their status, checked on 23 October 2026, is "Development", so attribute names can still change. Some content fields are opt-in; system instructions, for example, are not recorded unless you choose to. The conventions are a format, not a product: you still need a backend to store, search and score the traces. Instrumenting against them is the cheapest insurance against being locked into one vendor.
The tools compared
Every vendor bills on a different unit: units, traces, spans, requests or gigabytes. Entry prices below are the lowest paid tier each vendor lists, read on 23 October 2026, in US dollars per month.
| Tool | Open source | Self-host option | Pricing model and entry price | Best at |
|---|---|---|---|---|
| Langfuse | Yes, MIT licence | Yes, all core features free; Enterprise self-host at custom price | Per unit, where traces, observations and scores each count. Hobby free with 50k units; Core $29 with 100k units, then $8 per extra 100k | Tracing, prompt management, evals and datasets in one tool you can run yourself at no licence cost |
| LangSmith | No | Enterprise plan only, hybrid or fully self-hosted | Per seat plus traces. Developer free for one seat and 5k traces; Plus $39 per seat with 10k traces, then pay as you go | Teams building on LangChain or LangGraph, which come from the same company |
| Arize Phoenix and Arize AX | Phoenix is source-available under the Elastic License 2.0 | Phoenix with Docker or Kubernetes; AX self-hosted on Enterprise | Phoenix free to run yourself. AX Free with 25k spans; AX Pro $50 with 50k spans | Tracing and evals built on OpenTelemetry that you can start locally and move to a managed service later |
| Helicone | Yes, Apache 2.0 | Yes, with Docker or Helm | Per request. Hobby free with 10,000 requests; Pro $79 plus usage; Team $799 | The fastest start: route model calls through its gateway by changing the base URL, then see cost and latency per request |
| Braintrust | No | Enterprise plan, with the data plane in your cloud | Per GB of processed data plus scores. Starter free with 1 GB and 10k scores; Pro $249 with 5 GB and 50k scores | Eval-led teams: experiments, scoring and regression tests next to production logs |
| W&B Weave | SDK under Apache 2.0 | Self-managed needs a W&B platform install and a Weave-enabled licence | Per GB ingested. Free with 1 GB; Pro from $60 with 1.5 GB, then $0.10 per MB | Teams already training models in Weights & Biases |
| Datadog LLM Observability | No | No, it runs in Datadog's cloud | Per LLM span; tool, workflow and retrieval spans are not billed. Free up to 40K LLM spans; Pro from $160 billed annually ($200 month to month) for 100K | Teams already on Datadog who want agent traces next to infrastructure and APM data |
| OpenTelemetry GenAI | Yes, an open standard | Not applicable; send data to any backend that accepts it | Free | Instrumenting once and keeping the freedom to switch backends |
Retention differs as much as price. On the free tiers whose retention is published, it ranges from 7 days (Helicone Hobby) to 30 days of data access (Langfuse Hobby); Langfuse Pro gives 3 years of data access, LangSmith keeps base traces 14 days and extended traces 180 days, and Datadog sells 30, 60 and 90-day retention as add-ons. If you need a year of traces for audits, price that in before anything else.
Price on your own volume
Because the units differ, count your own. Take a typical run and list its steps. An agent that makes three model calls, two tool calls and one retrieval produces one trace and six spans per run. At 20,000 runs a month, that is 140,000 units on Langfuse, which counts the trace and every observation. On Datadog it is 60,000 LLM spans, because only the model calls are billed. Request-based and data-based pricing need a different count again: model requests per run for Helicone, gigabytes of trace data for Braintrust and Weave. Run that arithmetic for your top two candidates before you compare headline prices.
How to choose
Pick by situation, not by feature list. The products above overlap heavily on core tracing, and the differences that decide most choices are hosting, the pricing unit and what each was built around.
- One agent, small team, want traces this week. Helicone's gateway is a base-URL change. Langfuse's free Hobby tier or $29 Core tier is the other easy start.
- Data must stay on your own infrastructure (health, finance, strict EU customers). Self-host Langfuse (MIT) or Helicone (Apache 2.0) at no licence cost, or Phoenix under the Elastic License. If you want a vendor-supported private deployment, LangSmith, Braintrust, Arize AX and Weave offer one on enterprise terms. Self-hosting means you run the database and its upgrades. For agents handling PHI, read HIPAA-compliant AI agents first.
- Built on LangChain or LangGraph. LangSmith fits most naturally, since one company makes both.
- Quality is the main worry and evals gate every release. Braintrust is built around experiments and scores; Langfuse and Phoenix also run evals and cost less to start. Build the test set first: how to build an eval set from support tickets.
- Already paying for Datadog. Its LLM Observability keeps agent traces beside the rest of your monitoring, and because it bills per LLM span, an agent's tool and retrieval steps do not add to the bill.
- Already on Weights & Biases for model training. Weave keeps experiments and production traces in one place.
- Want to keep options open. Instrument against the OpenTelemetry GenAI conventions and check, in each vendor's own docs, that it accepts them before you commit.
One honest note: a single low-volume agent can start with structured logs in your own database, keyed by a trace ID, and a weekly review of a sample. Buy a tool when you find yourself reading traces every week and the logs slow you down.
Questions to ask any vendor
Copy these into your evaluation sheet and get the answers in writing:
- What exactly is a billable unit, and how many will one of our runs produce?
- Where is our data stored, and can we choose the region?
- Can we redact or mask PII before data leaves our servers?
- What is the retention on the plan we would buy, and what does a longer one cost?
- Can we export raw traces through an API or in OpenTelemetry format?
- Can we attach eval scores and user feedback to a trace after the run?
- Does a multi-step run show tool calls and sub-agents nested in order?
- Who at the vendor can see our traces, and is that logged?
What Gigabit sets up in production agents
Observability is one of six layers in every agent we ship, alongside a model tier, orchestration, retrieval, evals and guardrails, as our production AI agent stack describes. In that stack every action is logged with full context, cost and latency are tracked per run, and drift is surfaced on dashboards. Permissions are bounded per step, low-confidence states escalate to a person, and rollback paths are rehearsed. Before an agent ships we build a scored test set from real cases, including the awkward ones, and every later change to prompt, model or retrieval runs against it. No green run, no deploy.
A Gigabit Agents build is a flat fee from $8,000 per agent, and that fee includes evals and observability rather than treating them as change orders. It covers testing against real cases, guardrails and escalation paths, deployment and 30 days of tuning. After that, Managed AI Operations from $3,000 a month handles monitoring, eval regressions, model updates and new intents.
