AI 101 — What is LLM observability?

LLM observability is the practice of recording every step an AI system takes — each model call, tool use, search, token count and dollar — as structured data you can search later, so that a bad answer becomes a question with an answer instead of a mystery.
Ordinary software tells you when it breaks: the page fails to load, the request returns a 500, the alert fires. An AI feature does not do that. It returns a perfectly formatted 200 with a confident, wrong sentence, after three unnecessary tool calls and four times the tokens you budgeted. Nothing crashed. Nothing is red on the dashboard. The system did the wrong thing politely, which is the hardest failure to notice and the easiest to ship for a month.
That gap is the whole reason this discipline exists.
Why it matters right now
Two things happened this year that turned LLM observability from a niche side-tool into standard plumbing.
The first is that model providers started emitting it themselves. On 14 October 2026, Anthropic put OpenTelemetry trace export into public beta, covering the Messages API, Claude Managed Agents and its SDKs, according to Anthropic's announcement. Instead of building a bespoke logging layer, a team points its existing observability tool at a standard endpoint and gets one record per model call and per tool use, carrying latency, token counts and identifiers.
The second is money moving in. ClickHouse acquired Langfuse, the open-source tracing platform, on 16 January 2026 as part of a $400 million funding round that valued ClickHouse at $15 billion, the company said. Langfuse's core stays MIT-licensed and self-hostable, and it ended 2025 with more than 20,000 GitHub stars and 26 million SDK installs a month — a lot of teams, quietly, already watching their models this way.
The timing is not a coincidence. Agents are now trusted with real work: our own reporting covered a hundred agents run through a live audit, one of which needed a human. Every one of those setups has the same question after an incident — what exactly did it do, in what order, and what did each step cost?
The mental model: a tree of steps
Observability data comes in two shapes, and everything else is built on them.
A span is one step: a model request, a tool call, a database lookup, a retrieval from a document index. It has a start time, an end time, and attributes — which model, how many input and output tokens, which tool, whether it errored.
A trace is the full record of one request, with every span nested under the step that caused it. The user's question is the root. Under it sits the first model call; under that, the tool the model decided to run; under that, the second model call that read the tool's result. A session or thread ID then stitches several traces together, so you can follow a multi-turn conversation rather than one isolated exchange.
Three questions follow directly from that structure, and they are the ones teams actually ask. Why was this slow — which span ate the seconds? Why did this cost so much — which step inflated the token count? Why was the answer wrong — what did the model actually see at the step where it went sideways?
The analogy: parcel tracking
When you ship a package, you do not get one status. You get a history: picked up, sorted, customs, out for delivery, delivered — each with a time and a place. That history is the trace. Each scan is a span.
Now imagine the parcel is late and all you have is "delivery delayed". You know something went wrong. You cannot tell whether it sat in a sorting centre for two days or missed a truck by ten minutes, so you cannot fix it, and the next parcel will be late the same way.
An AI feature without observability is that "delivery delayed" screen. With it, you can say the agent stalled because a search tool returned empty results and the model retried three times — which is a bug you can close.
Common misconceptions
"It's just monitoring." Uptime monitoring answers whether the service is up. Observability answers what it did on request number 4,812 at 14:02. A healthy service that gives wrong answers passes every uptime check ever written.
"Evals already cover this." Evals answer a different question: how good is this on a fixed set of cases? They run before you ship. Observability watches what real traffic does after. The two connect — good teams turn their worst production traces into new eval cases — but neither replaces the other. If you want the grading side, AI 101 — What is LLM-as-a-judge? covers how models get scored when no test can decide correctness.
"It's whatever tool the vendor sells." The useful part is the schema, not the dashboard. The OpenTelemetry GenAI semantic conventions define the shared vocabulary — model name, token usage, tool name, operation type — so the same data can feed a self-hosted tool or a managed backend without re-instrumenting. Those conventions are still in Development status, per the specification repository, which means attribute names can change without a major version bump. Plan for that: keep the mapping in one place and expect to update it.
"We log the prompts, so we're covered." A prompt log is a string in a file. It cannot tell you which model version served it, what the step cost, or which parent step caused it — and prompts carry user data, so dumping all of them is a privacy liability as much as a shortcut. Structure it, sample it, redact what you don't need.
"More data is better." A trace with every span and no filtering is expensive to store and slow to read. Capture the steps you act on — model calls, tool calls, errors, spend — plus the harness around the model that decides what it does next.
Where to learn more
The OpenTelemetry GenAI conventions are the primary document — start with the spans and metrics pages, which define what a well-formed trace contains. Langfuse's data model docs explain how traces, observations and sessions fit together in a system you can self-host, and Anthropic's tracing documentation shows what the provider side emits today.
Related reading: AI 101 — What is an agent harness? · AI 101 — What is LLM-as-a-judge? · AI 101 — What is an AI agent?
If your agent gave a wrong answer last week, could you say which step caused it? Write to us via our contact page.







