Observability exposes the path to an agent failure

An agent can produce a bad result for at least two very different reasons:

1. The model is not capable enough for the task.
2. The model did not receive the context it needed.

Harrison Chase says that, in his experience, the second problem is more common. This is an observation from the talk, not a quantified result. A failed outcome alone cannot tell you which explanation is correct. You need to inspect the run.

What observability makes visible

Observability means being able to see what happened inside an agent run. In this chapter, that includes:

• what entered the model's context window;
• how the context accumulated over time;
• which agent steps ran;
• which tools ran and what they returned; and
• details of the model calls involved.

This connects directly to the harness described earlier. The harness is responsible for supplying the right context at the right time, calling tools, and returning their observations to the loop. Observability lets a builder check whether that orchestration actually happened as intended.

For example, suppose an agent gives an incomplete answer after using a search tool. There are several possible explanations:

• the model could not reason from the available evidence;
• the search tool could have returned poor results;
• the harness could have omitted useful results from a later model call; or
• earlier steps could have filled the context with distracting or incomplete information.

These possibilities require different fixes. Improving the model may help the first case. Changing retrieval, context management, or tool handling may help the others.

A trajectory is useful, but it is not the whole trace

A trajectory is a readable sequence of an agent's messages or visible interaction steps. It gives a quick story of the run: the request, the model's actions, tool calls, observations, and eventual answer.

A trace goes deeper. It allows the builder to drill into the calls and inspect more of the details behind that visible story. The transcript does not specify the exact fields or nesting used by the full trace interface, so the important distinction is about depth of inspection rather than a particular UI schema.

The useful design is therefore both:

• a collapsed view for quickly scanning many runs; and
• an expanded trace for investigating one suspicious run.

An attractive transcript may show that a tool was called. It may not show enough to explain what the tool returned, how that result entered later context, or what happened inside the surrounding model calls. Treating the transcript as complete observability can hide the actual failure path.

Click to enlarge

At 13:56, the visible interface presents an agent run as a thread. Several tool calls appear in a compact summary, while additional run details are available in the surrounding interface. The visible “2 more” wording supports the idea that calls can remain collapsed for quick scanning; the frame does not show how the calls are expanded or hidden.

Reconstructing the failure path

When debugging, read the trace as a sequence of context changes:

1. Start with the request. What did the agent need to accomplish?
2. Inspect the first model step. What context did the model receive, and what did it decide to do?
3. Follow each tool call. What tool ran, and what observation came back?
4. Check the next model step. Was the observation included in the context supplied to the model?
5. Track accumulation. Did useful information remain available, or was it lost, buried, or crowded out as the run continued?
6. Compare the result with the evidence. Does the final failure point to a missing context element, a harness decision, or a model capability limit?

This process turns “the agent failed” into a more testable diagnosis. The trace may reveal that the model had the necessary information but used it poorly. It may instead show that the information never reached the relevant model call. Those cases should not be treated as the same failure.

How observability works with evals

An eval measures the outcome against a standard. It can tell a team that one run or one version performed poorly. Observability shows the path that produced that outcome.

Together, they support a practical feedback loop:

1. An eval identifies a bad or weak result.
2. The trace exposes the steps, tools, and accumulated context behind it.
3. The builder forms a hypothesis about the cause.
4. The team changes the model, the context, or the harness.
5. The eval checks whether the change improved the task.

So observability is not a replacement for evaluation. A trace does not, by itself, define what “good” means. Conversely, an evaluation score does not usually explain which part of the system should change next. The score identifies the outcome; the trace helps identify the next experiment.

The central lesson is simple: do not diagnose every bad agent result as a bad model. First make the run inspectable. When you can follow context accumulation, tool results, and model steps, you can decide whether to improve the model, improve the information given to it, or improve the harness that coordinates the work.