LangSmith Engine automates trace curation and fixes

The previous chapter described a feedback loop for improving an agent: run it, collect traces, find patterns, experiment, and change the model, context, or harness. This chapter shows the idea as a product workflow. LangSmith Engine is presented as an agent that works on top of the traces produced by other agent runs. Its job is not to perform the original user task. Its job is to inspect evidence about those runs and help find and fix recurring problems.

The basic mental model

Think of two agents with different jobs:

1. The task agent performs work for a user. Its run creates a trace: a record of the steps, model calls, tool calls, and resulting behavior.
2. Engine examines many of those traces. It looks for repeated failure patterns, organizes them as issues, connects each issue to supporting runs, and proposes an intervention.

This distinction matters. Engine is not the score assigned to a task, and it is not the original agent being evaluated. It is an improvement agent that uses run data as its working material.

The speaker describes the underlying Engine as a coding agent with three important resources:

• a prompt that tells it what improvement work to do;
• access to the LangSmith CLI, so it can explore trace data; and
• sub-agents that help explore and filter that data.

The exact internal implementation is not shown in the supplied material. The useful claim is about the workflow: Engine turns a large collection of traces into organized problems and candidate changes that a builder can inspect.

From incoming traces to an issue board

The narrated demo follows this progression:

incoming traces
      ↓
Engine explores and filters the traces
      ↓
recurring issue is identified
      ↓
issue is documented with supporting traces
      ↓
candidate fix is proposed

The first step is collecting runs. A trace is valuable because it contains more than the final answer. It can show the path that led to the answer, including the context supplied to the model and the tools used along the way. Engine can therefore search for a pattern that appears across multiple runs rather than reacting to one isolated failure.

Click to enlarge

LangSmith's dark tracing interface shows the Engine entry point and a table of incoming traces.

The frame shows the Engine entry point and a populated trace table with incoming docs_agent entries. It supports the demo's starting point: Engine has trace data to examine. The frame does not, by itself, show the agent actively analyzing those rows.

The next step is to turn trace patterns into issues. An issue is a written description of a recurring problem. The speaker presents an Engine overview with trace and issue monitoring, including categorized issues, severity, and issue trends. These categories give the improvement process a place to organize what Engine finds.

Click to enlarge

A LangSmith Engine overview dashboard shows issue and trace monitoring for chat-langchain-external.

This dashboard connects the two kinds of information: the underlying traces and the issues derived from them. It is a monitoring view, not proof that every issue was automatically created or that every classification is correct.

Why supporting traces matter

An issue description is a hypothesis about agent behavior. For example, it might say that a particular kind of task repeatedly fails in a similar way. The description becomes much more useful when it links to the runs that demonstrate the pattern.

The links provide an evidence path:

issue claim → supporting traces → inspect the original runs → decide what to change

This is the practical value of trace curation. Engine does not merely produce a vague statement such as “the agent needs improvement.” It associates the statement with runs that a builder can open and inspect. That lets the builder check whether the proposed pattern is real and understand the surrounding context before changing the system.

Click to enlarge

A LangSmith issue page pairs a written diagnosis with a Linked Traces list of supporting runs.

The visible issue view shows a written diagnosis alongside a Linked Traces list. This directly supports the evidence-backed part of the workflow. The source does not specify a complete labeling procedure for deciding which traces belong to an issue, so the important point is the connection between the diagnosis and inspectable runs.

Fixes can target different layers

The proposed fix is not necessarily a new prompt. The speaker describes several possible intervention points:

Proposed change	What it changes
Prompt change	The instructions given to the model or agent are revised.
Context change	The information assembled for the model is changed.
Instruction change	A rule or guidance used by the task agent is adjusted.
Harness-code change	The surrounding program is changed, for example at a tool boundary.

These are different ways to act on the three-part system from the lesson. A prompt or instruction affects how the agent is guided. A context change affects what information reaches the model. Harness code changes the orchestration around model and tool calls. Treating them as separate options helps avoid reducing every debugging task to prompt editing.

In the demo, one issue view shows proposed changes at two layers: updated instruction or prompt content, and a sanitizing helper at the tool boundary. The tool-boundary proposal is a harness intervention. It changes how information is handled around a tool call rather than only changing what the model is told.

Click to enlarge

A LangSmith issue view shows proposed prompt and tool-boundary fixes for documentation-agent failures.

The neighboring view also shows a separate proposed prompt rule concerning Python naming conventions. The frame supports the distinction between a prompt-level proposal and a tool-boundary proposal. It does not establish that either proposal was applied or that it solved all of the associated failures.

The demo then shows Engine adding code. This is significant because it demonstrates a broader meaning of “improvement.” If the trace pattern points to a limitation in the harness, the candidate repair can be code rather than a wording change.

Click to enlarge

A LangSmith Engine issue view displays a green code diff for sanitizing raw MDX documentation output.

The visible frame contains a multi-file code diff in the Engine area of LangSmith. It supports the speaker's reference to Engine adding code. The code is too small in the supplied frame to transcribe reliably, so the grounded conclusion is limited to this: Engine can present a concrete harness-code change as part of an issue's proposed fix.

How this completes the flywheel

The demo automates the labor between raw run data and the next engineering experiment:

1. Run the task agent. Its work produces traces.
2. Explore the traces. Engine uses the LangSmith CLI and sub-agents to search and filter them.
3. Curate a pattern. Repeated behavior is organized into an issue.
4. Attach evidence. Supporting traces are linked to the issue so the diagnosis can be reviewed.
5. Propose a change. The change may affect prompts, context, instructions, or harness code.
6. Experiment. The proposed change can become the next thing to test in the evaluation loop.

The speaker presents this as an attempt to automate a process that is otherwise difficult and time-consuming. The underlying loop remains simple—run, inspect patterns, fix, and test again—but Engine supplies assistance at the inspection, curation, and fix-generation stages.

That last qualification is important. The demo shows an improvement workflow and proposed changes. It does not establish that every suggested fix is correct, that every issue is found, or that changes are automatically deployed. A builder still needs to inspect the linked traces, judge the diagnosis, and evaluate the change.

The larger lesson

Observability tells the team what happened inside an agent run. Evals tell the team whether the result meets its standard. Engine connects those capabilities to engineering action: it uses observable traces to form issues and suggests changes that can be measured in later experiments.

So the value is not simply “an agent that writes fixes.” The more general pattern is an evidence-backed improvement system:

traces make behavior inspectable → issues make patterns discussable → proposed changes make experiments actionable

This keeps the improvement loop grounded in actual runs. It also preserves the central choice from the lesson: when an agent needs to improve, the right intervention may be in the model, the context, or the harness.