LangSmith Engineでトレースの整理と修正を自動化する

The question in this chapter

In the previous chapter, we saw the process of running an agent, collecting traces, finding problems, and improving the system. However, doing this process by hand every time is difficult and time-consuming. Here we look at how far the LangSmith Engine demo presented by the speaker tries to automate this work.

What kind of agent is Engine?

According to the speaker, Engine is not the original agent being evaluated. It is a separate agent that operates over traces and helps with improvement work. A trace is a record of information sent to the model, the model's responses, tool calls, and similar details from one run.

Engine itself is a coding agent with a prompt. It can access the LangSmith CLI (an interface used from the command line) and can also use sub-agents to inspect traces. It explores many execution records and finds similar failures or recurring problems.

Figure 1. A dark LangSmith screen shows the Engine entry point and a table of incoming traces.

This screen shows where Engine handles traces. Execution records for docs_agent appear in the table. However, the still image alone does not confirm that Engine is creating an issue at that moment.

From traces to an issue board

The demo proceeds as follows. First, traces arrive from agent runs. Engine examines them and organizes common problems. It then creates descriptions of issues on an issue board and links traces that support the causes.

Figure 2. The LangSmith Engine overview shows trace volume, categorized issues, severity, and issue trends.

This screen connects Engine's work with trace curation and issue monitoring. The idea is not simply to read failures one by one, but to group recurring patterns as issues. The screen shows the monitoring state, but not the moment when a background agent actually creates an issue board.

Attach evidence to the issue description

An issue description alone is not enough to decide immediately that a fix is needed. Engine links execution records related to the description as "Linked Traces." Builders can open those traces and check whether the same problem is really occurring. The issue claim and the evidence used to investigate it are therefore in the same place.

Figure 3. An issue description and a list of supporting execution records appear on the same screen.

This screen directly shows that Engine's diagnosis is linked to traces. The important point is not to trust the issue title. It is that builders can inspect the supporting traces and investigate the basis for the diagnosis.

Fixes are not limited to prompts

The fixes proposed by Engine can be made at several layers. Sometimes the prompt should change. In other cases, the context sent to the model should change. The instructions may also be changed. Engine can additionally propose adding or changing code in the harness.

Figure 4. The issue screen shows a change to the instructions or prompt and a proposed fix at the tool boundary.

The screen separates a proposal to change the instruction or prompt text from a proposal for a helper function that cleans content at the tool boundary. This distinction matters. Even when failures look similar, the needed change may be in the model instruction, the input context, or the harness processing.

Figure 5. The LangSmith Engine screen shows a multi-file code diff that cleans up MDX-formatted documentation output.

This diff is one example from the demo of Engine producing a code fix proposal. The individual code lines are not reliably readable in the image. Therefore, we should not infer the specific implementation or conclude that the fix was automatically applied to production.

The improvement loop stays the same even with automation

Engine may look complex, but it does not replace the process from the previous chapter. The basic flow is still: run the agent, collect traces, find recurring patterns, create a proposed fix, and use it in the next experiment. Engine tries to automate trace selection, experiment creation, and presentation of fix proposals in this process.

Example: When the same output problem happens repeatedly

For example, suppose the format of documentation output is broken in several traces. Engine gathers related traces, describes the common problem, and links the executions that support it. It then shows a proposal to change the instructions or code that cleans the content returned by a tool. The builder checks the proposal and the traces, then experiments with the fix if needed. This is an explanatory example. The speaker does not say that every proposal is correct or that proposals are applied automatically.

What the demo allows us to conclude

The speaker says that the demo might fail, and then reports that it worked. However, the demo does not explain in detail how an issue was created on each screen or whether the proposed fixes were actually applied. What we can confirm is that it is trying a workflow that moves from trace curation to evidence-backed issues and then to fixes at several layers. This is not proof that the proposals are always correct.

Summary

LangSmith Engine is a coding agent that operates over traces. Using the CLI and sub-agents, it examines traces, organizes recurring problems, attaches related traces as evidence, and creates proposed fixes for prompts, context, instructions, or harness code. It tries to automate an improvement process that is difficult and time-consuming. However, the builder still has to make the final decision and verify the result.