Engineを自社利用し、ハーネスの動作をベンチマークする

What the Q&A reveals about Engine's role

In the Q&A, Harrison Chase explained that LangSmith Engine is also run as an agent. Engine is not the run of the original agent being evaluated. It is the agent on the supporting side: it examines traces, finds problems, and supports improvement work.

The team uses that Engine on its own work and on traces related to Engine. Using what you are building in your own work is called dogfooding. The phrase “using Engine on Engine” does not mean a new model architecture. It means using the Engine agent to investigate Engine development and related problems.

If we organize the speaker's explanation as an operational flow, it looks like this.

1. Run Engine as an agent and investigate Engine-related issues or traces.
2. Send reports about Engine's work to Slack.
3. The team uses the results in its own development and checks Engine by using it in practice.

Issue Bench as a Harbor-formatted comparison benchmark

The speaker also explained that the team uses a benchmark called Issue Bench to compare Engine-related problems continuously. It is not Harbor itself. It is a team benchmark built in Harbor's format. In other words, Harbor is the framework that runs evaluations, while Issue Bench is the concrete set of problems prepared for that framework.

With a common benchmark, different models and harnesses can be run on the same Engine-related issues and compared. This shows not only a single ranking of “which is best,” but also the strengths and weaknesses of each one. One harness might be good at examining long traces. Another might handle a different kind of issue more consistently. This is an explanatory example; it does not report actual rankings or scores.

The value of this comparison is that it lets the team consider causes under the same conditions. If only the model changes while the same problems are used, the model difference is easier to examine. If the model stays the same and only the harness changes, differences in tool use or trace handling are easier to examine. A benchmark therefore becomes a common measuring stick for harness behavior.

An observation from Codex's trace analysis

While comparing models and harnesses, the team said it noticed one characteristic in Codex's behavior. Codex aggressively wrote small analysis scripts against traces. Scripts make it possible to search directly for needed patterns instead of reading a large number of traces only by hand.

However, this was a behavior observed in that comparison or sprint. It does not show that Codex always uses this method or is always better than other methods. The speaker explained that the team used the useful pattern it observed for the next improvement.

More specifically, the team brought the learning about writing small scripts against traces into the core Engine harness. It turned a method found in the experiment into something that could be used as a shared harness capability. This means the method does not end as a one-time experiment; it becomes a candidate for the same kind of analysis in later Engine work.

The transcript calls this transfer by a word that sounds like “codexification.” However, the original speech is fragmented, so the term's exact spelling and formal definition are uncertain. What is clear is the action: bringing the learning from the Codex experiment back into the core Engine harness.

Connection to evals and observability

This Q&A makes concrete how evals and observability help improve a harness. Issue Bench measures model and harness behavior under the same conditions. Trace analysis provides material for examining which methods helped solve problems. A method that works well can then become a candidate for the core harness. In this way, a benchmark does more than compare results: it also provides evidence for deciding how to change the harness next.