Dogfooding Engine and benchmarking harness behavior

This chapter comes from the question-and-answer section. It gives a practical example of the talk's larger idea: a team can improve an agent by measuring not only the model, but also the harness that controls the agent's work.

Engine is used on its own work

Harrison says that Engine itself runs as an agent. This does not mean that “Engine on Engine” is a new model architecture. It means that the same kind of agent system discussed earlier is used to examine Engine-related work and traces.

This is a form of dogfooding: a team uses its own system while building and improving that system. In this case, the system under use is an agent that works over traces and improvement tasks. The team also has Engine report about its work to Slack. That creates an operational path from an automated investigation to a place where the team can inspect or discuss the result.

The important distinction is:

• The agent being evaluated: the original system whose behavior produces traces.
• Engine: another agent that examines work and traces to help find improvements.
• Dogfooding: using Engine in the team's own improvement process rather than treating it as only a demo.

The source describes this internal use, but it does not claim that every Engine suggestion is correct or that every change is deployed automatically.

Issue Bench makes Engine behavior comparable

The team dogfoods Engine through an Issue Bench. Harrison describes Issue Bench as a benchmark in Harbor format for continuously comparing models and harnesses on issues related to Engine.

Here, benchmark means a repeatable set of work used to compare alternatives. The value is not just a single winner. A common set of issues lets the team ask more useful questions:

Comparison	Question it helps expose
Model versus model	Which model handles these issues more effectively?
Harness versus harness	Which orchestration strategy helps or harms the work?
Model and harness together	Which combination produces the strongest behavior for this benchmark?

For example, if two models use the same harness and the same Issue Bench tasks, a difference in their results is evidence about model behavior in that setup. If the model is held constant while the harness changes, the comparison instead helps expose the harness's effect. In practice, these comparisons reveal strengths and weaknesses rather than proving that one system is universally best.

The plan does not provide Issue Bench scores, a named metric, or a claim that the benchmark settles all future choices. Its role in the Q&A is to make repeated comparison possible.

What the comparison revealed about Codex

Harrison reports one observed difference from comparing models and harnesses. Codex wrote small scripts against the traces, and it did so aggressively. A trace-analysis script can turn repeated inspection into a concrete computation or filter. That may help an agent investigate many traces without relying only on a long sequence of manual reading steps.

This is an observation from the team's reported comparison. It should not be read as a universal claim that Codex always analyzes traces this way, or that this strategy is best outside the observed sprint. The source also does not give benchmark scores that would quantify the advantage.

Bringing a successful pattern back into the shared harness

The team then brought that useful behavior back into the core Engine harness. The transcript uses a fragmented term that sounds like “codexification” for this transfer step; the exact term is uncertain. The action itself is clear: learning from an experiment with one model and incorporating the useful pattern into the shared system.

The loop looks like this:

Engine runs on Engine-related work
        ↓
Issue Bench compares models and harnesses
        ↓
The team observes a useful behavior
        ↓
The behavior is brought into the core Engine harness
        ↓
The updated harness can be compared again

This is more than choosing a model once. The benchmark makes harness behavior visible, while dogfooding tests the system in the team's own workflow. A successful experiment can therefore become a candidate for the shared harness, where it can be measured again rather than accepted only because it looked promising in one run.

Connection to the lesson's thesis

The example reinforces the three-way ownership model from the talk:

1. Models can differ in how they approach the same work.
2. Context and traces provide the material an agent or an evaluator can inspect.
3. The harness determines how the agent operates and can absorb useful patterns discovered during experiments.

Issue Bench supplies the comparison loop. Engine supplies an agent that can help investigate the evidence. Dogfooding supplies a real internal use case. Together, they turn observability and evaluation into a feedback process for improving the harness, while leaving open which model or strategy will be best in every setting.