The unresolved spectrum from off-the-shelf to fully custom

The final Q&A returns to the central design question: how much of an agent harness should a team build itself? Harrison Chase does not give a final answer about where the industry will end up. Instead, he describes a spectrum and a practical starting rule:

1. Start with a general, off-the-shelf harness.
2. Add targeted customization when the task moves farther from the model's usual distribution or needs more control.
3. Use evals and observability to measure whether the extra complexity is worthwhile.

The important point is that “off-the-shelf” and “custom” are not two permanent categories. A team can move gradually between them.

The spectrum of harness choices

Position	What the team uses	When it can make sense
Off-the-shelf	A general-purpose harness with its existing loop and capabilities	The task is relatively basic or resembles work the model already handles well
Hooks and middleware	A general harness with additional insertion points, checks, or specialized behavior	The team needs more control without replacing the whole flow
Fully custom	An explicit, domain-specific cognitive architecture	The use case is far out of distribution or requires highly predictable, controlled steps

Here, a cognitive architecture means a deliberately designed sequence that guides the agent through a particular kind of work. It is more explicit than simply allowing a general harness to repeat model and tool calls. A fully custom architecture is therefore not just “a better prompt.” It can make the important stages of the work visible and controlled.

That does not mean a custom system must be written from nothing. Hooks and middleware can modify a general harness at selected points. A team might first add one check or wrapper, then add more as it learns where the general behavior is insufficient.

Why distribution shift changes the decision

The speaker connects customization to distribution. In this context, a model is more likely to perform reliably when the task resembles patterns it has encountered during training or use. A task farther from those patterns is more out of distribution.

This is a gradual change, not a formal boundary. The talk does not provide a numerical threshold that says when a task becomes out of distribution. Its heuristic is qualitative: the farther the use case moves from the model's distribution, the stronger the case for tuning the harness around that use case.

For a basic task, a general harness may provide the quickest path to useful results. It already supplies a working way to run the agent, use tools, and continue the task. Replacing that system immediately could add engineering work before the team knows which behavior actually needs improvement.

For a narrower or more unusual task, the general loop may leave too much behavior to the model. The team may then add gates, checks, or domain-specific stages. These additions make the harness compensate for a mismatch between the general workflow and the target work.

A teaching example

Imagine two teams using the same general harness. One needs an agent for a routine task. The other needs an agent for a specialized workflow where each stage must be checked before the next one begins.

The first team may reasonably keep the off-the-shelf harness. The second may add middleware that checks intermediate results. If those checks are still not enough, it may replace the repeating general flow with an explicit sequence of domain-specific stages.

This example illustrates the decision pattern, not a result reported for a particular benchmark. The source's recommendation is to begin generally and increase control as the use case demands it.

Control and predictability are separate reasons to customize

Distribution is not the only reason to move along the spectrum. The speaker also points to predictability and control. A general harness can be capable while still allowing behavior that a customer does not want to leave implicit.

Financial-services customers are given as an example of teams that may prefer a more controlled and explicit architecture, even when a general harness is available. In a setting like this, the question is not only:

Can the agent complete the task?

It is also:

Can the team make the important behavior explicit enough to inspect, constrain, and predict?

More explicit stages can make that behavior easier to reason about. They can also make the system more expensive to build and maintain. This is why the choice is a trade-off rather than a claim that fully custom is always better.

Models and harnesses can adapt to each other

The Q&A also discusses a smaller but important form of customization: model-specific behavior inside a portable harness.

The speaker says that OpenAI and Anthropic models use different methods for editing files. The implication is not that one universal file-editing implementation will preserve every model's strengths. A harness may instead select an implementation that fits the model it is using.

This creates a form of co-adaptation:

1. A model is developed with certain useful conventions or behaviors.
2. A harness is built to use those behaviors effectively.
3. The harness may need a different implementation when the team changes models.

The larger harness can still be shared across models, while smaller operations adapt to the active model. This is another reason the off-the-shelf-versus-custom choice is not all-or-nothing. Customization can be concentrated at the parts where model and harness interact most closely.

Will harnesses converge?

Harrison explicitly says that he does not know whether harnesses will converge. The Q&A presents possibilities rather than a settled forecast.

One possibility is convergence in coding. If model labs and harness builders discover that similar coding workflows work best, their approaches may become more alike. The discussion of different file-editing methods shows why convergence would not necessarily mean that every implementation is identical today.

Another possibility is divergence across domains. Coding may develop a broadly shared harness pattern, while a field such as biology may require different structures because the tasks, checks, and useful context differ. The biology example is hypothetical. The talk does not present it as a measured result or a prediction that has been established.

So the unresolved question is not simply whether one harness will “win.” It is whether general-purpose patterns will become sufficient across more domains, or whether domain-specific work will continue to justify distinct architectures.

Evals and observability decide what to keep

Because the space is moving quickly, intuition alone is a weak basis for choosing between harness designs. The closing point returns to two measurement tools:

• Evals compare behavior against a defined standard. They help a team ask whether one harness completes its target work better than another.
• Observability shows what happened during a run. It helps the team inspect the steps, context, and tool behavior that produced the result.

Together, they connect a design choice to evidence. An eval can show that a custom architecture changes performance. Observability can help explain whether the difference came from a control gate, a context-handling choice, a model-specific operation, or another part of the harness.

The speaker also mentions an unnamed benchmark participant who believed one file-editing method was strictly better. The talk supplies no benchmark name, metric, or supporting evidence for that claim, so it should be treated as an example of a disagreement that measurement could investigate—not as a conclusion about file editing.

Practical takeaway

The final answer is a decision process, not a prediction about the industry's endpoint:

1. Begin with the general harness when the task is basic and time to value matters.
2. Add hooks or middleware when a small number of checks or specialized behaviors are needed.
3. Move toward an explicit custom architecture when distribution shift, control, or predictability makes the general flow inadequate.
4. Keep model-specific pieces adaptable when different models use different conventions.
5. Measure every step with evals and observability before deciding that more customization is better.

The resulting mental model is a sliding scale. Off-the-shelf is a starting point, not a promise that customization will never be needed. Fully custom is an available endpoint, not a requirement to rewrite everything. The right position depends on the task, the model, the required control, and the evidence collected from real runs.

The talk leaves convergence open. What it does make clear is how to operate under that uncertainty: build enough harness to serve the use case, observe its behavior, and use private measurements to decide what to change next.