When distribution shift justifies a custom harness

The choice between an off-the-shelf harness and a custom one is not all or nothing. A useful question is:

How closely does the target task match the kinds of work the model already handles well?

In this section, Harrison Chase presents that match as distribution. An off-the-shelf harness is most useful when the task is close to the model's training distribution. As the task moves farther away, the case for customizing the harness becomes stronger.

This is a qualitative scale, not a precise threshold. The source does not define a score that separates “in distribution” from “out of distribution.”

Distribution is about the task—and its parts

In distribution means that the model is familiar with the kind of operation or situation it is being asked to handle. Out of distribution means that the larger task is less familiar or less well covered by what the model learned.

The important detail is that a single agent task can contain both kinds of work. The overall domain may be unfamiliar even when one small operation inside it is familiar.

The legal-AI example

Chase uses legal AI to make this distinction. A legal-AI application may be out of distribution for a general-purpose agent harness because legal work requires domain-specific orchestration. The harness may need to organize the work in ways that are particular to that application.

But file editing can still be in distribution for the underlying models. The model may already be effective at the concrete operation of changing files. Therefore, the right response is not necessarily to replace every part of the general harness.

The nested structure looks like this:

Overall task: legal AI                         -> may be out of distribution
    Subtask: edit a file                       -> may be in distribution

This distinction prevents a common mistake: assuming that an unfamiliar domain makes every operation within that domain unfamiliar too.

Customize the outer flow, preserve the strong inner operation

The causal argument is simple:

1. The domain-level task needs specialized orchestration.
2. The model already has a useful way to perform a smaller operation, such as editing a file.
3. A custom outer harness can manage the domain workflow while preserving that model-specific operation.

This gives the builder control where the application needs it without discarding behavior that already fits the model.

For example, a legal-AI harness could use its own domain-specific flow around document work, while delegating file changes to an edit-file tool suited to the active model. The custom part is the orchestration around the operation, not necessarily the operation itself.

The speaker also says that OpenAI and Claude models edit files differently. A single generic implementation may therefore fail to preserve the strengths of both. The harness should keep the model-specific behavior close to the model that uses it.

Model profiles route model-specific tools

Chase describes model profiles as a way to select among implementations according to the model in use. In simplified form:

Active model
    ├── Model A -> edit-file implementation suited to Model A
    └── Model B -> edit-file implementation suited to Model B

The profile acts as a routing layer. The larger harness can remain portable across models, while a smaller component uses the implementation that matches the active model's file-editing behavior.

The speaker makes a broad claim connecting file editing with reinforcement learning. That claim is part of his explanation in the talk, not an independently verified account of each model's training history. The practical point for the harness decision is narrower: different models may have different conventions or strengths, so model-specific tool behavior can matter.

Distribution shift increases the value of customization

The decision can be understood as a progression:

Situation	Likely approach
The task is close to familiar model behavior	Start with an off-the-shelf harness.
The domain is becoming more specialized	Add targeted harness behavior around the general loop.
The domain requires orchestration that the general flow does not express well	Build a more custom outer harness, while retaining useful model-specific operations.

This does not mean that an off-the-shelf harness has no value in a specialized product. It is still a starting point, and some parts of the task may remain familiar to the model. Nor does a custom harness require rewriting the entire agent. Customization can be limited to the layer where the distribution shift appears.

Teaching context: think in layers

One practical way to apply this idea is to inspect the task at several layers:

• Domain layer: What kind of work is the agent coordinating?
• Workflow layer: Which steps, decisions, or checks are specific to that domain?
• Operation layer: Which concrete actions can the model already perform reliably?

If the domain and workflow are unusual but an operation is familiar, customize the workflow and preserve the operation. This layered view leads to a more precise design than simply labeling the whole application “in distribution” or “out of distribution.”

The main heuristic

The speaker's recommendation is to begin with a general or off-the-shelf harness, then tune the harness as the use case moves farther out of distribution. The farther the shift, the more likely domain-specific orchestration is needed.

That recommendation balances two goals:

• Fast initial progress: a general harness provides the easiest starting point.
• Domain-specific performance: a custom outer flow can add the control needed for a narrower use case.

The result is not a binary choice between “use the default” and “build everything yourself.” It is a layered decision: keep the general machinery where it fits, and customize the parts that must understand the application's domain.

Source grounding: This chapter explains the speaker's discussion from approximately 07:36 to 09:08. The distribution boundary is presented qualitatively, and the claims about model training and product behavior are attributed to the talk rather than independently verified.