Owning intelligence: model, context, and harness

An agent is more than a language model answering a prompt. In this part of the talk, Harrison Chase explains agent intelligence as the result of three connected parts:

1. The model produces reasoning and language.
2. The context gives the model the information it can use for the current task.
3. The harness organizes the work around the model. It decides what context to provide, when to call the model, how to use tools and external systems, and how to return the resulting observations to the agent.

The key idea is that a team can own intelligence at all three layers. In this usage, “owning” means having control over the model choice, the context supplied to the agent, and the orchestration that makes the agent act. It does not mean that the team must train a model from scratch.

The three parts of an agent

The model is the part that generates a response or chooses a next action. The context is the material made available to it. The harness is the surrounding control system that connects the two and connects the agent to the outside world.

The speaker says the model layer is not the main topic of this section and refers to an earlier Fireworks presentation. The focus here is the relationship between the three parts, especially the harness.

Click to enlarge

At about 1:16, the diagram presents an agent as a harness containing the model and context. The diagram supports the three-part decomposition, but it does not itself show arrows or the orchestration process.

These parts should not be collapsed into one another:

• A harness is not the model. Changing the harness changes how the model is used; it does not, by itself, change the model's learned capabilities.
• A harness is not merely a list of tools. Tools are capabilities the harness can make available, but the harness also manages context, model calls, actions, and observations.
• Context is not only long-term memory. Memory is one possible source of context, alongside semantic knowledge and previous conversations.

This separation is useful because a poor result can have different causes. The model may need to be changed, the agent may need better information, or the orchestration may need to handle the task differently. “The agent” is the whole system, not just the model at its center.

What the harness does in the loop

The harness acts as an orchestrator. A simplified sequence is:

1. It receives a task or request.
2. It selects and assembles relevant context.
3. It places that context in the model's context window at the appropriate point in the run.
4. It invokes the model.
5. It interprets the model's response and can act on it, including interacting with an external system.
6. It can take the external system's output and feed that observation back into the agent's context.
7. The model can then continue from the new observation.

The important timing is the middle of this sequence. The harness does not simply attach every piece of information at the beginning and wait for an answer. It manages what the model sees at the right time for the task.

Click to enlarge

At about 2:19, the slide frames the harness as the mechanism that gives the model the right context at the right time for a task. The visual is conceptual; it does not visibly distinguish fixed from dynamic context.

Fixed and dynamic context

One useful distinction is between fixed context and dynamic context:

• Fixed context is information that remains stable across a run or across many runs.
• Dynamic context is information selected or updated as the task progresses.

The speaker names memory, semantic knowledge, and previous conversations as examples of context. Some of that information may be stable, while other parts change with the user, the task, or new observations. The harness is responsible for managing this distinction and deciding what belongs in the model's active context window now.

For example, imagine an agent helping with a task that depends on organizational knowledge. A relevant previous conversation or a piece of semantic knowledge may be brought into the window for one step. After the agent interacts with an external system, the returned result may become the next piece of dynamic context. The example here is a teaching illustration of the timing; the source's central claim is that the harness assembles and supplies context as the task requires.

Model choice is part of ownership

Owning the model layer does not necessarily mean building a new model. It can also mean retaining the ability to choose among models and to switch models when appropriate.

The speaker presents model switching as valuable for two reasons:

• It reduces lock-in to one model provider or model family.
• It lets a team use the best available model for a particular need.

This does not imply that switching is frictionless. Different models can have different strengths, interfaces, or expectations about how they should be used. The point is that a harness designed around interchangeable model choices gives the team more control than a system tightly bound to one model.

The three ownership areas therefore interact. A team might keep the same broad task while changing the model. It might keep the model while improving the context retrieved for it. Or it might keep both while changing the harness so that information and external actions are handled more effectively.

Why a harness is needed

As an agent runs for longer or works with external systems, the surrounding orchestration becomes increasingly important. The harness must connect the agent to data and manage context over long runs, among other needs named on the following slide.

Click to enlarge

At about 2:38, the slide lists six reasons an agent needs a harness. It visibly includes connecting an agent to data and managing context over long runs. It does not show an external-system response or a complete agent-loop diagram.

This explains why “just give the model some tools” is an incomplete mental model. The tool is only one point of contact. The harness also decides when the tool is relevant, how its result enters the context, and what the agent should do next.

A practical mental model

When diagnosing or designing an agent, ask three separate questions:

Layer	Question to ask
Model	Which model should generate the response or next action?
Context	What information should be available to that model at this point?
Harness	How should the system assemble context, invoke the model, perform actions, and return observations?

These questions prevent a common mistake: treating every failure as a model failure. If the model never receives the relevant memory, semantic knowledge, previous conversation, or external-system observation, replacing the model may not address the real problem. The harness may need to supply better context or manage the interaction differently.

The broader lesson is a control surface, not a single component. General-purpose harnesses can provide a starting point, but later chapters examine how teams customize orchestration, choose between general and domain-specific flows, and measure whether a change helped. Evals and observability matter because they let a team see which part of this three-part system should change next.