Harbor turns agent tasks into comparable benchmarks

Harbor is presented as an open-source evaluation runner. It is not the agent, and it is not the model. Its job is to run agents on repeatable tasks, collect the results, and make those results comparable.

This matters because an agent harness is a system, not only a model call. Two agents can use the same model but behave differently because their harnesses provide different context, tools, or control logic. A useful benchmark must therefore let a team compare the complete setups on the same work.

Source note: In the talk, Harrison Chase describes Harbor as an emerging industry standard and connects it with references such as Terminal Bench 2 and Frontier Bench. The lesson plan does not provide independent evidence for that industry-status claim, so it is treated here as the speaker's characterization.

These names refer to related but different things in this section: Harbor is the evaluation runner, Terminal Bench 2 and Frontier Bench are benchmark references, and LangSmith is the platform view used later to inspect experiment and operational results.

The basic benchmark: many agents, many task environments

The basic Harbor arrangement is:

1. Start with an agent whose behavior you want to evaluate.
2. Give Harbor a dataset containing multiple tasks.
3. Run the agent on each task in its own sandbox.
4. Use the task's solution and test definitions to check the result.
5. Record a score and compare the runs.

A sandbox is a separate task environment in which the agent can do its work. The important point in the talk is that each task can have its own environment, and many tasks can be run in parallel. This makes the benchmark suitable for stateful agents: the agent can interact with files or other task state without all tasks sharing one changing workspace.

The following slide shows the comparison view that motivates this setup. It is a Frontier-Bench leaderboard with entries organized by model, reasoning effort, accuracy, release date, and organization.

Click to enlarge

At 10:54 — A Frontier-Bench leaderboard compares agent entries by model, reasoning effort, accuracy, release date, and organization. The frame supports the comparison idea, but it does not show the specific coding-task details mentioned in the surrounding discussion.

The lesson from this kind of table is not simply “which model wins?” The same work can be used to compare:

• different models;
• different agent harnesses around those models; and
• different amounts or modes of reasoning effort.

Because the task set is held constant, the comparison is closer to an apples-to-apples test. The benchmark does not remove every possible source of variation, but it gives the team a shared work set instead of asking each system to demonstrate itself on unrelated examples.

What one Harbor task contains

The speaker then breaks the benchmark into task-level pieces. A task needs more than an instruction. It needs a place to run, a definition of the desired work, and a way to decide whether the run succeeded.

The Harbor Evals diagram makes the outer structure visible: a dataset contains several tasks, and each task is paired with an agent running in a sandbox.

Click to enlarge

At 11:18 — The Harbor Evals diagram shows agents in sandboxes working against tasks in a dataset. It directly supports the idea that the tasks are evaluated as separate environment-and-agent runs.

For each task, the described ingredients are:

Part	Role in the benchmark
Agent	The system being run and evaluated.
Dataset	The collection of tasks used for repeated evaluation.
Environment or sandbox	The task-specific place where the agent performs its work.
Instruction prompt	The directions given to the agent for that task.
Golden solution	A reference solution used for a sanity check.
Test or verifier	The evaluation procedure that checks the agent's result and determines the score.

The word golden here means “reference” rather than “the only acceptable implementation.” In the talk's description, the golden solution is used to check that the task and its evaluation setup make sense. The test is the separate mechanism that evaluates the agent's run.

The environment is part of the task definition

The task layout shown in the presentation includes a container environment, a Markdown instruction, and shell-based evaluation. It also shows these pieces alongside an example repository tree.

Click to enlarge

At 11:35 — The Harbor Tasks slide shows files for a container environment, a Markdown instruction, and shell-based evaluation, alongside an example task repository tree.

One way the task specifies its environment is with a Dockerfile. A Dockerfile is a text file that describes how to prepare the environment in which the task runs. In this benchmark, that lets a task author describe the tools and starting state needed for the task rather than relying on an unspecified machine setup.

This is a teaching explanation of the term; the source specifically describes Dockerfile-based task environments but does not make a broader claim about Docker's security properties.

The task can also include an evaluation script and a separate solution directory. The next view shows the evaluation script together with a solve.sh file in that solution area.

Click to enlarge

At 11:58 — The Harbor Tasks slide lists an environment, an instruction, and an evaluation script beside a task repository tree. It also shows a separate solution directory containing solve.sh. The frame supports the distinction between solution material and evaluation material, but it does not itself label solve.sh as a golden solution or show how the verifier behaves.

The test decides how success is scored

The test or verifier is where the benchmark designer defines what counts as a successful run. The talk lists several possible implementations:

• ordinary code;
• unit tests;
• another language model acting as a judge; or
• another agent acting as a judge.

These options do not mean that every benchmark uses all of them. They are different ways to implement the test. The designer chooses the scoring method that fits the task.

This distinction prevents a common mistake:

• The golden solution is a reference used for a sanity check.
• The test is the procedure that checks the evaluated agent's work.

For example, a task might give an agent instructions and a prepared environment, then run a test script after the agent finishes. Or the evaluation might use a judge model to inspect the result. In either case, the benchmark must make the checking rule explicit enough that different agents can be compared under the same rule.

From benchmark score to operational results

Harbor makes the agent's task performance measurable, but task success is not the only concern when choosing a system. The talk connects these evaluations to LangSmith, where teams can inspect experiment results and operational measurements.

Click to enlarge

At 12:43 — LangSmith's Experiments view shows experiment results with reward feedback, latency percentiles, token input/output measurements, and error-rate information.

The visible experiment view supports a broader evaluation model. A team can track an outcome such as accuracy or reward alongside properties such as:

• latency — how long the run takes;
• tokens — how much model input and output the run uses; and
• cost — an operational consequence of model and harness choices.

A high task score is therefore not automatically the best production choice. A team may need to consider the trade-off between task performance and the resources or time required to obtain it. The exact chart values and interface labels are not supplied in the source description, so no particular numerical result is inferred here.

Why this changes harness decisions

The main value of Harbor is the shared experimental frame:

same tasks and tests
          ↓
run model + harness combinations
          ↓
compare scores and operational measurements
          ↓
decide what to change next

Suppose one harness gives a model better tools or better context handling. Running both harnesses over the same task dataset can reveal that difference. The team can then investigate whether the change improved the harness, changed the model's effective behavior, or introduced a cost or latency trade-off.

That is how Harbor fits the talk's larger argument. Agent intelligence is distributed across the model, the context, and the harness. A repeatable benchmark helps a team measure the whole agent setup instead of treating the model as the only variable. Observability then supplies the operational details needed to understand the measured runs and choose the next experiment.

Key takeaways

• Harbor runs agents against repeatable datasets of sandboxed tasks.
• A task includes an environment, an instruction, and an evaluation method; it may also include a reference solution for a sanity check.
• A test can be implemented with code, unit tests, an LLM judge, or an agent judge.
• The benchmark enables comparisons across models, harnesses, and reasoning efforts on the same work.
• LangSmith adds visibility into reward or accuracy together with latency, token use, and errors.
• The goal is not only to obtain a score. It is to create evidence for deciding what to improve in the model, context, or harness.