Harborでエージェントのタスクを比較可能なベンチマークにする

Harbor's Role

In this talk, Harbor is presented as an open-source evaluation runner (eval runner). Harbor itself does not produce answers as an agent. It is a system for evaluating agents running with a model and harness repeatedly on tasks under the same conditions.

What Makes Up a Benchmark

It is helpful to think about one evaluation in the following order.

1. Agent: The combination of the model and harness being evaluated.
2. Task dataset: A collection of multiple tasks for the agent to run.
3. Environment or sandbox: A separate workspace prepared for running each task.
4. Solution: A reference for checking that the result is not far off. The talk explains an approach that treats this as a golden solution.
5. Test or verifier: Something that examines the agent's run and determines the score.
6. Instruction prompt: An explanation that tells the agent what to do for that task.

Separating the sandbox for each task has two meanings. Even a stateful agent does not mix the work state of one task with another task. It also allows many tasks to run in parallel in their respective environments. This explanation is about the setup for execution locations and parallel runs; it makes no additional claim about security.

Figure: The Frontier-Bench leaderboard compares agents by model, reasoning effort, accuracy, release date, organization, and other factors.

This screen shows the idea of comparing agent configurations on the same benchmark. The table shows comparison columns and accuracy, but this material does not provide specific values or task names, so they are not inferred here. The relevant moment is 10:54 in the video.

Preparing an Environment for Each Task

In Harbor tasks, the environment for doing the work is defined along with the work itself. The way to build the environment can be specified with a Dockerfile. A Dockerfile is a file containing the steps for assembling a workspace with the required software and files. This makes it easier to give the same task to different agents in the same way.

Figure: Harbor Evals shows agents running in separate sandboxes for multiple tasks in a dataset.

The diagram places a sandbox and an agent for each task under a dataset containing multiple tasks. This helps the reader see the structure for evaluating tasks one by one in the same format. The relevant moment is 11:18 in the video.

Files That Define a Task

The talk's slide shows the environment, Markdown instructions, and an evaluation script as parts of a task. The evaluation script defines what to check after the agent finishes its work. A repository tree is also shown, making clear that a task is not just a one-line prompt but a complete set of executable work.

Figure: The Harbor Tasks slide places a container environment, Markdown instructions, and shell-based evaluation beside an example repository structure.

This screen shows the environment required by the task, the instructions for the agent, and the evaluation method separately. This allows the evaluation criteria to be managed separately from the task's execution steps. The relevant moment is 11:35 in the video.

Figure: Another Harbor Tasks screen shows an evaluation script and a solution directory containing solve.sh.

This screen shows a structure in which evaluation-script files and solution files are stored separately. However, the screen itself does not call solve.sh a golden solution, nor does it show how the verifier behaves. The relevant moment is 11:58 in the video.

A Solution and a Test Are Not the Same

A solution (golden solution) is a reference for checking whether the expected approach or result has been missed by a large margin. A test or verifier, in contrast, examines the work actually performed by the agent and determines the evaluation result. Having a solution does not necessarily mean comparing the agent's output with it character by character.

Determining the Score with Tests

There are several options for designing a test.

• Use ordinary code to check the artifact or final state.
• Run unit tests to check the required behavior.
• Have another large language model (LLM) judge the result.
• Use another agent as a judge.

The benchmark designer decides which method to use and how many points to award. A judging LLM or agent has a different role from the agent being evaluated. Without this distinction, the worker and the scorer can be confused.

Comparing the Same Work Under the Same Conditions

When the tasks and evaluation method are fixed, different models, different agent harnesses, and different reasoning efforts can be compared on the same work. Rather than comparing results produced under different conditions, this is, in a sense, a way to compare apples with apples. Harbor is positioned as a runner for repeating this comparison across many tasks.

Example: Give the same task dataset to Model A and Model B. Both use the same sandbox, instructions, and tests. The difference in scores can then be investigated in relation to the difference in the models or harnesses. This is an example for explaining the mechanism, not a specific measurement reported in the talk.

Figure: LangSmith's Experiments view shows reward feedback, latency, input and output tokens, and error rate together with experiment results.

In addition to benchmark accuracy or reward, operational metrics can also be tracked. The screen shows reward, latency percentiles, input and output token counts, error rates, and a table of experiment results. Specific values and fine-grained labels on the screen are not inferred here. The relevant moment is 12:43 in the video.

Distinguishing the Related Names

The talk also mentions Terminal Bench 2, Frontier Bench, and LangSmith alongside Harbor. They appear as related benchmarks or platforms, but they are not the same thing. The statement that Harbor is becoming an emerging industry standard is the speaker's presentation. It is not treated here as a fact confirmed by external evidence.

Summary

Harbor's central idea is not to try an agent only once. It puts the task, environment, instructions, solution reference, and test together so the same work can be evaluated repeatedly. The results can be used to compare models, harnesses, and reasoning settings. By viewing accuracy or reward together with latency, tokens, cost, and other measures, teams can also investigate practical differences that a score alone cannot show.