When building an agent harness, it is not enough to build something that runs. We need to find out how well it ran, why it produced that result, and what to change next. This chapter explains private evals and observability for that purpose.
The speaker says that an organization using a mission-critical agent—an agent whose failures have serious effects—needs its own evaluation standard. General benchmarks or model marketing alone cannot show whether the agent is good enough for that organization’s work.
評価と可観測性は、別の問いに答える
Evals and observability answer different questions
An eval judges an agent’s result against a standard defined in advance. For example, it can determine whether the agent returned the correct answer or completed the required steps. An eval answers the question, “Did this run meet the standard?” It is important that the speaker does not give a formal metric or scoring formula in this chapter. We should read this as an explanation of the role of evals.
Observability makes what happened inside a run visible. We can inspect what context entered the model, how the context grew during the run, and which steps and tools ran. Observability answers the question, “What happened internally before this result was reached?”
Suppose an agent has a low success rate. An eval shows the low result, but it does not by itself show the cause. With observability, we can investigate whether necessary information was missing from the context, whether a tool result was lost along the way, or whether another problem occurred. In other words, an eval measures performance, while observability shows where to look next.
If we give an agent important work from our organization, we need to evaluate it with data and standards that fit that work. In the idea the speaker presents, a private eval defines what “good” means inside the organization. This is different from expressing the model’s overall ability as one number.
For example, for one company, a “good” answer might not only be correct. It might also follow internal procedures, cite the necessary information, and use a required format. This is an explanatory example. The speaker does not give a specific scoring formula for such a standard. The important point is to fit the evaluation standard to the organization’s actual goal.
The speaker refers to three principles from an article by Satya while explaining how to build organizational advantage through AI investment. The article itself is not included in the supplied material, so this chapter covers only what the speaker explains. We cannot fill in the exact quotation or the article’s details from this source.
図：AIへの投資で組織の優位性を作るための三つの原則を示すスライドです。
Figure: A slide showing three principles for creating organizational advantage through AI investment.
This slide is a visual reference for checking the order and wording of the three principles the speaker introduces. Nearby frames show the same slide, and the three principles are clearly visible. However, the figure does not by itself show specific evaluation metrics or how to implement each principle.
Saving only evaluation results gives us limited information for the next improvement. The speaker emphasizes the importance of owning memory, traces, feedback, decisions, and the context accumulated by the organization. Here, a trace is a record that lets us follow what happened during an agent run afterward.
これらを残すと、組織は次のような記録を、将来の実行と実験に使えます。
By retaining these things, an organization can use records like the following for future runs and experiments.
どの入力とコンテキストが使われたか
どの手順やツールが実行されたか
利用者や評価者がどのようなフィードバックを返したか
どの修正を試し、どの判断をしたか
What inputs and context were used
Which steps and tools ran
What feedback users or evaluators returned
Which changes were tried and which decisions were made
測定して、少しずつ改善する学習ループ
Measure and improve a little at a time with a learning loop
The speaker calls this a “continuous learning loop” or a “hill-climbing machine.” Here, this means measuring a run, examining the result, trying a change, and measuring again. It does not mean training only the model.
流れを単純化すると、次のようになります。
In simplified form, the process is as follows.
組織のプライベート評価でエージェントを実行します。
成績と実行の記録を確認します。
問題に対応する変更を一つ以上試します。
同じ基準で再び測定し、変更の影響を比べます。
Run the agent with the organization’s private eval.
Examine the performance and the run record.
Try one or more changes that address the problem.
Measure again with the same standard and compare the effect of the change.
The target of a change is not limited to one part. Based on evaluation results and traces, we can change the harness steps. We can change or fine-tune the model. We can also update memory or context so that the next run receives the knowledge it needs. Therefore, the places where agent intelligence can be improved span all three parts: the model, the context, and the harness.
Suppose an eval finds this failure: “The agent found the correct source, but could not reflect it in the final answer.” When we inspect the trace, the search result may have entered the context, but an important part may have been dropped during summarization at the final stage. In that case, we can change the harness’s summarization method or the way it passes context, and run the same eval again. This concrete example was created for explanation; the speaker did not report these failure details.
The “hill-climbing” metaphor does not guarantee that every change improves the system. A change can improve one eval while making another task worse. That is why we need to use the same private evals and run records repeatedly, and check what changed as a result of which intervention.
The idea so far is to use evals and observability to improve an agent that fits the organization. The next chapter turns to Harbor, which the speaker introduces as a concrete example. Harbor is described as a runner that repeatedly evaluates agents on tasks in sandboxes. There, the way to compare models and harnesses on the same work becomes more concrete.