なぜ検証だけでは十分ではないのか

AI agents are often said to require humans to understand the code they write. The first reason people give is "to check whether it is correct." This chapter starts with that idea and examines the speaker's claim that verification alone cannot explain all the purposes of understanding.

Is understanding only for checking correctness?

If humans understand an agent's output, they can find mistakes. This is an important reason. However, "correct" has several meanings here.

• Matching the specification. We check whether it performs the requested behavior.
• Not causing an incident in production. We check whether it will break the environment where it is actually used.
• Being well designed. We look at whether the overall system structure is appropriate, not only at the behavior in front of us.

These three ideas are similar, but they are not the same. Passing a test for the specification does not guarantee that nothing will go wrong in production. Also, working without problems does not guarantee that the design is good.

Agents run verification loops

The speaker describes the change as follows. A human gives the agent a verification procedure. The agent can then use that procedure to check its output. It asks the necessary questions, investigates the answers, fixes problems if it finds them, and checks again. It can repeat this flow many times as a verification loop.

The important point is not that the agent tests only once. It can spend time repeating checks and fixes. As an agent's verification ability improves, the human share of the work of checking correctness becomes smaller.

As a result, the problem of code review being a bottleneck also changes. If the main purpose of code review is only to judge whether an agent's output is correct, the agent's verification loop may replace much of that role.

Example: The points we examine differ according to the purpose, even for the same code

Example (for explanation)

Suppose an agent writes a change that makes a user authenticate again after a certain amount of time has passed since login.

From a verification perspective, we check the following:

1. Does reauthentication happen after the time specified?
2. In production, could it cause an incident in which even normal users are suddenly logged out?
3. Are the location and structure of the change appropriate as part of the system's design?

If we give the agent procedures for checking these three points, it can repeat tests and related checks. The human may end up only looking at the verification results and deciding "pass" or "fail." This is an example of thinking about understanding for verification.

Understanding for verification and understanding for participation

Here the speaker separates two purposes of understanding.

• Understanding for verification: Understanding used to judge whether an output is correct and meets the criteria.
• Understanding for participation: Understanding that lets humans grasp the system and the flow of work, and continue taking part in what comes next.

If we consider only the first purpose, then humans need less understanding once agents can verify sufficiently well. But it is too soon to conclude from that that "humans no longer need to understand." The speaker is not saying that verification is useless. He is saying that verification is not the whole reason to understand.

Example (for explanation)

Suppose the agent checks the earlier login change against the three criteria—specification, production safety, and design—and passes all of them. From those results, the human can judge that the code meets the criteria.

Even so, receiving verification results is not the same as understanding the meaning of the change. In this chapter's distinction, the former is "judging correctness." The latter concerns understanding that enables participation in the project. The next question—how much humans should understand—is broader than a simple pass-or-fail judgment.

The question that remains beyond a correct output

When AI agents can verify their own work, the human role is no longer only to "check correctness." What, then, should humans understand, and how should they take part in the work? The speaker presents this as a problem of understanding beyond verification.

In other words, an agent's ability to produce correct code and a human's ability to participate in that work are separate things. Automating verification reduces the burden of checking correctness. At the same time, it makes the problem of maintaining human understanding more visible.

This distinction is the turning point of the talk. Understanding is not limited to a safety check for finding mistakes. It is also necessary for humans to remain in the work and connect it to their next ideas and contributions.

Source

This chapter is based on approximately 2:23–4:09 of Geoffrey Litt's talk, "Understanding is the new bottleneck — Geoffrey Litt, Notion." Watch this section of the talk on YouTube