診断器：より高いレベルのフィードバック抽象化

This chapter examines a diagnoser that treats several feedback loops as one system. In the presenters' view, the diagnoser is not another image-editing agent. It is a higher-level, or meta-level, layer that examines information from each loop and decides which agent's configuration should be fixed.

First, receive different kinds of feedback

The design assumes that feedback does not have only one entry point. For example, there are loops such as these:

• Model loop: It uses human ratings and production data to find drift and mismatches.
• Dogfooding loop: Internal teams and merchant-side participants use the app and send good/bad reactions or free-form comments.
• Marketplace loop: It observes production quality and outcomes from adding an item to the cart through completed orders, then uses them as clues for improvement.

What does the diagnoser do?

The diagnoser shown in the presentation sits above these different loops. Its job is not merely to collect feedback. It examines the incoming information, reflects on the system, and localizes the problem to one responsible agent or to multiple agents.

The process, in order, is:

1. Receive inputs from several feedback loops.
2. Reflect on those inputs and consider where the expected result and the actual result did not match.
3. Localize the problem to one or more responsible agents or configurations.
4. Route a configuration-specific fix to the tuning process for the identified location.

The benefit of this structure is that the team does not need a separate tuning script for every feedback entry point. For example, free-form comments in the app and marketplace metrics can have very different forms. If the diagnoser works as a shared higher-level layer, it can search for the relevant agents and configurations and connect any entry point to the needed fix. The presentation does not explain the internal routing rules or configuration format. Therefore, the important idea here is not a specific algorithm, but a design that handles several loops together.

Keep diagnosis, tuning, and release decisions separate

The diagnoser's work is not the same as the work that follows. Diagnosis finds where to make a change. Tuning changes the prompt or configuration of the identified agent. The team does not immediately release that change to production; it checks it again with a benchmark. Only a change that meets the guardrails is treated as a safe candidate for a release decision. This separation prevents the roles of finding a problem, creating a fix, and deciding whether the fix may be released from being confused.

Supplement: Even when the diagnoser finds a problem, it does not necessarily rebuild the whole system. The idea shown in this presentation is to route a targeted configuration fix to one or more relevant agents.

The diagnoser in the diagram

Figure: In the "Closed Loop Feedback" diagram, three feedback paths converge on Diagnoser. The diagnoser's Reflect and Route stages select the agents to tune through an agent-configuration output. In other words, the diagnoser is not an individual component beside each loop. It is a layer that looks across several loops and directs the fix to the right place. You can check this scene at 19:27 in the video.

This abstraction connects different signals—model evaluation, internal use, and the production marketplace—to the same improvement process. The diagnoser is not responsible for generating a "good image" itself. It organizes which agent and configuration are worth tuning, then passes that fix to the next tuning and benchmarking steps.