The diagnoser: a higher-level feedback abstraction

The system now has several ways to learn that something is going wrong. A model-evaluation loop may find a mismatch with human judgments. Product use may reveal a bad result. Marketplace behavior may show that an apparently good change is not helping a particular segment. These signals do not necessarily identify the same component or require the same repair.

This chapter introduces the diagnoser. It is a higher-level layer that receives feedback from different loops, reasons about what the feedback means, identifies the agent or agents that need attention, and routes a targeted configuration fix.

Source focus (19:19–19:47): The presenters place a diagnoser above multiple feedback loops. It reflects on their inputs, identifies which agent or agents need optimization, and routes an agent-configuration fix to the selected agents. The source does not specify the diagnoser's internal routing rules.

The problem: feedback arrives at the wrong level

A feedback loop tells us that an outcome is unsatisfactory. It may not tell us which part of a multi-agent system caused the problem.

Consider the image-enhancement workflow. It contains several distinct responsibilities:

• understanding an image and its metadata,
• deciding whether to enhance or skip,
• generating an edit prompt,
• editing the image,
• checking the result, and
• deciding whether the result is safe to publish.

An observed failure could therefore have different causes. The router may have sent a good image to enhancement. The editor may have changed content that should have remained faithful. A QA configuration may have missed a failure. The same final symptom—an unsatisfactory image—does not prove that the same agent needs to change.

This is why a separate tuning workflow for every feedback source is a weak abstraction. Each workflow would need its own way to interpret evidence, select a target, and produce a repair. As the number of agents and feedback loops grows, those one-off connections become difficult to maintain.

What the diagnoser does

The diagnoser is not simply another image-editing agent. It operates about the agents. Its job is to decide where a change should be made, not to perform the image enhancement itself.

The process can be understood as four responsibilities:

1. Accept feedback. The diagnoser receives evidence from the available feedback paths. The evidence may describe a mismatch, a failure, or an undesirable outcome.
2. Reflect on the system. It examines the feedback in relation to the system's behavior. Reflection means asking what pattern the evidence reveals, rather than immediately changing the last component that ran.
3. Localize responsibility. It identifies the agent or agents whose behavior or configuration should be optimized. A problem may belong to one agent, or it may require changes in multiple agents.
4. Route the fix. It sends a targeted configuration repair to the selected agent or agents. The repair is directed by the diagnosis instead of being applied indiscriminately to the whole system.

The exact procedure inside each step is not given. In particular, the source does not define the routing rules, diagnosis categories, or configuration format. The important abstraction is the separation between receiving feedback and deciding which part of the system should respond.

Click to enlarge

At about 19:27, the diagram shows three feedback paths converging on a Diagnoser. Its Reflect and Route stages lead to an Agent config output, which is directed to selected agents for auto-tuning. The diagram supports the layer's role above multiple feedback sources; it does not disclose the internal routing logic.

Why this is a meta-level abstraction

An ordinary agent performs a task in the workflow. For example, an editor changes an image according to its inputs. The diagnoser performs a different kind of task: it interprets evidence about agents and decides where optimization should occur.

That makes it a meta-level component. “Meta-level” means that the component reasons about the behavior of other components rather than directly carrying out their primary task.

The distinction matters:

Layer	Main question	Typical result
Task agent	“What should I do for this case?”	An image description, route, edit, or QA judgment
Diagnoser	“Which part of the system explains this feedback?”	A selected agent or set of agents to optimize
Tuning process	“What configuration change should we try?”	A revised agent configuration
Release benchmark	“Does the revised version meet the safety and quality bar?”	Evidence to continue, release, or iterate

The diagnoser can therefore send work to a downstream tuning process without being identical to that process. Diagnosis chooses the target. Tuning proposes or applies a configuration change. Benchmarking checks whether the changed version is acceptable. Keeping these roles separate makes the causal chain easier to inspect.

One feedback source, several possible targets

Suppose a feedback loop reports that an enhanced image is not faithful to its source. There are several possible interpretations. The router might have failed to recognize that the input was unsuitable for enhancement. The editor might have introduced an unsupported object. Or a QA check might have failed to detect the change.

The diagnoser's value is not that it magically proves one explanation. Its value is that it provides a common place to analyze the evidence and select a repair target. If the evidence points to the editor, the configuration fix can be routed there. If it points to the router and the editor, the repair can target both. The source explicitly allows for one or multiple agents; it does not describe the criteria used to choose them.

This targeted behavior is safer than retuning every component whenever any loop reports a problem. A global change could fix one symptom while disturbing components that were already working. A configuration-specific repair narrows the intended change, which makes later evaluation and regression checking more meaningful.

The last sentence is a teaching implication of the architecture, not a reported measurement from the talk. The presenters' stated point is that the diagnoser identifies the agent or agents to optimize and routes the configuration fix accordingly.

How it fits the closed loop

The diagnoser connects feedback collection to system improvement:

feedback from several loops
            │
            ▼
       diagnoser
   reflect → localize → route
            │
            ▼
 configuration fix for selected agent(s)
            │
            ▼
       tuning and evaluation

This is not a one-way pipeline. After a configuration is changed, the system still needs the tuning and evaluation safeguards described earlier in the lesson. A proposed configuration should be checked before it becomes the version used in production. The diagnoser chooses where to direct the repair; it does not replace the benchmark or the release decision.

The abstraction also prepares the next step. Once a feedback signal has been converted into a target and a configuration repair, the system can replay relevant examples, benchmark the result, and release a new version only after the checks succeed. Those later steps validate the repair; they are not part of the diagnoser's localization decision.

Central idea

Multiple feedback loops create a coordination problem: feedback can say that the system needs improvement without directly saying which agent should change. The diagnoser is the proposed higher-level answer. It accepts feedback, reflects on system behavior, localizes the responsible agent or agents, and routes a targeted configuration fix.

It is therefore useful to keep three questions separate:

1. What went wrong? Feedback supplies the evidence.
2. Where should we change the system? The diagnoser localizes the target.
3. Is the change safe and effective enough to use? Tuning, benchmarking, guardrails, and release controls answer this later.

The source leaves the internal diagnosis method unspecified. Its architectural lesson is clear: put a reusable diagnosis layer above heterogeneous feedback loops so that new feedback sources can be connected to a common optimization workflow instead of requiring an unrelated tuning script for each one.