プロンプト最適化・バージョン管理・安全なクローズドループ展開

Key point of this chapter

The auto-tuning described by the speakers is not a process of retraining model weights. It updates the target agent's prompt or configuration, evaluates that change again, and only then sends it to production. Two sub-agents, Reflect and Synthesize, analyze mismatches and create a new configuration. The system then registers only versions that pass, and operates them safely with observability, guardrails, and rollback.

The path from tuning to production

1. Use mismatches found by human evaluation as material for tuning the target agent.
2. Have the prompt optimizer organize the mismatches and create a new agent configuration.
3. Benchmark that configuration and register only versions that meet the criteria in the agent store.
4. A later production run uses the new version. During operation, monitor its behavior and revert quickly if there is a problem.

1. Tune the configuration, not the weights

The target of tuning here is not the weights of the image model itself. In the speakers' explanation, the system updates the target agent's prompt or related configuration. Therefore, the main change is not what the model relearns, but what instructions and settings the agent receives.

This means that even a small prompt change can change production output. At the same time, a configuration-only change can affect quality and safety, so evaluation after the change must not be skipped. The speakers do not show the concrete configuration format or internal schema.

2. Reflect and Synthesize turn mismatches into a configuration change

At the center of auto-tuning is a prompt optimizer. It does not simply repeat one operation. Instead, it uses sub-agents with different roles to create the next configuration from evaluation results.

• Reflect examines cases where human evaluation and the agent's output did not agree. Before analyzing them, it removes noise and checks whether the issue is an accident in a particular image or a systematic problem in the dataset as a whole.
• Synthesize receives the feedback organized by Reflect and turns it into a new agent configuration. This includes an updated prompt and related settings.

This division matters because it prevents mismatches from being pasted directly into a prompt. First, Reflect considers what the problem is and how broadly it occurs. Then Synthesize converts that analysis into an executable configuration change. This process tries to fix recurring problems rather than making an ad hoc fix for one failure.

Figure 1: Evaluation and benchmark results enter the Prompt Optimizer, pass through Reflect and Synthesize, and return from the Agent Config Store to the target agent.

This diagram shows a cycle in which evaluation results are used to tune a prompt. Reflect reviews the mismatches, and Synthesize creates the next configuration. Because that configuration returns to the target agent, tuning does not end after one attempt; evaluation and improvement can continue. The diagram's structure is visible, but it does not show the actual prompt contents or configuration format.

Watch the relevant part at 754 seconds on YouTube

3. Keep unseen human-evaluation samples

The speakers' flow includes a target agent and human-evaluation samples that it has not yet seen. Mismatch cases can provide material for proposing a configuration change, but the same cases cannot show whether the changed configuration works on other samples. Keeping unseen samples in the evaluation story makes it possible to check the proposed change on a broader set of cases.

As a supplement, "unseen" does not mean that the speakers explained a particular training/evaluation split. They do not give the split procedure or proportions. The reliable point is that the evaluation flow includes not only the mismatches used for tuning, but also unseen human-evaluation samples.

4. Send only benchmark-passing versions to production

Creating a configuration is not enough to send a new version to production. In the release procedure described by the speakers, the order of changing, reevaluating, registering, and using the version is preserved.

1. Update the agent configuration using the results from Reflect and Synthesize.
2. Benchmark the updated agent again on evaluation data.
3. Register the version that passes the guardrails in the agent store.
4. A later production run retrieves and uses that version from the store. If it does not pass, do not register or deploy it; return to the tuning loop.

Figure 2: The diagram "How Do We Tune Our Agent?" shows a closed loop including the target agent, evaluation, prompt optimization, and configuration registration.

In this diagram, results from evaluation and benchmarking return to prompt optimization, and the updated version is registered as a configuration. This makes the idea visible: do not stop after changing the configuration; use only a version that has passed the benchmark in the next run. The detailed flow of observability, guardrails, and rollback mentioned by the speakers cannot be confirmed from this diagram alone.

Watch the relevant part at 793 seconds on YouTube

5. Put autonomous tuning inside a safety envelope

Because this flow can update configurations automatically, a person does not need to rewrite the prompt every time. However, "no human-in-the-loop" does not mean that humans are completely absent. Humans provide labels for evaluation data, and they support the criteria used to check whether a version passes. Automated tuning, human-aligned evaluation data, and release safeguards have different roles.

For safety, the speakers combine three elements.

• Observability: Make it possible to track how the new version behaves in production.
• Guardrails: Monitor the conditions that a version must meet for quality and safety.
• Quick rollback: When a problem is found, return to a previously stable configuration.

This safety envelope is not only meant to stop autonomous tuning. It reduces the risk that a bad configuration remains in production for a long time while automatic improvement continues. Therefore, the system must be designed with both optimization speed and a way to detect and undo failures.

Example: From a mismatch to a safe version

This is an explanatory example for understanding the mechanism. Suppose a human evaluator decides that an agent's output is faithful to the food content but lacks natural plating, creating a mismatch. Reflect checks whether this is noise in one case or a problem that recurs for the same type of image. Synthesize turns that analysis into a new prompt configuration. The new configuration is benchmarked, and it is registered in the agent store only when it also meets the guardrails on unseen samples. If production behavior worsens, monitoring can trigger a return to the previous version. This example illustrates the idea; it does not mean that the speakers presented these specific evaluation criteria or steps.

What is not disclosed

The speakers do not disclose the configuration-file format, the implementation of the agent store, benchmark pass criteria, guardrail values, or the split method for unseen samples. The captions also contain a cue in which Synthesize sounds different, but here it is treated as Synthesize because that fits the context. Therefore, do not infer a concrete schema or threshold from this chapter.

Summary

The central idea of this closed loop is to turn failure records into configuration improvements, benchmark those improvements again, and only then send them to production. Reflect analyzes mismatches, and Synthesize creates a configuration. Unseen human-evaluation samples help check the change, while the agent store manages passing versions. Finally, observability, guardrails, and rollback form the boundary that makes autonomous improvement safe to operate.