静的なオフラインモデルからオンラインのドリフト補正へ

This chapter explains how to maintain and keep improving a model that was tuned once offline after it enters production. The central claim of the presentation is that a static model gradually fails as real-world conditions change, so production data must be evaluated regularly and the results must feed back into the next tuning step.

Static models become outdated in production

Here, drift means that data conditions or model behavior change, so a configuration that was once adequate no longer fits production. For example, if the types of dishes, regions, or ways of taking photos change, a router tuned on past data may miss images that need enhancement or select images that do not need it. This is not a problem that ends when a model is trained or tuned once.

Example: a router that was correct until yesterday

This is an author-created example for understanding. Suppose that an increase in photos of a new dish in one region causes the router to stop judging their blur or composition correctly. If the model remains fixed, the failure will remain in production. Examining new production examples can provide clues about what changed and where tuning is needed.

Run a closed loop with production data

In the flow shown in the presentation, it is not enough merely to record production processing. The system regularly samples data that was actually used, then compares human evaluation with agent output. When it finds a mismatch, it diagnoses the cause and tunes the configuration automatically. It then checks the changed version again against safety criteria.

1. Regularly select a sample from production data.
2. Have human labelers label the sample using the same guidelines as in offline evaluation.
3. Compare the agent's output on the same data with the human labels.
4. If there is a mismatch, diagnose the problem by broadly classifying where it occurred.
5. Use the diagnosis to tune the prompt, agent, or system configuration automatically.
6. Benchmark the tuned agent on golden data.
7. Ship it if it satisfies the guardrails; otherwise, tune it further.

This sequence becomes a closed loop because the final result—shipping or tuning again—leads to evaluation of the next production data. In a simple one-way pipeline, a failure found in production does not return to the initial routing or configuration. In a closed loop, evidence from operation becomes the input to the next improvement.

Keep diagnosis separate from tuning

Diagnosis is the work of identifying where in the system a problem occurred. For example, the team separates a router judgment, an editing agent's change, and a QA miss. Tuning is the work of changing a prompt, agent, or configuration in response to that diagnosis. If the team changes the whole system without identifying the cause, it may alter parts that were not responsible and create another failure.

Supplement: an example of diagnosis

For example, if the system often makes the wrong enhance-or-keep decision only for images of a particular dish, the team first investigates the router's conditions. If the router is correct but only the color becomes unnatural after editing, the team investigates the editing configuration. Diagnosis therefore connects a mismatch to the place that should be fixed instead of simply labeling the whole system as bad.

Human labels are still needed for automatic tuning

The presenters explain that a person does not manually approve every change in this tuning loop. However, this does not mean that people are unnecessary. Samples selected from production are sent to human labelers and evaluated using the same objective guidelines as offline evaluation. Those human labels become the reference for comparison with agent output.

Golden data is another safety net

Golden data is a representative dataset labeled by people. Fresh production samples help find drift, while golden data helps check whether tuning has damaged performance relative to the previous baseline. Online checking is therefore not a replacement for offline golden data or guardrails. Using both makes it possible to check adaptation to change and preservation of existing quality and safety at the same time.

Run benchmarks and guardrails before shipping

The tuned configuration is not sent straight to production. It is first benchmarked on golden data. Only when it satisfies the required guardrails is it shipped. If it does not meet the criteria, the change is stopped and diagnosis and tuning are repeated. This mechanism prevents online automation from lowering quality or safety without limit.

Figure: A slide showing a closed loop that moves from production traffic through verification, diagnosis and tuning, benchmarking, and shipping, then returns to new production traffic.

This diagram directly shows the flow described in the presentation. Production traffic is routed and verified with the same golden procedure. A mismatch leads to diagnosis and tuning, followed by benchmarking before shipping. The dashed line below represents starting the flow again with fresh production traffic. The diagram does not explicitly show a "static offline model" or the word "labeler". The relevant section is around 11:37 on YouTube.

Operational conditions that keep the loop alive

To keep this mechanism running, the system needs configuration-driven automation, observability that makes each process traceable, and guardrails that can detect and stop failures. The presentation shows keeping the system "alive" with these tools as an engineering consequence of the closed loop. The sampling interval, diagnosis categories, tuning method, and guardrail thresholds are not specified in the presentation.

The important point is not merely to add online evaluation. The system must compare newly appearing production failures with a human standard, locate their causes, change the configuration, and confirm safety on golden data before returning the change to production. With this flow, the system can keep adapting to changing data instead of accumulating failures while the model stays fixed.