人間に合わせたオフライン調整と再現率のガードレール

In this chapter, we examine how to tune a router that decides whether an image should be enhanced or left unchanged so that it aligns with human judgment. The key is to use human labels as the target, test on representative data, check guardrails, and ship only afterward.

Using Human Labels as a Golden Dataset

In the talk, human-provided labels are treated as the golden source of truth, or a dependable reference. First, human labelers inspect a variety of images and record judgments such as whether an image needs enhancement and which route the router should choose. The dataset made from these human labels is the golden dataset. The agent's output is evaluated by comparing it with the judgments in this dataset.

However, this does not mean that human labels are automatically perfect. People can disagree, and labels can contain noise. Their role here is not to create a standard based only on the agent. It is to align the agent with the outcomes people expect.

Choosing Representative Data

A golden dataset should not contain only a narrow type of image. The talk explains that representative samples should span several cuts, geographies, dish types, and image-quality types. It does not specify the exact meaning of cuts or the required sample size. The idea, however, is clear.

For example, if the router is tuned only with dishes from one geography or only with high-quality images, we cannot check its decisions on another geography or on low-quality images. Including several ways to divide the data makes it easier to find out whether the tuning fits only a particular market or dish. This check is necessary when the system handles listing images from around the world.

Objective Labeling Guidelines

If people label images using only their personal impression, their judgments about the same image may differ greatly. Therefore, the team prepares objective labeling guidelines. A shared standard can reduce subjective bias and label noise.

This does not completely remove disagreement. The talk does not say that the guidelines reduce disagreement to zero. Their purpose is to align the foundation for judgment and make comparison with the agent meaningful.

Example: Instead of asking only whether an image looks attractive, decide in advance whether it represents the dish correctly and whether enhancement can improve it. This reduces the risk that different labelers use different meanings of a "good image."

Compare, Tune, and Ship Offline

The offline evaluation flow is as follows.

1. Collect representative images and have people label them according to the guidelines.
2. Give the same images to the agent and obtain the router's decisions.
3. Compare the agent's decisions with the human-labeled reference.
4. Check guardrail metrics such as recall.
5. If the metrics are within an acceptable range, ship the router. If not, tune it and evaluate it again.

Human labeling, automated comparison with the agent, and the automated decision to permit a release must be considered separately. People create the evaluation reference, but comparing each candidate and checking whether it meets the guardrails can be automated.

The figure shows the offline flow from human labeling through ground truth, evaluation, tuning, and shipping.

This slide supports the talk's central flow: human-created ground truth enters the evaluation pipeline, and the result leads to tuning and shipping. However, the slide does not explicitly show the sampling cuts described here, such as geography or dish type. The detailed flow is available at the relevant point in the source video.

Why Use Recall as a Guardrail?

In this use case, it is especially problematic if a bad image slips through the router. Therefore, recall is used as a guardrail that emphasizes not missing images that need attention. Intuitively, it asks: among the images that truly needed enhancement or appropriate processing, how many were sent to the correct route?

When recall is low, more images that should have been processed are missed and allowed through. Those missed images may not receive later enhancement or quality checks. Thus, even if an overall accuracy number looks good, the router cannot ship if it misses too many bad images.

Recall here does not measure whether an image after enhancement looks good. First, it measures whether the router sent an appropriate image to an appropriate route. The talk does not provide the actual class names, thresholds, calculation method, or required numeric target.

Short Summary

This offline process is not merely a one-time procedure for choosing model settings. The team labels representative images using a human standard, compares the agent with that standard, checks guardrails, and tunes and compares again when necessary. Giving special weight to recall helps reduce the danger of missing a bad image and ending processing. Human labels and data cuts still contain uncertainty, so it is important to make the standard and the operational goal clear rather than leaving the release decision to numbers alone.