Human-aligned offline tuning and the recall guardrail

The router is not ready to ship just because it produces plausible decisions. It must first be aligned with judgments about which images should be enhanced and which should be left alone. In this chapter, the speakers describe an offline process that uses human labels as the reference point, tunes the agent, and releases it only after it meets its guardrails.

Start with a representative golden dataset

A golden dataset is a set of examples with human-provided labels. The agent's decisions are compared with these labels during tuning and evaluation. Here, “golden” means that the dataset is the working target for alignment. It does not mean that human labels are automatically perfect or that human disagreement disappears.

The examples need to represent the marketplace, not just the easiest or most common cases. The speakers describe sampling across several cuts—subgroups used to inspect system behavior—including:

• geography,
• dish type, and
• image-quality type.

This matters because a router can look strong on an overall average while failing on a particular region, food category, or quality level. Representative sampling makes those failures visible during offline evaluation. The talk does not specify the sample size, the exact meaning of every cut, or an annotation-agreement procedure.

Human labelers also use objective guidelines. Guidelines turn a vague question such as “Does this image need enhancement?” into consistent criteria. They can reduce subjective noise and bias, but they do not guarantee that every labeler will agree.

The offline compare–tune–guardrail–ship loop

The process is iterative:

1. Collect human labels. Labelers assess representative images using the objective guidelines.
2. Run the agent. The router evaluates the same examples and chooses a route, such as enhance or skip.
3. Compare the results. The system checks the agent's decisions against the human-labeled target.
4. Check the guardrails. The team examines the routing metrics, with special attention to recall.
5. Ship or tune. If the guardrails are acceptable, the routing configuration can be shipped. If not, the agent is tuned and the loop is repeated.

The important separation is between three jobs:

• Human labeling supplies the reference judgments.
• Automated comparison measures how the agent differs from those judgments.
• The release decision applies the guardrails to decide whether the current version is safe to ship.

The labels are therefore not just a training artifact. They are also an evaluation target and part of the release evidence.

Click to enlarge

Caption: A workflow diagram shows offline human alignment from human labeling through ground truth, evaluation, tuning, and shipping.

What the visual supports: The diagram shows the path from human labeling to ground truth, evaluation, tuning, and shipping. It does not visibly list the sampling axes of cuts, geographies, dish type, or image-quality type.

Why recall is the guardrail

For a binary router, recall depends on which class is treated as the positive class. In this use case, consider the positive class to be an image that is genuinely bad or otherwise should be caught by the routing policy. Then:

[ \text{recall} = \frac{\text{bad images correctly caught}}{\text{all truly bad images}} ]

High recall means that few genuinely bad images slip through as though they were acceptable. This is the risk the speakers emphasize. A missed bad image can reach a later stage or remain unaddressed, even though the routing system had the opportunity to identify it.

Recall is not the same as precision. Precision asks how many images marked for action really belonged in that class. Recall asks how many of all relevant cases were found. A router could catch nearly every bad image but also flag many good images; that would favor recall while creating over-routing and unnecessary enhancement. The following chapter examines that other failure mode. In the current design, recall is an explicit guardrail so that improving selectivity does not allow bad images to pass unnoticed.

The larger lesson

Offline tuning makes the router's objective concrete. Instead of optimizing only for a model score or a visually attractive output, the team checks behavior against human-aligned judgments across the marketplace's variety. The release gate then turns those judgments into an operational safety condition.

This is still an offline checkpoint, not proof that the system will remain correct forever. The data, inputs, and production conditions can change. Later, the lesson describes how fresh production samples and human labels are used to detect drift and tune the system again. The golden dataset and its guardrails remain the benchmark that a proposed change must satisfy before release.