This chapter examines an internal loop for improving an image agent while using the real app. It does more than collect user reactions. The team replays flagged problematic and good examples through the system, checks the metrics, and only then sends a new configuration to production.
Here, dogfooding means that the team uses its own product and gathers feedback from examples close to real operation. The talk explains that this internal loop collects thumbs-up / thumbs-down reactions to outputs and free-form comments. It is used both before and after launch to find real examples.
This loop is not a replacement for a formal human-labeled dataset. Human labels are an evaluation reference prepared under defined guidelines. Dogfooding, by contrast, is a feedback source that returns examples found in actual use to the improvement process. Using both makes it possible to examine controlled evaluation and concrete problems from the field as separate perspectives.
The figure shows two kinds of human input side by side. Thumbs provide a short signal about whether an output is good or bad. Free-form feedback adds a detailed opinion to that reaction. The figure shows how feedback is collected; it does not show that each comment automatically becomes a ground-truth label.
Thumbs-up / thumbs-down quickly communicates whether there may be a problem and which direction improvement should take. However, a binary reaction alone does not show what was wrong. Free-form feedback can preserve the specific point that concerned the user. Therefore, the two methods do not compete; they provide complementary information.
As a supplement, thumbs are useful for collecting signals across many examples with little effort. Comments help later when the team classifies failures or considers which agent to tune. However, the team should not immediately infer a root cause from one comment or one reaction.
The feedback contributors should not be collapsed into one group either. The talk names merchants, design teams, and other product teams. These are human opinions from different perspectives, and they should be treated as evidence separate from the automated scores calculated by the image agent.
Example (created for explanation): suppose an output receives a thumbs-down and the comment “It looks different from the original dish.” The thumbs-down signals a bad result. The comment is a clue that an investigator should examine faithfulness, meaning fidelity to the original content. This is an explanatory example, not an individual comment presented in the talk.
フラグを付けた例を再生する
Replaying Flagged Examples
フィードバックを設定変更へつなげる流れは、次のようになります。
The flow from feedback to a configuration change is as follows.
This replay is a check for controlling regression. When a configuration changes, one example may improve while another example that previously worked becomes worse. Running previously flagged examples through the system again creates an opportunity to find that change before release. Good and bad examples do not necessarily have the same label or root cause, so replay does not treat every case as identical.
Figure 2. A “Dogfooding/Internal Loop” diagram showing feedback collection, failure bucketing, agent tuning, replay and metric validation, and production configuration release.
図2では、フィードバックが失敗の分類とエージェント調整へ進み、その後に replay と metric validation が続きます。最後に production config release へ到達するため、調整した設定をすぐに本番へ出すのではなく、過去の例と指標で確かめる構造が見えます。これは、発表が説明する回帰制御の考え方を支える図です。
Figure 2 shows feedback moving to failure bucketing and agent tuning, followed by replay and metric validation. It reaches production config release only at the end, so the adjusted configuration is checked against earlier examples and metrics instead of being sent to production immediately. This diagram supports the talk’s idea of regression control.
In this flow, human feedback is not the same as an automated model score. Human reactions provide material for finding examples that matter in actual use. Replay and benchmarking are the stage that checks whether a change based on that material has broken expected existing behavior. The talk’s key point is that the team pushes a new configuration version only after checking the metrics.
The talk does not provide specific metric values, the number of examples to replay, the rules for bucketing failures, or the release thresholds. Therefore, this loop should not be understood as a system that automatically fixes every problem. It collects feedback, investigates causes, validates changes through replay, and updates the configuration within the range that has been checked for safety.