Dogfooding, human feedback, and regression replay

An evaluation system should not learn only from a carefully prepared offline dataset. It should also learn from people using the product and encountering real outputs. In this chapter, the presenters describe an internal dogfooding loop: people use the image-enhancement experience, report what they see, and help the team find cases that need investigation before a new configuration is released.

Source focus (19:47–20:30): The presenters describe thumbs-up and thumbs-down signals plus free-form feedback from merchants and Uber teams, including design and other product teams. Flagged good and bad examples are fed back through the system. The resulting metrics are benchmarked before the latest configuration is pushed to production.

What dogfooding adds

Dogfooding means using a product internally so that the people close to the product can expose real failures and useful successes. Here, the product is the image-enhancement workflow. Its users provide feedback on actual generated results rather than on an abstract model output in isolation.

This feedback is different from a formal, human-labeled evaluation dataset:

Feedback source	Main value	Typical form
Formal evaluation data	A consistent basis for comparing versions	Human labels collected under defined guidelines
Dogfooding	Real examples and practical reactions from product use	Thumbs signals and written comments

The distinction is important. A formal dataset makes comparisons repeatable. Dogfooding can reveal cases that the team did not think to include, such as an unusual dish, presentation, or merchant context. Dogfooding therefore complements structured evaluation; it does not replace it.

Click to enlarge

At about 19:53, the visible slide supports the two feedback forms: thumbs-up/thumbs-down controls appear below an example output, alongside a free-form feedback collection step.

Two kinds of human signal

The loop combines a quick binary signal with an explanation:

• Thumbs up or thumbs down gives a compact judgment about an output. It is easy to provide and easy to aggregate across many examples.
• Free-form feedback lets a person describe what seems wrong or useful. It can preserve details that a binary value cannot express.

These signals answer related but different questions. A thumbs-down result says that someone disliked or rejected the output. A comment may help explain whether the concern was an unfaithful edit, an unhelpful visual change, a policy issue, or something else. That interpretation is a teaching model for using the signals; the source does not provide a fixed taxonomy for comments.

The people contributing feedback also bring different perspectives. The presenters name merchants, design teams, and other product teams. A merchant may notice whether the result represents the offered food. A design team may notice whether the visual treatment fits the intended experience. Other product teams may find problems in the way the feature behaves in the wider application. These examples clarify why feedback should not be treated as just another automated model score. They are teaching illustrations, not additional source-reported findings.

From a flagged example to a safer configuration

The important mechanism is not simply collecting comments. The comments and ratings become test cases in a release workflow:

1. Collect feedback. People identify good and bad outputs through the app and can add written context.
2. Flag examples for investigation. The team gathers cases that may reveal a failure or a behavior worth preserving.
3. Bucket the failures. Related cases can be grouped so the team can reason about a pattern instead of fixing one image blindly. The diagram shows a failure-bucketing stage, but the presenters do not specify its taxonomy.
4. Tune the agent configuration. The suspected issue is used to propose a configuration change. The source discusses configuration tuning, not a particular model-retraining procedure.
5. Replay the examples. The flagged good and bad cases are run through the updated system again.
6. Benchmark the metrics. The team checks whether the change behaves acceptably on the replay and benchmark data.
7. Release only after the check. The latest configuration version is pushed to production after the metric validation step.

Click to enlarge

At about 20:19, the visible workflow supports this order: feedback collection, failure bucketing, agent tuning, replay and metric validation, and production configuration release.

Replay is regression control

Regression replay means running known examples again after a change to check that the change did not make important behavior worse. It is a form of memory for the system. Without replay, a team may fix the newest complaint while accidentally breaking a case that already worked.

The good and bad labels should not be treated as if every example has the same cause. A flagged bad output might expose a routing problem, an editing problem, or a QA gap. A flagged good output can also be valuable: it gives the team a behavior that a new configuration should preserve. Replay puts both kinds of evidence back into the evaluation path.

The release condition can be expressed as a simple safety boundary:

feedback
   ↓
flagged examples → failure buckets → configuration tuning
                                      ↓
                              replay and benchmark
                                      ↓
                         metrics acceptable? ── no ──→ revise
                                      │
                                     yes
                                      ↓
                              push configuration

The diagram is a teaching abstraction of the described workflow. The source does not state the metric names, thresholds, replay-set size, or exact bucketing rules. Those details matter in an implementation, but they cannot be inferred from the talk.

Why this closes the loop

A one-way pipeline produces outputs. A closed-loop system uses evidence about those outputs to change the system, then checks the change before deployment. Dogfooding supplies feedback from product use. Diagnosis and tuning turn that feedback into a configuration update. Replay and benchmarking test whether the update is safe enough to release.

This creates a useful separation of responsibilities:

• People provide evidence about outputs they encounter.
• The workflow organizes and diagnoses that evidence.
• Tuning proposes a configuration change.
• Replay checks both the newly targeted failures and previously observed behavior.
• Benchmarking gates the production release.

The central lesson is that human feedback becomes operationally valuable when it is connected to regression testing. Thumbs and comments alone are reports. Once flagged examples are replayed and metrics are checked, those reports become evidence for controlled improvement rather than an uncontrolled stream of ad hoc fixes.