19

14:13 - 14:49

Worked example: sweet potato fries and pass at K

Watch from 14:13

The enhancement loop becomes easier to understand through a concrete food-image example. The input is an image of sweet potato fries. The first generated edit does not satisfy QA because the portion size and plating are wrong. QA sends that feedback back to the enhancement step. The next edit applies the suggestion and passes.

The important pattern is not simply “the second image looks better.” It is a controlled correction cycle:

initial image and editing instructions
                ↓
          first generated edit
                ↓
       QA: portion and plating wrong
                ↓
       specific feedback to the editor
                ↓
          revised generated edit
                ↓
               QA: pass

What the example teaches

1. Generation is not acceptance

An image generator can produce an output without producing an acceptable output. The first attempt is therefore only a candidate. The QA gate checks whether the candidate meets the required criteria.

In this example, the failure is not described as a simple pixel defect. Portion size concerns how much food is represented. Plating concerns how the food is arranged and presented. These are semantic and presentation constraints: a useful evaluator must reason about the food and its arrangement, not only about whether individual pixels changed.

That distinction matters because an edit can be visually different while still being wrong. Conversely, a useful correction may preserve most of the image while fixing the specific issue that QA identified.

Source visual at about 14:25 — The slide shows the first generated result marked QA FAIL, with portion-size and plating issues, and a second result marked QA PASS after a QA suggestion was applied. The slide supports the before-and-after sequence; it does not establish a general success rate or a numeric iteration limit.

2. Feedback must be actionable

A rejection alone would tell the editor only that something went wrong. The useful part of this example is the specific QA feedback. “Portion size and plating are wrong” gives the next enhancement attempt a direction for correction.

The loop therefore carries more than the latest image forward. It reuses the original enhancement inputs together with the QA feedback:

original description + routing directives + current image + QA feedback
                                  ↓
                         next enhancement attempt

This helps preserve the image's intended content while correcting the identified failure. The next attempt is not an unrelated generation from scratch. It is a response to an observed mismatch.

What “pass at K” means

The presenters call the measured outcome pass at K. In their explanation, this means the pass rate at the Kth iteration. It is an iteration-based success measure for the enhancement-and-QA process.

The sweet-potato-fries example reaches a pass on its second attempt. That illustrates how a feedback loop can turn an initial failure into a passing result. It does not mean that every image will pass by the same iteration, and it does not disclose a universal value for K.

It is also important not to confuse this measure with other evaluations in the system:

Measure or decision Question it answers
QA on one iteration Does this particular generated image satisfy the checked criteria?
Pass at K What pass rate is reached at the Kth iteration?
Final publish-ready QA Is the resulting item safe and ready to publish?

The source gives the name and meaning of pass at K, but it does not give a formula, reported values, or a numeric K. Those details should remain open rather than being inferred from this one example.

More iterations improve coverage, but have a cost

The speakers expect the pass rate to improve as the system has more opportunities to use feedback and correct an output. In practical terms, a larger iteration budget can help more difficult images reach the QA bar instead of being left unenhanced.

This creates a system tradeoff. Each additional attempt can consume more compute and add latency. More attempts can also create opportunities for unnecessary changes or over-editing. Therefore, increasing K is not automatically better. The team must compare the additional pass rate with the cost and the risk of changing an image too much.

This is why the previous chapter describes K as a bound. The system should retry only within a defined limit. If an image still does not pass within that limit, the safer outcome is to stop enhancing it rather than force a failed result toward publication. The exact numeric bound and the resulting coverage values are not stated in the source.

The larger lesson

This example captures the role of a bounded QA loop in a production multimodal agent:

  1. Generate an image-specific result.
  2. Check it against meaningful criteria.
  3. Explain the failure through QA feedback.
  4. Use that feedback to revise the result.
  5. Stop after a bounded number of attempts.
  6. Publish only a result that passes the required gates.

A single successful second attempt is only an illustration of the mechanism. The production objective is broader: measure how often cases pass at each allowed iteration, preserve food and presentation fidelity, and choose an iteration bound that balances quality, coverage, compute, and latency.

Source visuals

Generation Evals slide showing a sweet-potato-fries input, a failed first iteration, and a passed second iteration after a QA suggestion.

The slide visibly presents the requested sweet-potato-fries QA example as a before-and-after sequence: the first generated result is labeled QA FAIL with portion-size and plating issues, while the second is labeled QA PASS and accompanied by a QA-suggestion-applied label.

Source at 14:25
100% Space + drag to pan | Ctrl/Cmd + wheel to zoom