Scope: a production multimodal eval problem

This chapter sets the problem boundary for “Building Closed-Loop Evals for a Multimodal Agent at Scale” by Soumya Gupta and Jai Chopra of Uber. After restarting the introduction, the presenters place the talk in a computer-vision production use case. Their focus is not only how an agent produces an image, but how to design evals and recurring eval loops that show whether the agent is doing the intended job.

The central question: is the agent behaving as intended?

Teaching context: what “eval” means here

An eval is a check of an agent's behavior against an intended outcome. The check might ask whether the agent made the right decision, preserved something important, or avoided an unacceptable result. The important point is that an eval defines success in terms of the system's purpose, not merely whether a model returned an output.

For example, a production image system might need to decide whether a photo should be processed at all. The relevant eval would not simply ask whether an image was generated. It would ask whether the decision matched the intended behavior for that photo.

An eval loop adds time and feedback to this idea:

1. Measure the agent on cases that matter.
2. Compare its behavior with the intended outcome.
3. Feed the evidence back into the system or its configuration.
4. Run the check again after an iteration.

A one-time offline test gives a snapshot. An eval loop is a recurring process for finding problems and improving the system. This distinction matters because a model that passes a fixed test today can still behave poorly when its inputs, users, or operating environment change.

Why production changes the evaluation problem

The following explanation makes the production implications explicit; it is teaching context built from the problem framing, not an additional quotation from the presenters.

A toy image demo can be judged by looking at a few appealing outputs. A production computer-vision system has a wider obligation. It must handle many real inputs, make decisions consistently, and provide enough evidence to understand failures. It also needs safeguards for behavior that should not reach users.

That makes several concerns part of the evaluation problem from the beginning:

• Scale: the process must work across a large and varied stream of cases, not only a hand-picked example.
• Safety: the system needs checks that can stop an unacceptable result.
• Observability: the team needs records of what happened so it can diagnose a case instead of seeing only the final image.
• Feedback: failures and new evidence must be able to influence later iterations.

These are connected. If a system operates at scale but does not record its decisions, the team cannot reliably locate a failure. If it can detect failures but has no feedback path, the same failure can return. If it improves outputs without safety checks, a higher apparent quality can still produce an unacceptable production outcome.

The roadmap for the rest of the lesson

To make the framing concrete, the lesson follows a representative production workflow. The later stages will cover:

1. Routing: understand the input and decide whether it should follow an enhancement path or be left alone.
2. Editing: improve an image when enhancement is appropriate.
3. Quality assurance: check the result against the requirements before it can proceed.
4. Continuous learning: use evaluation evidence and feedback to tune the system over time.

This is a roadmap, not a claim that every stage is the same model. A useful production design separates the responsibilities: deciding what to do, performing the action, checking the result, and learning from evidence are different jobs even when they are coordinated as one agentic system.

The key idea to carry forward is therefore broader than “make the image better.” The production question is:

Can the system make the right choice, produce an acceptable result, detect when it has failed, and use that evidence to improve safely?

The next chapters apply this question to a real marketplace setting, where visual quality is tied to heterogeneous inputs, operational constraints, and user experience.

Source boundary

The source segment for this chapter runs from approximately 00:19.75 to 00:56.31. The opening transcript contains unclear speech that appears to refer to “evals” and “eval loops”; that interpretation is based on the surrounding context and is not treated as an exact quotation.