Pairwise comparison: deciding what counts as better

An enhancement is not successful merely because it produces a new image. The new image must be better than the original in the ways that matter for the product. This chapter introduces pairwise comparison: the evaluator looks at an input image and its output image together, then decides whether the output is better.

Source focus (14:49–15:47): The presenters describe an enhancement evaluator that compares an input with its output. Its definition of quality is aligned with product, design, policy, and legal stakeholders. The named dimensions include faithfulness, completeness, naturalness, and realism. The evaluator returns yes, no, or unsure. The detailed implementation is proprietary and is intentionally not described.

The basic question is relational

An isolated score asks:

“How good is this output image?”

A pairwise evaluator asks:

“Compared with the input image, is this output better?”

The second question fits an enhancement system more directly. The system starts with an existing image. Its job is not to create an attractive image in the abstract. Its job is to improve that particular image without losing what the source already contains.

The evaluator therefore receives two related pieces of evidence:

1. the input or original image, and
2. the output or generated image after enhancement.

It then judges the relationship between them. A change can be visually noticeable without being an improvement. Conversely, a useful improvement may be small, such as making the subject easier to see while preserving the original dish and its presentation.

Click to enlarge

At about 15:04, the visible slide supports this input-versus-output framing. It describes pairwise comparison for editing or enhancement, but it does not show specific food-image examples or the evaluator's detailed implementation.

“Better” is a product definition

The word better sounds objective until a team must decide how to measure it. A brighter or more polished image might look better to one reviewer while seeming less trustworthy to another. An edit might improve composition but change the food so much that it no longer represents what the merchant sells.

The presenters address this problem by aligning the quality definition with several kinds of stakeholders. The following descriptions are teaching context for the stakeholder labels; the source does not spell out each group's process:

• Product helps define the customer and marketplace outcome the system should support.
• Design helps define the intended visual experience.
• Policy helps define behavior that is acceptable and safe.
• Legal helps define constraints the system must respect.

This means the evaluator is not simply rewarding visual polish. It encodes a shared product decision about the behavior of an enhancement system. The stakeholders' requirements become evaluation criteria, and those criteria determine what counts as an acceptable improvement.

The source does not give the complete rubric, weights, thresholds, or decision logic. The important design principle is that quality is specified before it is measured. Otherwise, an evaluator may reward an easy-to-measure change that the business or its users do not actually want.

Four dimensions that should not be collapsed

The presenters name four example dimensions: faithful, complete, natural, and realistic. These words are related, but they do not mean the same thing.

Dimension	Useful question	What can go wrong if it is ignored?
Faithfulness	Does the output remain true to the source image and its content?	The edit may invent or alter something that was not there.
Completeness	Did the output preserve the important content from the input?	The edit may remove or hide part of the original scene.
Naturalness	Does the visual treatment look like a normal, believable edit rather than an artificial effect?	The result may look overprocessed or obviously generated.
Realism	Do the objects and scene remain physically and visually plausible?	The result may contain incoherent shapes, relationships, or details.

The table gives a teaching distinction for the terms. The talk names the dimensions but does not disclose their exact operational definitions. In practice, one output can perform differently on each dimension. For example, an image might look more natural after color correction but be less faithful if the correction changes the appearance of the food. Another output might preserve every source object yet still look physically unrealistic.

This is why a single undifferentiated “aesthetic quality” score is not enough for the stated goal. It could hide a serious failure behind an improvement in surface appearance. Pairwise evaluation keeps the original available as a reference while the rubric checks several properties of the change.

The three-way result preserves uncertainty

The evaluator's result is not limited to a forced binary choice. It can return:

• Yes: the output is judged better than the input under the defined criteria.
• No: the output is not judged better than the input.
• Unsure: the available evidence is not sufficient for a confident decision.

The unsure state is important. An evaluator may be unable to determine whether a change preserved a detail, whether an object is physically plausible, or whether the output is genuinely more natural. Converting that case into an arbitrary yes or no would hide the evaluator's lack of confidence and make the resulting metric look more certain than the evidence supports.

Keeping uncertainty visible does not by itself specify the release policy. A downstream system still needs to decide how to handle unsure cases. The point here is narrower: the evaluation record should distinguish a negative judgment from an unresolved judgment.

How this differs from neighboring checks

Pairwise improvement is one evaluation question in a larger system. It should not be confused with the router or with the final publication gate.

Check	Question it answers	What it evaluates
Router evaluation	“Did the system choose the appropriate branch?”	Whether to enhance, skip, or select among model routes
Pairwise enhancement evaluation	“Is this output better than its input?”	The quality of the change relative to the original
Final publish-ready QA	“Is this result acceptable to release?”	A later, holistic quality and policy decision

For example, a router can correctly decide that an image needs enhancement, while the editor still produces an output that is not better. The routing metric would not detect every editing failure. In the same way, a pairwise evaluator can judge the improvement relationship without replacing the final policy and publication checks. Each check protects a different boundary in the workflow.

A small teaching example

Consider an original food image that is dark but clearly shows the dish. An editor produces a brighter result. Now compare the two images across the dimensions rather than asking only whether the result looks more polished:

1. Did the brighter result preserve the dish and its visible contents? This tests faithfulness.
2. Did it retain details that were visible in the original instead of cropping or covering them? This tests completeness.
3. Does the lighting look believable rather than artificially exaggerated? This tests naturalness.
4. Do the objects, edges, and spatial relationships remain plausible? This tests realism.

If the evidence supports improvement across the agreed criteria, the pairwise result can be yes. If the edit introduces a clear harmful change, it can be no. If the images or accompanying information do not allow a confident comparison, it can be unsure. This example is a teaching illustration, not a result reported by the presenters.

The central idea

Pairwise comparison turns “make the image better” into an explicit, source-grounded question: compare the original and the enhanced result, then judge the change against a quality definition agreed across product, design, policy, and legal concerns. Faithfulness, completeness, naturalness, and realism expose different ways an edit can succeed or fail. The yes/no/unsure outcome preserves the difference between a supported judgment and a case that needs caution.

The detailed evaluator remains proprietary, so its prompts, schema, thresholds, and implementation are not specified here. What the high-level design makes clear is the role of this evaluator in the closed loop: it supplies evidence about whether enhancement produced a meaningful improvement, while routing and publish-ready QA continue to answer their own separate questions.