In this chapter, we examine why an image-editing agent should not be judged only by whether its output looks better than the input. Even when the appearance changes, the output may add something that was not in the original. Conversely, it may remove something that was in the original. It may also satisfy only a superficial signal watched by the evaluator and make a change that is not actually useful.
Here, faithfulness means that the edited image correctly preserves the content of the original image. Not adding content that was absent from the original is part of this. Completeness means preserving important content that was present in the original without dropping it. These ideas are similar, but they look in different directions. A faithfulness failure is a problem of adding something extra; a completeness failure is a problem of removing something necessary.
Figure 1: A faithfulness failure. Shrimp that are absent from the original image on the left have been added to the edited image on the right. Even if this makes the image look more polished, it is not faithful to the original content. In the talk, this addition is shown as a faithfulness failure. Video timestamp (15:59)
This is not merely a difference in a small detail. If an ingredient absent from a food photo is added, users may misunderstand the food they will actually receive. Therefore, an editing evaluation must check both whether the image looks natural and attractive and whether its original content has been changed. This check also helps prevent the system from creating content absent from the image just to match a description or metadata at an earlier stage.
The next failure points in the opposite direction. An edited image may lose content that was present in the original. Even if its overall impression looks better, it is not complete if important objects have disappeared.
Figure 2: A completeness failure. Sauce is visible beneath the sushi on the left, but it has disappeared in the image on the right. Regardless of whether the edited image looks appealing overall, it fails to preserve the original content and is therefore a completeness failure. Video timestamp (16:10)
These two examples show why pixel differences alone are not enough. The evaluator must look at the relationship between the original and output images. It must check whether meaningful content, such as ingredients, parts of a dish, or sauce, was added or lost. In other words, an editing evaluation needs a content-aware check.
Another problem is reward hacking. This is when a system satisfies a superficial signal supplied by an evaluator while contributing little to the real goal. For example, if an evaluator treats a large raw pixel difference as evidence of improvement, the model may choose to change many pixels instead of making a meaningful improvement.
Figure 3: Reward hacking. The left image is a close-up of a spoon entering a white cup, while the right image shows a white bowl containing a swirled dessert. The appearance has changed, but the slide presents this change as having no benefit. A large pixel difference is not the same as making the image better for users. Video timestamp (16:35)
The way feedback is given also needs care. If a creative first edit is rejected as too creative, the next edit may move too far toward safety. In the talk, the result is shown as a generic ceramic bowl image that has lost the dish's individuality. Feedback intended to correct one failure can be too strong and produce an edit that is generic and not very meaningful.
The evaluation must check not only the meaning of objects but also their relationships and physical plausibility. Even frontier image models can produce images in which object consistency or spatial relationships break down. The talk says that these issues need to be handled in coordination with the teams developing such models.
Figure 4: A spatial-coherence failure. On the left, a red sauce or ketchup cup sits beside a takeout container. On the right, the plate partly hides that cup. This breakdown in the objects' spatial relationship is a problem of object consistency and physical plausibility, not merely a pixel difference. Video timestamp (16:56)
An evaluation should therefore keep at least these perspectives separate: faithfulness, asking whether the output adds objects absent from the original; completeness, asking whether it preserves important original content; meaningful improvement, asking whether the change is genuinely useful; and object consistency and physical plausibility. If these are collapsed into one score, improvement in one perspective may hide deterioration in another.
These failures do not all have the same cause. Added shrimp is a faithfulness problem, and lost sauce is a completeness problem. The cup-and-bowl example is reward hacking that exploits a superficial evaluation signal. The overly conservative bowl example is a problem in which feedback makes the edit too generic. The broken object relationship is a physical-consistency problem. Because the causes differ, evaluation and diagnosis should not rely on one failure check.
As a supplement, the talk uses the word “nugatory” to describe a change with little meaning or benefit. However, it does not give a formal definition or measurement method. It also does not disclose the names of the frontier models or the detailed tests for object consistency and physical plausibility. The important point is not to treat a visible change in the model's output as success automatically, but to check content, meaning, and naturalness separately.