Evaluating routing with binary and multi-branch metrics

Routing is a quality decision, not only an engineering convenience. The router decides whether an image should be enhanced or left alone. If it makes the wrong choice, the rest of the pipeline starts from the wrong premise. The presenters therefore evaluate the router like a classifier, and then extend that evaluation when the router chooses among several model branches.

Source focus (07:45–08:35): The presenters describe confusion-matrix outcomes, true and false cases, precision, and recall for enhance-versus-skip routing. They then describe an n-by-n matrix for routing among multiple models, including tradeoffs among cost, latency, and output quality. The source does not specify the exact class labels, thresholding procedure, or formula conventions.

Start with the routing question

The previous stage turns the image, its description, and its metadata into a structured assessment. A rubric then supports an operational decision:

image + text/metadata
          ↓
   structured assessment
          ↓
     enhance or skip

For evaluation, we need two decisions for each test image:

1. Reference decision: What route is appropriate for this image according to the evaluation target?
2. Router decision: What route did the system actually select?

Comparing these two decisions tells us whether the router found the appropriate path. The reference decision might come from a labeled evaluation process, a rubric, or another agreed target. The exact source of that target is described in the next part of the lesson; this chapter only needs the distinction between the intended route and the selected route.

Binary routing: enhance versus skip

The simplest router has two branches:

• Enhance: send the image to the editing path.
• Skip: retain the image without enhancement.

For the following teaching convention, call enhance the positive class. This is a convention for explaining the standard metrics, not a claim about the presenters' internal label names.

Reference decision	Router selects enhance	Router selects skip
Enhance is appropriate	True positive (TP): the router sends the image to the needed path	False negative (FN): the router misses an image that should be enhanced
Skip is appropriate	False positive (FP): the router over-routes an image into enhancement	True negative (TN): the router correctly leaves the image alone

The words true and false describe whether the route agrees with the reference. They do not describe whether the later edit is visually good. A true positive can still produce a poor edit, and a false positive can send an otherwise good image to an editor that changes it unnecessarily.

Click to enlarge

At about 07:52, the slide shows the four binary outcomes as TP, FN, FP, and TN. Its footer connects the routing evaluation to classifier-style precision and recall. The slide supports the binary decision framing; it does not reveal the exact labeling or threshold procedure.

True positives and false negatives

A true positive means that the router selected enhancement when enhancement was the appropriate route. This is a successful catch. It does not yet prove that the editor will improve the image. It only says that the image entered the branch intended to handle it.

A false negative means that enhancement was appropriate, but the router selected skip. This is a miss. The image proceeds without the intervention that the reference decision called for. In the production framing of this lesson, such misses matter because a low-quality image can pass through the system unchanged.

Recall is especially useful for seeing these misses:

[ \text{recall} = \frac{TP}{TP + FN} ]

This standard formula is included as teaching context. In plain language, recall asks:

Of the images that should have been enhanced, how many did the router find?

A router with low recall is failing to cover part of the problem. Improving recall usually means catching more candidates for enhancement, but it can also increase the number of images sent downstream. The right operating point therefore depends on the quality and safety consequences of both branches.

False positives and true negatives

A false positive means that the router selected enhancement even though skipping was the appropriate decision. This is over-routing. The system spends additional compute and latency on an image that did not need the branch. Processing can also expose an already acceptable image to an unnecessary change. These are the cost and degradation risks behind selective enhancement.

Precision makes this behavior visible:

[ \text{precision} = \frac{TP}{TP + FP} ]

Again, this is the standard explanatory convention rather than a disclosed implementation detail. Precision asks:

Of the images sent to enhancement, how many were appropriate candidates?

High precision means that the enhancement budget is focused more narrowly on images that need it. Low precision means that many enhancement calls are unnecessary according to the reference. A true negative, by contrast, is a correct skip: the router avoided extra processing for an image that should remain on the original path.

Precision and recall answer different questions

It is easy to treat precision and recall as interchangeable because both use true positives. They measure different denominators and expose different operational failures.

Metric	Denominator	Routing question	Failure it highlights
Precision	All images sent to enhancement: TP + FP	“When we enhance, how often was that route appropriate?”	Unnecessary enhancement and wasted or risky processing
Recall	All images that should be enhanced: TP + FN	“How many needed enhancements did we catch?”	Bad or unsuitable images slipping through the skip path

Consider a hypothetical batch of 100 images. Suppose the reference says that 20 should be enhanced. The router sends 25 to enhancement, and 15 of those are among the 20 appropriate candidates. Then:

• TP = 15
• FN = 5
• FP = 10
• TN = 70

The precision is 15 / 25 = 60%, while the recall is 15 / 20 = 75%. The two numbers tell different stories. The router catches three quarters of the images that need enhancement, but two fifths of its enhancement calls are over-routing according to this reference. The numbers are illustrative only; the talk supplies no metric values.

The choice between these errors is not a purely mathematical preference. A false positive spends resources and may degrade a good source. A false negative leaves an image untreated. The evaluation should make that tradeoff visible instead of hiding it inside one overall accuracy number.

When there are several model branches

The decision is not always just “enhance” or “skip.” A production system may route an image to one of multiple destination models. For example, an imagined system could offer a smaller, faster model and a larger, slower model. The smaller model may lower cost and latency, while the larger model may provide a possible quality advantage. These branch descriptions are illustrative; the source does not name the actual models or their measured performance.

With two branches, a binary matrix can still be useful. With n branches, the natural generalization is an n-by-n confusion matrix:

• each row represents the reference branch that would be appropriate;
• each column represents the branch the router selected;
• diagonal cells are correct branch selections;
• off-diagonal cells are routing errors between specific branches.

For example, imagine three destination choices: Skip, FastEnhance, and QualityEnhance.

Reference \ Selected	Skip	FastEnhance	QualityEnhance
Skip	42	3	1
FastEnhance	4	35	6
QualityEnhance	2	8	19

This table is a teaching example, not data from the talk. The diagonal contains 96 correct selections. The off-diagonal cells explain the errors more precisely than one total count does. For instance, the QualityEnhance → FastEnhance cell means that an image whose reference route was the quality-oriented branch was sent to the faster branch. That may be a quality shortfall. The Skip → QualityEnhance cell means that an image that should have stayed untouched was sent to the more expensive branch. That may be an unnecessary-processing problem.

The multi-branch matrix also supports per-branch questions:

• Which branch is selected correctly most often?
• Which branch is confused with another branch?
• Is the system systematically sending difficult cases to a cheaper branch?
• Is it sending too many easy cases to an expensive branch?

The exact multi-class metric summary can vary. Teams may inspect diagonal rates, per-branch precision and recall, or cost-weighted error rates. The important source-grounded point is that an n-by-n matrix preserves which destination was intended and which destination was chosen.

Accuracy is not the whole routing objective

The presenters connect branch selection to three practical quantities:

1. Cost: a route may require more or less compute.
2. Latency: a smaller model may return sooner and improve the experience.
3. Output quality: a more capable or more expensive route may offer a quality benefit, but the tradeoff must be measured.

This means the best router is not automatically the one with the highest raw route accuracy. A routing mistake toward a slow, expensive branch has a different consequence from a mistake toward a cheaper branch that cannot meet the desired quality. A useful evaluation therefore reports route correctness together with the consequences of each branch choice.

This does not mean that teams should hide incorrect routing behind a favorable cost average. It means that the matrix and the operational measurements should be read together. A system can be cheap because it skips too often, or fast because it sends difficult cases to an inadequate model. The evaluation must make those possibilities visible.

Keep routing errors separate from downstream failures

There are at least two different questions in the end-to-end system:

1. Did the router choose the right branch?

The binary or multi-branch matrix answers this question. It compares the selected route with the reference route.

2. Did the selected branch produce an acceptable result?

Editor evaluation and QA answer this later question. They inspect the output for dimensions such as quality, faithfulness, or policy compliance.

These questions must not be collapsed. If the router correctly selects an enhancement branch but the editor produces a bad image, that is primarily a downstream editing or QA failure. If the router skips an image that should have been enhanced, that is a routing miss even if no editor had a chance to fail. The logged trace should preserve both facts so the team can fix the responsible stage.

The mental model

Think of routing evaluation as measuring the system's choice of intervention:

reference:  which path should handle this image?
router:     which path did the system choose?
                      ↓
        binary or n-by-n route metrics
                      ↓
       cost + latency + later output-quality checks

For a binary router, precision exposes unnecessary enhancement and recall exposes missed enhancement candidates. For a multi-branch router, the diagonal and off-diagonal cells show which destinations are being confused. The resulting measurements do not replace editor or publish-ready QA. They locate an earlier decision and show the resource and quality consequences of that decision.

The durable design principle is simple: evaluate the router against an explicit intended route, preserve the type of mistake it made, and measure the tradeoffs created by each branch. That gives the later tuning loop something more useful than a single “worked” or “failed” label.