This chapter looks at how to evaluate a router that decides whether to send an image for enhancement or keep the original, treating the router as a classifier. It also explains how to measure whether the correct model is selected when the enhancement destination branches to several models.
話者の主張：ルーターは混同行列（confusion matrix）を使って、真陽性・偽陽性などのケース、適合率（precision）、再現率（recall）で評価できます。複数のモデルの分岐へ送る場合は、n × n の行列によって、各分岐を正しく選べたかを調べられます。そのうえで、コスト、遅延、出力品質のトレードオフも見ます。
Speaker's claim: A router can be evaluated with a confusion matrix, true and false cases, precision, and recall. When images are sent to several model branches, an n × n matrix can show whether each branch was selected correctly. The evaluation also considers tradeoffs among cost, latency, and output quality.
The router's first job is to decide whether to send an image for enhancement or skip enhancement and keep the original. This can be viewed as a classification problem in which one of two classes is selected. It is important to separate evaluation of the router's decision from evaluation of whether the enhancement model later produced a good edit.
Supplement (example for explanation): Below, we call an image that truly should be enhanced positive, and an image whose original should be kept negative. This is a choice for explaining the mechanism. The speakers do not disclose the actual class names, thresholds, or decision procedure.
混同行列でルートの当たり外れを見る
Use a confusion matrix to see which routes are right or wrong
For explanation, let the rows represent the "appropriate route" determined by human evaluation or another reference, and the columns represent the "predicted route" chosen by the router.
TP (true positive): An image that needs enhancement was sent for enhancement. The router's decision is correct.
FN (false negative): An image that needs enhancement was skipped. A needed improvement was missed.
FP (false positive): An image that does not need enhancement was sent for enhancement. This uses unnecessary computation and may degrade the original image.
TN (true negative): An image that does not need enhancement was skipped. The original can be kept safely.
This matrix does not only produce one overall "accuracy" number. It separates the kinds of mistakes that occurred. For example, many FNs suggest that poor images may be passing through without enhancement. Many FPs suggest that good images may be sent through an expensive enhancement process.
Precision is the proportion of images sent for enhancement that truly needed enhancement. It decreases as FPs increase. It is therefore useful for seeing how well the system avoids unnecessary enhancement work.
Recall is the proportion of images that truly needed enhancement that were actually sent for enhancement. It decreases as FNs increase. It therefore measures how well the system avoids missing images that should be enhanced.
These two measures cannot be substituted for each other. For example, sending almost every image for enhancement may reduce missed targets and raise recall. However, it also sends many images that do not need enhancement, so precision may fall. Conversely, selecting enhancement targets very strictly may raise precision while increasing FNs and lowering recall. Which measure matters more is decided by operational goals such as compute cost, quality, and safety.
The figure separates binary-routing outcomes into TP, FN, FP, and TN. Its footer also shows the connection to classifier-style precision and recall. In other words, it visually confirms the idea of measuring the router's destination decision separately from the downstream editing result.
The router's destinations are not necessarily only "enhance" and "skip." For example, images may be routed to a lightweight model, a standard model, or a high-quality model. In that case, a correct decision includes more than deciding whether to process the image. It also includes choosing the model branch appropriate for that image.
補足（説明用の例）：分岐が三つある場合、行を適切なモデル、列を選択されたモデルとする行列は次のようになります。一般には n × n です。
Supplement (example for explanation): If there are three branches, the matrix below uses the appropriate model for its rows and the selected model for its columns. In general, it is an n × n matrix.
The cells on the diagonal are cases in which the appropriate model was selected. The cells off the diagonal are cases in which another model was selected. This matrix shows not only the total number of correct decisions, but also which models were confused with which others. The speakers do not specify the actual branch names or the concrete rules for deciding the appropriate branch.
正解だけでなく、コスト・遅延・品質も見る
Consider cost, latency, and quality—not only correctness
A lightweight model may reduce compute cost and shorten response latency. This may improve the user experience. On the other hand, its output quality may be lower than that of a high-quality model. Choosing a high-quality model may help quality, but it may also increase cost and latency.
For this reason, evaluating multi-branch routing is not enough if we only maximize the diagonal of the matrix. While checking whether the correct branch is selected, we also look at questions such as these:
その分岐の選択は、必要な品質を保てていますか。
計算コストは許容できる範囲ですか。
遅延は利用者の体験を損なわない範囲ですか。
Does the branch choice preserve the quality that is needed?
Is the compute cost within an acceptable range?
Is the latency within a range that does not harm the user experience?
Here too, it is important not to confuse a routing error with a downstream failure. If the router sends an image that should go to the high-quality model to the lightweight model, that is a branch-selection error. If the lightweight model then fails to edit the image well, that is an editing or QA failure. Both can happen in one case, but they should be recorded separately when investigating the cause.
この評価で分かることと、まだ分からないこと
What this evaluation tells us—and what it does not
混同行列、適合率、再現率、n × n 行列は、ルーターがどの行き先を選んだかを整理する道具です。しかし、これだけで編集後の画像が良いとは言えません。編集の忠実さ、見た目の品質、QAの合否は、後段の評価として別に確認する必要があります。また、この講演では、クラス名、しきい値、適合率と再現率の正確な計算方法は示されていません。ここでの定義や表は、話者の主張を理解するための標準的な説明と具体例です。
A confusion matrix, precision, recall, and an n × n matrix are tools for organizing which destination the router selected. They do not by themselves show that the edited image is good. Faithfulness of the edit, visual quality, and QA status must be checked separately as downstream evaluations. The talk also does not give the class names, thresholds, or exact calculation conventions for precision and recall. The definitions and tables here are standard explanations and concrete examples used to clarify the speakers' claim.
まとめ
Summary
二値ルーティングでは、改善とスキップの判断を混同行列で分解します。
適合率は改善へ送った結果の正確さ、再現率は改善対象を取り逃がさない度合いを見ます。
多分岐ルーティングでは、n × n 行列の対角線が正しいモデル選択を表します。
実運用では、正解率だけでなく、コスト、遅延、品質のバランスを評価します。
ルーターの判断と、後段の編集・QAの結果は、別の評価として追跡します。
In binary routing, break down the enhance-versus-skip decision with a confusion matrix.
Precision measures the correctness of enhancement selections, while recall measures how well enhancement targets are not missed.
In multi-branch routing, the diagonal of an n × n matrix represents correct model selection.
In production, evaluate the balance among cost, latency, and quality, not only correctness.
Track the router's decision separately from the downstream editing and QA results.