evaluate-intent-routing

Evaluating intent routing

The number that has misled everyone

Training reports a weighted F1 computed on a random split of the same synthetic utterances the model trained on. Every utterance for a command comes from a handful of personas expanding one seed list, so the "test" rows are near duplicates of the training rows. That number measures memorisation. The measured gap on a 160-command workflow: ~0.94 reported F1 against 46.2% held-out top-1.

It is still reported, named in_distribution_f1 so it can no longer be mistaken for a generalisation measure, and printed with a footer saying so. Judge models on top-1, in-list and escalation recall. Never quote in_distribution_f1 as accuracy.

Two axes, never blended

Routing and escalation trade against each other, so one blended score would hide the trade.

Routing — did the classifier name the right command?

• top-1 — the expected label came back as the single, confident answer. This is the only outcome that is a correct route.
• in-list — the expected label appears anywhere in the returned candidates. At runtime that is a clarification prompt, not a route. It is a real outcome worth tracking, and it is not a win.

Escalation — did the classifier correctly say "this command lives upstairs"? Scored as recall only. Correct only when the escalation label comes back alone and confident, because only a lone escalation label makes the runtime walk the parent chain. An escalation label returned beside local candidates takes the ambiguity branch and the signal is silently discarded, so counting it would report behaviour the runtime does not have.

Two populations, and only one is comparable across runs

Population	Source	Comparable across runs?
persona holdout (routing, holdout_escalation)	whole personas reserved from the generated utterances	No. The split is re-drawn every run, so two runs score different cases.
benchmark (benchmark_routing, escalation)	<workflow>/intent_benchmark.json	Yes. The file is fixed and its cases pair by construction.