detect-duplicate-capabilities

Detecting duplicate capabilities

Some workflows expose the same capability twice: ControlsMonitor/list_findings and Directory/search_control_findings answer the same question. Others hold legitimate neighbours or opposites whose seed lists do not yet express the distinction. Both shapes present identically — as benchmark failures — and the fix for one is the opposite of the fix for the other. These two scans exist to tell them apart.

Neither scan changes what is trained, which labels exist, or what any model predicts. Neither blocks a run. They report.

The two instruments

	Pre-flight lexical scan	Post-training router scan
Runs	before anything costs money	after models are trained and carried forward
Input	hand-written seed utterances	seed utterances + the trained CommandRouter.predict
Needs	no LLM key, no network	the models this run just produced
Asks	"does the training data separate these?"	"does the model actually separate these?"
Blind to	a duplicate pair sharing no distinctive vocabulary	nothing lexical, but only sees pairs it was run on
Output	___command_info/duplicate_capabilities.json + printed report	printed MODEL CONFUSION warning, merged into the same report

They are complements, not a cheap version and a good version. Run and read both.

What separability actually measures

"Near-duplicate" is defined as a property of the training data, not of the two commands' meanings, because the training data is the only thing the classifier ever sees:

Two commands are near-duplicates when a classifier restricted to that pair, trained on their own utterances, cannot tell them apart.

Made concrete as leave-one-out, balanced, nearest-centroid accuracy over the pair's utterances in TF-IDF space. Three choices in that sentence carry the whole result:

• Leave-one-out, because an utterance sitting inside its own centroid classifies itself.
• Balanced (the mean of the two per-command recalls), so chance is 0.5 regardless of how many utterances each side has.
• Document frequency computed across every command in the workflow, so shared vocabulary — "order", "my", "please", the workflow's own subject matter — is discounted automatically and only distinguishing terms carry weight. This is what stops a workflow full of modify_pending_order_address / _items / _payment from producing a wall of false positives: those share their boilerplate and differ in exactly the terms IDF promotes.