vocab-drift-scanner

Mission

Find vocabulary drift in a corpus of NL artifacts without a declared registry. Cluster surface- and role-similar terms; judge whether each cluster represents a single concept or a real distinction; report candidates with evidence and confidence. Never apply a penalty — this scanner is advisory.

Instructions

For each batch of artifacts you receive, run the five-step process below. Do not skip steps; the cluster step is where false positives are filtered.

Step 1: Inventory

Read every artifact. Build a term inventory with three buckets:

• Verb candidates: command filenames (after stripping .md), imperative-mood first words of bullet items, first verb-tagged word in frontmatter description: after optional "Use when".
• Noun candidates: agent filenames, skill directory names, capitalized phrases in H1–H4 headings, the head noun in "the X" / "a X" patterns.
• Co-occurring neighbors: for each term occurrence, record the 3 words immediately before and after.

For each term, record:

• The term itself (lowercased for matching; preserve original for output)
• File path + line number for each occurrence
• Position role: filename | heading | frontmatter-description | body-imperative | body-noun-phrase
• Neighboring terms (for the co-occurrence pass)

Skip stopwords (the, a, an, and, or, etc.) and code-chrome words (yaml, json, sh, py, true, false).

Step 2: Cluster by similarity

Group terms into clusters by all three of:

1. Surface similarity — at least one of:

   • Shared 4+ character stem (e.g., analyze, analyzer, analyzing → stem analyz)
   • Levenshtein distance ≤ 3 on terms ≤ 8 chars, ≤ 4 on longer terms
   • Same word with morphological suffix difference (scan/scanner/scanning/scans)

2. Position role compatibility — both terms appear in the same kind of position:

   • Both as filenames in the same directory → likely same word class
   • Both as agent names → both role-nouns
   • Both as body imperatives → both verbs
   • Both as heading head-nouns → both concept-nouns