dedup-skills

Dedup skills

Contract

Inputs

• Target tree path (default skills/). A prompt-directory tree may be supplied instead.
• The tree's LICENSES/NOTICE/attribution file, if present, is read in full before scanning.

Procedure

1. Vendored set. Read the tree's LICENSES/NOTICE/attribution file in full before any scan. Enumerate every vendored or license-covered path. Report findings inside vendored paths with a vendored annotation and never propose them for deletion. Exclude the attribution file itself from the scan because it is a registry, not prompt content. Done when: every vendored or license-covered path is enumerated and the attribution file is excluded from the scan.
2. Cluster pass. Run a mechanical script over all markdown in the tree. Strip YAML frontmatter first. Frontmatter is load-bearing routing data, never a dedup target, and must not enter the scan. Segment the text into paragraphs (skip fenced code), normalize tokens, and cluster near-identical spans. Verify every cluster member against its cluster center; chained transitive grouping pollutes clusters with sub-threshold members. A cluster is a span appearing in ≥2 locations cross-file or ≥2 times in one file. Threshold is verbatim/near-verbatim only (≈0.85 token-shingle Jaccard as guidance, not a rule). Each cluster carries file:line locations and a snippet. Done when: every near-identical span cluster is identified with file:line locations and a snippet, with frontmatter stripped and no transitive pollution.
3. Conflict pass. Index directive-modal sentences (must / always / required vs never / do not / forbidden) with sentence-level line attribution, not the enclosing paragraph's first line. Pair sentences that use opposing modals and overlapping content words. Each conflict candidate carries both sentences, both file:line locations, and the overlap score. Before trusting the live result, verify that the detector can fail end to end: point the scanner's root at a fixture tree containing one markdown file with a known opposing pair, run the full discovery → frontmatter-strip → pairing path, and confirm that the pair is flagged. Only then is a zero-candidate live result a real outcome rather than a dead detector. Done when: every opposing-modal pair is identified with both sentences and file:line locations, and the detector is falsifiability-verified against a fixture.
4. Judgment pass. Classify every finding from the two separate, complete sets exactly once so the finding set is MECE. The sets are repetition clusters and conflict candidates:
   • repeat — real repetition, all copies live, no sync-lineage note. Remedy: shorten in place — every copy stays where it is, compressed to its load-bearing core; no pointer consolidation, no copy deleted. Recommending a sync-lineage annotation is allowed; consolidation is not.
   • conflict — genuine opposing directives on one subject. Quote both sides verbatim from source (read each cited line in context before confirming). No default winner: the ledger proposes no resolution; the user resolves each conflict at apply time.
   • intentional-keep — documented replication (sync-lineage note, byte-duplicated-by-design header), vendored self-contained relocation, template mirroring its exemplar, or a short self-contained rule. State the reason.
   • not-a-finding — false positive (opposing modals on different subjects, scoped exceptions such as rules governing different states or routes, coincidental overlap, template scaffolding). Discharge with a one-line reason. Two forms of duplication are legitimate and never findings: a short self-contained rule repeated where needed (duplication of one short rule beats a pointer chain), and replication carrying a sync-lineage note that names its counterpart.