find-code-duplication

TODAY=!date +%Y-%m-%d

Find Code Duplication

Detects copy-pasted blocks and near-duplicate logic across the codebase and produces a prioritized extraction plan. Duplicated code increases the cost of every future change and is a common source of bugs where a fix is applied in one copy but not others.

Prerequisites

• Working directory is the root of the repository
• Optional: $1 — path to limit the scan (defaults to .)
• Language-specific tools (degrades to heuristics if unavailable):
  • Any language: jscpd (npm install -g jscpd)
  • Python: pylint similarity checker or flake8-bugbear
  • Java/others: cpd (from PMD, brew install pmd)

What to Look For

Type	Description	Threshold to flag
Exact duplication	Identical code blocks copy-pasted verbatim	≥ 5 lines
Near-duplication	Same structure with minor variations (different variable names, literals)	≥ 8 lines
Structural duplication	Same algorithm implemented independently in multiple places	Judgment
Cross-file patterns	The same sequence of operations repeated across many call sites	≥ 3 occurrences

Steps

1. Detect Language and Available Tools

find ${1:-.} -type f | sed 's/.*\.//' | sort | uniq -c | sort -rn | head -10
command -v jscpd && jscpd --version
command -v cpd && cpd --version 2>/dev/null

2. Run Duplication Detector

jscpd (preferred — language-agnostic, handles near-duplicates):

jscpd ${1:-.} \
  --min-lines 5 \
  --min-tokens 50 \
  --reporters console,json \
  --output .jscpd-report \
  --ignore "**/*.test.*,**/*.spec.*,**/node_modules/**,**/.venv/**" \
  2>/dev/null | tail -60

Parse the JSON report for the top duplicates:

python3 -c "
import json, sys
try:
    data = json.load(open('.jscpd-report/jscpd-report.json'))
    dupes = data.get('duplicates', [])
    dupes.sort(key=lambda d: d.get('lines', 0), reverse=True)
    for d in dupes[:20]:
        a = d['firstFile']
        b = d['secondFile']
        print(f\"{d.get('lines',0):4d} lines  {a['name']}:{a['start']}  <->  {b['name']}:{b['start']}\")
except: pass
" 2>/dev/null