dsh-topic-curator

DSH Topic Curator

Keep data/repositories.csv, data/dsh-plugin-topic-candidates.csv, data/audit-results.csv, and data/verified-plugins.csv in sync with the GitHub dsh-plugin topic, adding only genuine, qualifying extensions and respecting the star floor.

When to use

• The user asks to scan / refresh / sync the dsh-plugin GitHub topic into the catalog.
• The user reports "the topic has new repos" and wants them vetted and merged.
• Periodic catalog maintenance ("继续扫", "再扫一轮").

Workflow

1. Collect topic repos (gh Search API, paginated)

• The dsh-plugin topic exploded to ~10,600 repos (2026-08; 9923 of them were pushed in 2026-08 alone). It is far too big for WebFetch pagination (~260 pages).
• Search API hard cap: every query returns at most 1000 results (page=11 → HTTP 422 "Only the first 1000 search results are available"). A single sort=stars query therefore only ever sees the top-1000-by-stars — missing the ~9,900 long-tail repos where genuine new small plugins hide.
• Full-coverage scan (recommended): use scripts/full_topic_scan.py. It partitions the topic into disjoint sub-queries (star tiers for >10, then the stars:<=10 remainder split by created: year through 2024 and month from 2025 onward — single-sided created:YYYY / created:YYYY-MM; an overflowing period is split into its days and a day into 24 adjacent hour buckets) so every leaf is fully retrievable, then dedupes by repo name. Outputs both a rich JSON and a diff_topic.py-compatible {name, stars} list.
• python3 scripts/full_topic_scan.py --dry-run → print the partition plan + counts.
• python3 scripts/full_topic_scan.py --execute → fetch everything (rate-limited, ~9 min; run Bash with dangerouslyDisableSandbox: true).
• python3 scripts/full_topic_scan.py --execute --actionable-only → fetch only the six disjoint >10-star tiers used by the daily curator run.
• --only ">500,11..20" / --since 2026-06 to scope a run.
• Writes lean {name, stars} arrays to /tmp/dsh_topic_full.json and /tmp/dsh_topic_repos.json (compat).
• Writes /tmp/dsh_topic_scan_summary.json; continue only when coverage_ok is strictly true and both failures and over_cap are empty. The summary is invalidated at the start of every run, so never trust an older file.
  • GOTCHA (measured 2026-08-23): day-level created:A..B is END-INCLUSIVE — created:2026-08-13..2026-08-14 = 522 + 1702 = both days. Never write a day range with a different END day; query a single day as D..D, a month as single-sided YYYY-MM, and hours as adjacent buckets T00..T01, T01..T02, ....
• Quick top-1000 only (legacy): if you only want the popular slice, gh api "search/repositories?q=topic:dsh-plugin&sort=stars&order=desc&per_page=100&page=N" --jq '.items[] | {name: .full_name, stars: .stargazers_count}' for N=1..10, then jq -s . > topic_repos.json. The diff script's star floor makes deeper pages irrelevant for curation.
• Dedupe by lowercased name (the topic occasionally repeats a repo).