know-me-better

Know Me Better

Turn an existing paper collection into a structured knowledge base under <project>/.knowledge/ (or an advisor KB). The output uses the same KB format as the survey and how-to-download-ref skills — project and advisor KBs can coexist cleanly.

Step 1 — Identify the researcher and source. First, ask whose papers to index:

"Whose papers should I index? (Give me a name, or leave blank for your own collection.)"

Then ask which source to use:

"Where are the papers?"

• (a) Zotero library
• (b) A PDF folder (give me the path)
• (c) Google Scholar profile (give me the URL)

Note: the Zotero option is only meaningful for indexing your own collection (it's your local DB). For another researcher, choose (b) or (c).

Step 2 — Index the collection.

Zotero:

1. Locate zotero.sqlite — check in order: ~/Zotero/, ~/Library/Application Support/Zotero/, ~/snap/zotero-snap/common/Zotero/. If not found, use find ~ -maxdepth 4 -name "zotero.sqlite" as fallback. If still not found, ask for the path.

2. Run the bundled script:

python3 <skill-base-dir>/parse_zotero.py <path-to-zotero.sqlite> <output_dir>

The script handles: copying the DB to avoid locking, pivot queries to avoid cartesian products, author extraction, cite key deduplication, topic classification, and generating structured output.

Important — treat <output_dir> as a scratch directory, not the KB. The script writes legacy-format index files (a topic index and a .bib file) into <output_dir>. Pick a temp path (e.g., /tmp/zotero-export-$$/). Steps 3–6 are the authoritative writes — they read those intermediate files from <output_dir> as input data, then emit .raw/{arxiv,doi}/<id>.json into $KB and append to $KB/references.bib. After Steps 3–6 finish, the contents of <output_dir> can be deleted.

1. Review the output — the script's topic classification uses keyword matching and may need manual adjustment. Check the topic distribution it prints and offer to re-classify if the user's field isn't well covered by the default patterns.

2. For papers missing abstracts or DOIs, find the PDF via the itemAttachments table. PDFs are at <zotero-data-dir>/storage/<key>/<filename>.pdf. Read them to extract the abstract.

PDF folder:

1. List all PDFs in the given path.
2. Read each PDF — extract title, authors, year, abstract, DOI/URL from the content.
3. For bulk keyword search: pdfgrep -r -i "KEYWORD" <folder> (install via package manager if missing, e.g., apt install pdfgrep or brew install pdfgrep).

Google Scholar:

Note: Google Scholar actively blocks automated access — automated fetching may hit CAPTCHAs or rate limits. If scraping fails, suggest alternatives: export BibTeX manually from the Scholar profile page (Scholar → select all → export BibTeX), use ORCID or DBLP profiles instead (both have machine-friendly APIs), or switch to the PDF folder method with downloaded papers.

1. Fetch the profile page.
2. Extract paper titles, years, citation counts.
3. For each paper, search for the DOI and abstract via web search.

Processing tips:

• Always use bundled scripts (parse_zotero.py for Zotero). Don't try to do it inline with shell commands — even for small libraries, a script is more reliable and easier to debug.
• Topic classification in the script uses keyword matching ordered most-specific-first. The default patterns cover quantum computing, physics, CS, and math. For other fields, modify TOPIC_PATTERNS in the script or ask the user to provide keywords for their domain.