paper-preprocess

Paper Preprocess Skill

Preprocess CNS papers using HTML (text/links) and PDF (figures) to extract structured data.

Input Requirements

Before invoking this skill, provide:

1. Input Directory: A directory containing both *.pdf and *.html files for the same paper (Nature/Springer structured HTML). The folder name is used as paper_id.
2. Output Directory: The base directory to store preprocessing results

Workflow

Step 1: Extract Text (from HTML)

Use scripts/extract_text.py to extract full text from the HTML:

• Preserve document structure (headings, paragraphs, bold/italic)
• Preserve LaTeX formulas
• Remove citation reference numbers
• Skip non-content sections (References, Recommendations, Author information, etc.)
• Output in Markdown format

Step 2: Extract Figures (from PDF)

Use scripts/extract_figures.py to extract figures from the PDF:

• Detect figure/table captions via structured text blocks
• Render full pages containing detected figures/tables
• Fall back to embedded image extraction if no captions found, then to full-page rendering if still empty

Step 3: Extract Links (from HTML)

Use scripts/extract_links.py to extract links from the HTML:

• Extract hyperlinks from article body sections
• Classify into data_availability / code_availability / supplementary_information / other by section
• Identify common link types (GitHub, Zenodo, CodeOcean, etc.)
• Filter out internal anchors, citation references, and non-content sections

Usage

cd skills/paper-preprocess/scripts

python3 preprocess.py <input_dir> <output_dir>

# Example:
python3 preprocess.py ./s42256-019-0037-0/ ./s42256-019-0037-0/preprocessed

Output Structure

{output_dir}/
  text.md                 # Full paper text (Markdown format)
  figures/                # Figures directory
    Fig_1_page2.png
    Fig_2_page3.png
    ...
  tables/                 # Tables directory
    Table_1_page3.png
    ...
  links.json              # Extracted links with section classification