Building Paper Screening Rubrics

Building Paper Screening Rubrics

Overview

Core principle: Build screening rubrics collaboratively through brainstorming → test → refine → automate → review → iterate.

Good rubrics come from understanding edge cases upfront and testing on real papers before bulk screening.

When to Use

Use this skill when:

• Starting a new literature search that will screen 50+ papers
• Current rubric misclassifies papers (false positives/negatives)
• Need to define "relevance" criteria before automated screening
• Want to update criteria and re-screen cached papers
• Building helper scripts for evaluating-paper-relevance

When NOT to use:

• Small searches (<20 papers) - manual screening is fine
• Rubric already works well - no need to rebuild
• One-off exploratory searches

Two-Phase Process

Phase 1: Collaborative Rubric Design

Step 1: Brainstorm Relevance Criteria

Ask domain-agnostic questions to understand what makes papers relevant:

Core Concepts:

• "What are the key terms/concepts for your research question?"
  • Examples: specific genes, proteins, compounds, diseases, methods, organisms, theories
• "Are there synonyms or alternative names?"
• "Any terms that should EXCLUDE papers (false positives)?"

Data Types & Artifacts:

• "What type of information makes a paper valuable?"
  • Quantitative measurements (IC50, expression levels, population sizes, etc.)
  • Protocols or methods
  • Datasets with accessions (GEO, SRA, PDB, etc.)
  • Code or software
  • Chemical structures
  • Sequences or genomes
  • Theoretical models
• "Do you need the actual data in the paper, or just that such data exists?"

Paper Types:

• "What types of papers are relevant?"
  • Primary research only?
  • Reviews or meta-analyses?
  • Methods papers?
  • Clinical trials?
  • Preprints acceptable?

Relationships & Context:

• "Are papers about related/analogous concepts relevant?"
  • Example: "If studying protein X, are papers about homologs/paralogs relevant?"
  • Example: "If studying compound A, are papers about analogs/derivatives relevant?"
  • Example: "If studying disease X, are papers about related diseases relevant?"
• "Does the paper need to be ABOUT your topic, or just MENTION it?"
• "Are synthesis/methods papers relevant even without activity data?"