data-finder

Data Finder

Find and assess datasets for your research question. Two Explorer agents search in parallel across data source categories; an Explorer-Critic then stress-tests each candidate against the research design.

Input: $ARGUMENTS — a topic, or from spec to read the research question from quality_reports/.

Step 1: Read Research Context

1. Find the most recent quality_reports/project_spec_*.md or quality_reports/specs/*.md — extract:

   • Research question
   • Empirical strategy (DiD, RDD, IV, etc.)
   • Treatment variable (what varies)
   • Outcome variable (what we measure)
   • Controls needed
   • Time period of interest
   • Geography (national, state, county, individual)
   • Unit of observation (individual, household, firm, establishment)

2. Read references/domain-profile.md if it exists — extract the Common Datasets section (domain-specific datasets to check first).

3. If no research spec exists, extract the variables and strategy from $ARGUMENTS directly. If the request is vague, ask: "What are the treatment and outcome variables, and what empirical strategy did you have in mind?"

Step 2: Dispatch Two Explorer Agents in Parallel

Split the source categories between two Explorer agents to parallelize the search.

Explorer A — Institutional Data:

Task prompt: "You are an Explorer agent. Research question: [question].
Empirical strategy: [strategy].
Variables needed — Treatment: [X], Outcome: [Y], Controls: [list],
Time period: [period], Geography: [geo], Unit: [unit].
Domain datasets (check first): [list from domain-profile if available].

Your source categories to search:
1. Public microdata (CPS, ACS, NHIS, MEPS, SIPP, QWI)
2. Administrative data (Medicare/Medicaid, IRS, SSA, vital statistics, court records)
3. Survey panels (PSID, HRS, Add Health, NLSY97/79, BHPS/UKHLS)

For each dataset found, produce the full Explorer report format.
Follow the Explorer agent instructions."

Explorer B — Broader and Alternative Sources:

Task prompt: "You are an Explorer agent. Research question: [question].
Empirical strategy: [strategy].
Variables needed — Treatment: [X], Outcome: [Y], Controls: [list],
Time period: [period], Geography: [geo], Unit: [unit].
Domain datasets (check first): [list from domain-profile if available].

Your source categories to search:
1. International data (World Bank, OECD, Eurostat, IMF, IPUMS International)
2. Novel/alternative (satellite, web scraping, proprietary, RCT registries)
3. Any field-specific datasets not covered by Explorer A

For each dataset found, produce the full Explorer report format.
Follow the Explorer agent instructions."