proofworks-verifier

Proofworks Verifier Skill

A verify-and-backfill loop for research output. When a main agent produces a cited answer or report, this refines it: extract each claim and its citation, fetch every source, check each claim against its source (literal first, then judged), and backfill an extra source for any claim that ends up unsupported. Runs 100% client-side — nothing leaves your machine.

It does NOT search the literature for you. Its job is to make the citations you already have trustworthy, and to find a missing source when one is needed.

Comes with two stdlib Python helpers in scripts/ that do the parts an LLM cannot be trusted to do by hand: fetch_source.py (download + strip a URL to text) and presence_match.py (decide deterministically whether a quoted number/phrase literally appears in a source).

When to Use

• A main agent produced a research draft, report, or answer with citations and you want each cited claim checked against its real source before you trust it.
• You need to prove a specific quote, statistic, or number actually appears in the page it cites (LLMs fabricate or misattribute these).
• A claim in your output points at a source that does not clearly support it and you want a better second source found and pinned.

Don't use for: open-ended literature search, finding papers on a topic (a research agent like Elicit is for that), or fact-checking a claim with no citation at all.

Prerequisites

• Python 3 (stdlib only; no pip installs needed).
• Network access to fetch the sources you cite.
• An agent with an LLM capable of reading stripped source text and judging support (any current model).

Quick Reference

# Fetch a source to clean text, then check quoted fragments against it.
python3 scripts/fetch_source.py <url> --max-chars 20000
cat source.txt | python3 scripts/presence_match.py --claim "<quote>" --claim "<number>"

Procedure

Run this loop to verify a cited draft. Track every claim through to a final tag; the draft is not done until each claim is verified, unsupported, or exhausted.