arxiv-catchup

arXiv Catchup

Fetches cs.AI papers from arXiv published since the last checkpoint date, calls arxiv-article for each (via parallel subagents), then advances the checkpoint to today.

Prerequisites

ARXIV_DIRECTORY must be set to the directory where archived articles are stored (e.g., ~/arxiv-articles). If the variable is unset, stop and ask the user to set it.

echo "${ARXIV_DIRECTORY:?ARXIV_DIRECTORY is not set}"

State file

~/.arxiv-catchup/state.yaml

last_date: "YYYY-MM-DD"

• If the file does not exist, ask the user what start date to use before proceeding.
• The date is always in YYYY-MM-DD format.
• After all articles are processed, update the file with today's date.

Steps

1. Read the last-processed date

mkdir -p ~/.arxiv-catchup
cat ~/.arxiv-catchup/state.yaml 2>/dev/null

If the file does not exist or is empty, ask the user:

"No catchup state found. From what date should I start? (YYYY-MM-DD)"

Wait for the user's answer, then use it as last_date.

2. Fetch the catchup page

curl -L --silent --max-time 60 \
  -A "Mozilla/5.0 (compatible; research-bot/1.0)" \
  -o /tmp/arxiv_catchup.html \
  "https://arxiv.org/catchup/cs.AI/{last_date}"

If the response is empty or the status is non-200, report the error and stop without updating state.

3. Extract HTML article links

The catchup page contains three sections: New submissions, Cross-lists, and Replacements. Ignore all articles under the Replacements section.

Parse the HTML to collect links from New submissions and Cross-lists only. In the page source, replacement entries appear after a heading such as Replacements or replaced. Discard any /html/ links that appear after that heading.

Extract links following the pattern /html/{arxiv_id}:

rg -oP '(?<=href=")[^"]*' /tmp/arxiv_catchup.html \
  | rg '^/html/' \
  | sed 's|^/html/|https://arxiv.org/html/|' \
  | sort -u

If that yields nothing, also try the absolute-URL form: