data-scraper-agent

Data Scraper Agent

Build a scheduled agent that collects public data, enriches it with an LLM, stores it, and learns from the user's accept/reject decisions — without requiring paid infrastructure.

When to Activate

• User wants to scrape or monitor a public website or API on a recurring schedule
• User says "build a bot that checks...", "monitor X for me", "collect data from..."
• User wants to track jobs, prices, news, repos, sports scores, events, or listings
• User wants automated collection that improves over time based on their decisions

The Three-Layer Architecture

COLLECT → ENRICH → STORE
scraper     LLM      database
on a      scores/  (Notion/Sheets/
schedule  classifies  Supabase/etc.)

Two principles make this survive free-tier limits:

1. Batch every LLM call. Never call the model once per item — batch 5+ items into a single call. Scraping 30 items with one call each burns a rate limit instantly; batching the same 30 items into 6 calls of 5 stays inside almost any free tier.
2. Cascade through a model fallback chain. On quota exhaustion (HTTP 429) or an unavailable model (404), fall through to the next cheaper/faster model in a pre-ordered list rather than failing the run. Order fastest/cheapest-and-most-available first, since free tiers usually grant the smallest model the highest rate limit.

Workflow

Step 1: Understand the goal

Ask (or infer from an unambiguous request):

1. What to collect — URL / API / RSS / public endpoint?
2. What to extract — which fields matter (title, price, URL, date, score)?
3. Where to store — Notion, Google Sheets, Supabase, or a local file?
4. How to enrich — should the LLM score, summarize, classify, or match each item against user context?
5. How often — hourly, daily, weekly?

Common shapes to recognize: job boards scored against a resume, product prices with drop alerts, GitHub repos summarized on new release, news classified by topic/sentiment, sports results tracked in a running table, event listings filtered by interest.