WebReaper: Instructions file for Claude Code

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this is

WebReaper — declarative, parallel web scraper/crawler library for .NET, published as the WebReaper NuGet package. Library lives in WebReaper/; Examples/ and Misc/ are consumers, not packaged.

Commands

dotnet build WebReaper.sln                  # build all
dotnet test                                  # all tests (xUnit)
dotnet test WebReaper.Tests/WebReaper.UnitTests/WebReaper.UnitTests.csproj   # unit only (fast, offline)
dotnet test --filter "FullyQualifiedName~SimpleTest"   # single test by name
dotnet run --project Examples/WebReaper.ConsoleApplication   # run an example scraper

Test reality

• Unit tests (WebReaper.Tests/WebReaper.UnitTests) — offline, parse TestData/TestPage.html. Use these for fast iteration.
• Integration tests (WebReaper.Tests/WebReaper.IntegrationTests) — hit the live site alexpavlov.dev, launch real Puppeteer/Chromium, and assert after fixed Task.Delay (up to 25s). Slow and network-flaky by design; failures there are often environmental, not regressions.

Toolchain

All projects target net10.0; global.json pins SDK 10.0.100 (rollForward: latestMinor, no prerelease) and .github/workflows/CI.yml installs 10.0.x — aligned. Build with a .NET 10 SDK. CI runs dotnet restore → dotnet build → unit tests over the whole solution (Examples/ and Misc/ included), so verify with a full-solution build, not just the library project.

Architecture

Fluent builder → immutable config + pluggable spider → parallel job loop.

Build path (ADR-0025, widened by ADR-0040 + ADR-0067): a scrape begins with a Crawl seed. ScraperEngineBuilder.Crawl(urls) / .CrawlWithBrowser(urls) are static → return ICrawlSeed; the seed has four strategy terminals: .Extract(schema) (Schema-driven JSON via the deterministic fold), .AsMarkdown() (no-schema LLM-ready Markdown via MarkdownContentExtractor, ADR-0040), .ExtractInferred(goal?) (runtime schema inference via LearnedSchemaContentExtractor + ISchemaInferrer, ADR-0067), or .ExtractWithPrompt(chatClient, instruction) (schema-free LLM extraction via PromptContentExtractor, ADR-0084), all yielding the ScraperEngineBuilder. The builder's constructor is internal (test-only via InternalsVisibleTo), so it is unreachable without the seed: "build with no start URLs or no extraction strategy" is structurally impossible, not a runtime throw. It composes two internal builders:

• ConfigBuilder (internal) → immutable ScraperConfig (start URLs, the LinkPathSelector chain, crawl limit, headless flag).
• SpiderBuilder (internal) → runtime components (loaders, parsers, sinks, trackers, cookie storage), all with in-memory defaults.