substack-saved-mcp CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code when working with this repository.

Commands

Use uv for all development commands. Set up the environment with:

uv sync --extra dev

Task	Command
Run all tests	uv run python -m pytest
Run one test file	uv run python -m pytest tests/test_database.py
Run one test	uv run python -m pytest tests/test_database.py::test_fts5_search
Build the package	uv build
Run the CLI	uv run substack-saved-mcp --help
Reinstall tool globally	uv tool install . --no-cache --force

ruff is configured in pyproject.toml (rules: E4, E7, E9, F, I, UP, B, RUF, line length 88, target py311). Lint with uvx ruff check . or uv run ruff check ..

After any change to dependencies or tracked files, run uv lock to keep uv.lock current, then prek run --all-files (using the repo's .pre-commit-config.yaml) before considering the work done.

Architecture

• cli.py is the Click entry point (substack-saved-mcp). Its commands initialize the database and then delegate to the repository, sync engine, Playwright client, or MCP server.
• mcp_server.py exposes the same application operations as FastMCP stdio tools and resources. Read tools query the local cache; sync and save/unsave operations use the authenticated browser session.
• substack_client.py owns Playwright authentication and remote Substack interaction. login is the only intended headful workflow; normal sync and write paths use storage_state.json headlessly. Synchronous Playwright API calls are routed through _run_playwright_sync() to safely execute in a worker thread if an asyncio event loop is active (e.g. under FastMCP). Saved-post fetching prefers the reader inbox API (GET /api/v1/reader/posts?inboxType=saved), which exposes the real bookmark timestamp (saved_at) and an ISO publication date (post_date) per post. _fetch_all_saved_via_reader_api() cursor-paginates that endpoint (each page's oldest saved_at becomes the next after= cursor, until more is false), dedupes by canonical URL, and enriches each flat post with its publication object (from the response's publications array, matched by publication_id) and author_name (from publishedBylines). The full result is cached in _api_cache and sliced by offset; _fetch_saved_posts_page_impl() accepts an optional playwright_instance (same pattern as the notes fetcher and _fetch_via_dom) so this caching/slicing/fallback branch is directly testable with a Playwright double instead of only reachable through a real browser. Each page request goes through _reader_api_get(), which retries transient failures (HTTP 429 and 5xx) up to max_retries (default 3) with Retry-After-aware backoff: _retry_after_seconds() honors an integer Retry-After header (clamped to a 30s cap so a hostile value can't hang the sync) and otherwise uses capped exponential backoff (0.5s, 1s, 2s, ...); 401/403 and other 4xx are returned unretried so the existing auth/fallback handling applies. A 429 that survives all retries is deliberately treated as "unavailable" (partial list, or None → DOM fallback) rather than as a silent empty-success, so rate-limiting never masquerades as "you have no saved posts." Both _reader_api_get() and _fetch_all_saved_via_reader_api() accept an injectable sleep_func (default time.sleep) so tests exercise the backoff without real delays. If that endpoint is unavailable, it falls back to headless DOM extraction on https://substack.com/saved, which scrapes div.reader2-post-container cards, caches scrolling results in _dom_cache, and marks each dict with _dom: True; DOM cards only expose a localized relative publish string and no bookmark time, so saved_at stays unknown on that path. Both save_post() and unsave_post() were originally guesswork (an unverified DOM button selector plus an unverified POST /api/v1/bookmark call) and were fixed once real endpoints were captured via inspect-network. Every post page server-renders a window._preloads blob containing that post's own numeric ID and rich metadata (preloads.post.id, .title, .audience, .description/.subtitle, .post_date, plus preloads.pub.name); _save_post_impl reads this via page.evaluate("() => window._preloads") right after page load and, when an ID is found, calls the real POST https://substack.com/api/v1/posts/saved endpoint (body {"post_id": ...}) directly via the Playwright API request context — this also lets it populate the returned SavedPost with accurate title/publication/audience/excerpt instead of parsing page.title(). _unsave_post_impl does the mirror image: when the post's substack_post_id is already known (true for any post that has been through a normal sync, since the reader API's id field populates it), it calls DELETE https://substack.com/api/v1/posts/saved with the same body shape, no DOM interaction at all. Both treat an ok response as "confirmed" and only fall back to the old best-effort DOM click (_click_bookmark_toggle(), used when the numeric ID can't be obtained or the direct call doesn't confirm) — note its selector is English-only (aria-label*='save'/'bookmark') and can silently fail on non-English Substack UIs, which is part of why the direct API path is preferred whenever possible. _click_bookmark_toggle() fingerprints the button's aria-label/aria-pressed/class before and after the click and returns "confirmed" only if that fingerprint changed, else "unconfirmed", "not_found", or "click_failed" — save_post() returns (SavedPost, confirmation) and unsave_post() returns just the confirmation string; both accept an optional playwright_instance for test injection (same pattern as _fetch_via_dom). fetch_post_content() reuses the same window._preloads mechanism to retrieve a saved post's full content: _fetch_post_content_impl navigates to the post's page and reads preloads.post.body_html (Substack's field name for full content — parse_remote_post() already expects this key from the reader API, though the saved-list payload never actually populates it, only individual post pages do), returning None for body_html if the page's embed format doesn't expose it (frontend change) or the account lacks paywall access; this is the case where the caller should be told to run inspect-network against an open post page to re-discover the real content source. cli.py's inspect-network command (which logs any api/v1/bookmark/saved/notes?/comment/reader/feed/restack request's method, URL, status, and JSON response body) is the tool for discovering/re-verifying these endpoints when Substack's frontend changes. It supports --authenticated/--anonymous (reuses storage_state.json via get_storage_state_path() so /saved renders the real logged-in page instead of the marketing page), --url to target any page, --filter to override the response-matching regex, --max-body to cap logged response bytes, and --out PATH to append each exchange as JSON Lines for later grepping. Response bodies are captured via context.route("**/*", handle_route) + route.fetch()/route.fulfill(), not the more obvious page.on("response", ...) + response.text(): the latter can deadlock calling response.text() inside a sync-API response event handler on the same driver thread, and in practice silently produced zero captured bodies across an entire 108-request session before this fix (every response_body came back null with no error surfaced) — route.fetch() reads the body outside that handler and route.fulfill(response=...) re-serves the exact response so page behavior is unaffected; a body that still can't be read now logs a visible yellow warning instead of writing null silently.