Ratatoskr-mcp: Instructions file for Codex

Project: Ratatoskr-mcp

What this is

An MCP (Model Context Protocol) server that retrieves Wikipedia articles from local multistream dump files. No database, no ingest step, no internet required at query time. Named after Ratatoskr, the squirrel of Norse mythology who runs up and down Yggdrasil carrying messages.

Objective

Sub-3-second response time for article retrieval. Currently hitting 0.3–0.6s.

Architecture (4 files, flat — no subpackages)

• server.py — MCP server entry point using mcp.server.mcpserver.MCPServer with configurable transport. Exposes one tool: get_wikipedia_article(topic).
• index.py — Flat-file index lookup. find_page(title) -> (byte_offset, stream_id, title). Uses ripgrepy for 0.2s case-insensitive search of the 1.2GB multistream index, rapidfuzz for fuzzy fallback.
• retrieval.py — Orchestrates: find_page → seek bz2 block → decompress → parse → follow redirects → assemble Markdown output.
• parser.py — XML parsing (lxml) + wikitext parsing (mwparserfromhell). Extracts title, lead section, infobox. Detects <redirect> elements.

Data flow

1. Index lookup (index.py): ripgrepy searches the plaintext index file (byte_offset:stream_id:title format) for the query. Exact case-sensitive match → exact case-insensitive match → fuzzy match (cluster hits by byte_offset, score each cluster with rapidfuzz word-boundary scoring).
2. Block fetch (retrieval.py): seek to byte_offset in the bz2 dump, read 32MB chunk, decompress single bz2 block.
3. XML parse (parser.py): find the matching <page> element (two-pass: case-sensitive then case-insensitive, to avoid grabbing redirect pages with different casing).
4. Redirect (parser.py/retrieval.py): if <redirect title="X" /> present, recurse with X (max depth 5, loop detection via visited-title set).
5. Extract (parser.py): parse wikitext with mwparserfromhell. Extract infobox template, then lead section via get_sections(include_lead=True, flat=True).
6. Output: # Title\n\n{body}\n\n---\n\n{infobox} — body first (lead section only), infobox after as reference.