sparq: Skill for Claude Code

Rust parallel parsing for RDF ingest

[OPUS-4.8] Authored from this project's measured research. Verify any claim against research/custom-parsers-baseline.md and research/fast-ingestion.md before quoting it — those are the source of truth and were measured on an M1 Air.

The one rule that overrides intuition

There is no custom-parser goldmine here, and measurements have killed every "obvious" win. Before proposing a parser change, read the verdicts table in custom-parsers-baseline.md. The incumbent path is already a tuned custom parallel parser; byte-scanning is 0.4% of memory bandwidth (60 GB/s memscan vs ~234 MB/s parse). The cost is in interning + index build, not in scanning.

Measured facts to anchor on (M1, real Wikidata slice; absolute throughput numbers drift with hardware — they live in research/custom-parsers-baseline.md and the bench/parse/ harness, cite that, but the ratios below are the load-bearing point):

• Custom serial NT parse+intern is already ~1.27× faster than oxttl parse-only — it parses and interns in less time than oxttl takes to parse and throw triples away.
• Parallel NT parse+intern scales ~3.84× on 4P+4E; full ingest scales similarly.
• A zero-cost scanner would cap full serial ingest at only ~1.8×, and a realistic 2× scanner at ≤1.2× end-to-end at 8T, because intern+build are 65%+ of the parallel path. Not worth a second parser to differential-test.

Chunk-parallel scheme (the shipped design)

The pattern that works, by format:

• N-Triples / N-Quads — trivially correct. Records are newline-delimited and self-contained (no document scope). Split the byte buffer at newline boundaries into ~equal chunks, parse each chunk on the rayon pool into a per-chunk partial Dict, then merge. See sparq-core/src/nt.rs (parse_chunk) and build_external_ntriples_parallel in sparq-core/src/lib.rs. Upstream oxttl now also ships split_file_for_parallel_parsing for NT/NQ — so "parallel NT via the stock parser" no longer even needs custom code.
• Turtle — byte ranges are WRONG. Prefixes, multi-line literals, ;/, continuation, and document-scoped blank-node labels all cross arbitrary byte boundaries. You need statement-terminator chunking: scan for a . followed by whitespace/EOF/#, skipping over strings/IRIs/comments (turtle_chunks in sparq-core/src/lib.rs). A . cannot terminate a statement inside [...]/(...)/labels in valid Turtle (DECIMAL/DOUBLE need a digit after the dot; PN_LOCAL/BLANK_NODE_LABEL cannot end in a dot), so the scanner needs no bracket tracking; mis-splits on invalid input fail chunk parse → serial fallback.