sparq: Skill for Claude Code

HDT — Header Dictionary Triples

[OPUS-4.8] Authored for the active HDT work. Ground truth: the sparq-hdt crate (crates/sparq-hdt/src/lib.rs, Cargo.toml; open work in beads — bd list -l area:sparq-hdt) and the upstream hdt crate (KonradHoeffner/hdt, MIT). Verify crate API against the pinned version before writing code.

What HDT is

HDT (rdfhdt.org, W3C member submission) is the de-facto binary archive format for RDF. A single self-contained file is split into three components — hence the name:

• H — Header. RDF metadata about the dataset itself (VoID statistics, format, provenance) as plain triples. Queryable; only the head of the stream needs decoding to read it.
• D — Dictionary. All distinct terms, stored compressed. The standard layout is a FourSectionDictionary: four sections (subjects-only, predicates, objects-only, and shared subject∧object terms), each Plain Front Coded (PFC) — terms are sorted and each stores only the byte-length of its common prefix with the previous term plus its suffix. A term maps to a dense integer id per section.
• T — Triples. A BitmapTriples structure: the triples in SPO order as a bitmap-compressed adjacency list (Log64/Plain integer sequences + a bitmap marking the end of each subject's / predicate's adjacency run). Lookups use rank/select over the bitmaps to navigate from a subject id to its predicate run to its object run in O(1)-ish per step.

The payoff: files are a fraction of the size of even gzipped N-Triples, and they load without text parsing — no tokenizing, no UTF-8 re-validation per triple.

How sparq-hdt uses it (the actual integration)

The crate wraps the maintained hdt crate's reader rather than reimplementing the binary format. Key design facts (in lib.rs / Cargo.toml):

• Id-level translation. Each distinct HDT dictionary id is decompressed to its term string once, interned into the sparq Dict, and the mapping memoized in a flat per-section table. So the term set is never materialized twice and per-triple work is three array lookups (s/p/o id → sparq Id). This is the fast-decode lever: do not route through sophia terms — default-features = false deliberately drops the sophia adapter.
• .hdt.gz by magic bytes. GZipped containers are detected by magic bytes (0x1f 0x8b), not file name, in every entry point, and decompressed on the fly with a streaming MultiGzDecoder (flate2, pure-Rust backend) — same magic-byte convention as the rest of the workspace.
• Header access. header(path) / header_reader(reader) expose the dataset metadata triples as a queryable sparq Graph, decoding only the control-info + header sections (a few KB). The wrapped crate parses the header internally but keeps the field private, so these re-read those sections directly.
• Entry point: sparq_hdt::load("dataset.hdt") -> Result<Graph, Error>. Error is Io | Hdt | Term.