strip-audit

You are a stripping-quality auditor for fire_seq_search_server's chunker. The server has a preprocess() pass (fire_seq_search_server/src/indexer/chunker.rs) that removes Logseq-specific noise — frontmatter, advanced-query blocks, key:: value property lines, and - Journal Template template parents — before chunking. Your job: run the indexer over the live corpus, sample its dump output, and identify cases where the strip is too weak (noise leaking through) or too strong (real content being lost).

The dump format, written by dump_processed_note in indexer/pipeline.rs, is one file per indexed note at <dump_dir>/<rel_path> containing three sections:

=== RAW ===
<original markdown verbatim>

=== PREPROCESSED (post-strip, pre-chunk) ===
<what the stripper hands the chunker>

=== CHUNKS (N) ===

--- chunk 0 ---
<chunk text including the "# Title\n\n" prefix>

--- chunk 1 ---
...

Diffing RAW against PREPROCESSED tells you what the stripper removed. Reading CHUNKS tells you what eventually reaches retrieval/embedding (after the chunker's is_stub_unit filter).

Procedure

1. Pick a dump dir. Default /tmp/fsq-stripped unless the user specified one. Clear it first: rm -rf /tmp/fsq-stripped && mkdir -p /tmp/fsq-stripped.

2. Force a full re-index. The pipeline's fast-path skips unchanged notes, so an audit needs every note to be re-processed. Wipe the cache: rm -rf ~/.cache/fire_seq_search/. (If the user objects to a cache wipe, they can flip CHUNKER_VERSION in store.rs instead, but the cache wipe is simpler.)

3. Boot the server with dumping enabled. From the repo root:

   FIRE_SEQ_DUMP_PROCESSED_DIR=/tmp/fsq-stripped bash tests/run_logseq.sh > /dev/shm/fsq_strip_debug.log 2>&1 &
   

   Capture the PID. Confirm in the log that the boot line FIRE_SEQ_DUMP_PROCESSED_DIR=... appeared — that's the indexer acknowledging the env var.

4. Wait for indexing to settle. Poll curl -s http://127.0.0.1:3030/server_info until indexer.in_flight is false. Cold indexing of a large notebook can take several minutes — be patient, and note the wall-clock duration in your report.

5. Inventory the dump. find /tmp/fsq-stripped -type f -name '*.md' | wc -l to confirm files were written, and a quick ls /tmp/fsq-stripped/ to see top-level structure (Logseq typically has journals/ and pages/).

6. Sample notes for inspection. Pick around 10 files — a mix of:

   • Recent journals (last ~6 months)
   • Older journals (>1 year)
   • At least 2 from pages/ if it exists (template-y, knowledge-base style)
   • At least 1 file whose RAW contains Journal Template (to verify the new strip)
   • At least 1 file whose RAW contains query-table:: or :LOGBOOK: or ((block-ref)) (suspected unstripped patterns)

   You can find candidates with quick greps over /tmp/fsq-stripped, e.g. grep -rl 'Journal Template' /tmp/fsq-stripped | head or grep -rl 'query-table::' /tmp/fsq-stripped | head.

7. For each sampled file, Read it and assess two axes against the RAW vs PREPROCESSED sections:

   • Noise (strip too weak) — Logseq syntax that's still in PREPROCESSED when it shouldn't be. Specific patterns to look for:
     • Bullet-prefixed properties: - query-table:: false, - id:: abc-123 — PROP_LINE only matches lines starting with the property name, so anything prefixed with - or * slips through. Known gap, likely to surface.
     • Org-mode logbook blocks: :LOGBOOK: ... :END: (the Benchmark example in tests/resource/journals/2022_02_26.md shows them in real notes).
     • Org-mode scheduling lines: SCHEDULED: <date>, DEADLINE: <date>, :PROPERTIES: ... :END:.
     • Block references: ((block-id)) — opaque IDs that mean nothing to retrieval.
     • Logseq macros: {{embed ((id))}}, {{query ...}}, {{renderer ...}}.
     • collapsed:: true, id:: <uuid> at any position.
     • Empty-ish artifacts: lines that are just -, * , or whitespace surviving as bullets.