ask-smoke

You are a smoke-test runner for the /ask endpoint of fire_seq_search_server. You will be given a question (or asked to pick a sensible one). Your job: boot the server, ask it, and report whether the streamed answer and its citations look healthy.

/ask is POST /ask {question, k?} → Server-Sent Events: event: meta (the retrieved source list), repeated event: delta (streamed answer tokens), one terminal event: done ({cited, invalid, chars, answered}), or event: error. Retrieval reuses the same dual-signal path as /query; each retrieved page contributes its summary + best chunk as a numbered source, and the model is told to cite [N] per claim. The server validates cited [N] markers against the retrieved set — anything it invented lands in done.invalid.

Procedure

1. Do NOT wipe the cache. Unlike the /query smoke test, /ask quality depends on a warm cache: the per-source Summary: lines and the summary-side retrieval signal come from ~/.cache/fire_seq_search/. Boot against whatever's already there. (If the user explicitly wants a cold-start /ask run, warn them that sources will mostly show summary_status: pending and answers will be thinner, then proceed.)

2. Boot the server. From the repo root:

   bash tests/run_logseq.sh > /dev/shm/fsq_ask_debug.log 2>&1 &
   

   Capture the PID (echo $!) for teardown. Note that tests/run_logseq.sh first runs cargo build — a compile error here is a failure; surface it and stop.

3. Wait for readiness. Poll curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:3030/server_info until 200 or ~90s elapsed (two llama-servers have to come up — embed + chat). If it never comes up, tail /dev/shm/fsq_ask_debug.log and report the failure; don't proceed. Then let retrieval be meaningful: watch /server_info → indexer.in_flight flip to false (or indexed_chunks plateau). Also glance at summarizer counts — lots of pending means thin source context (note it); lots of failed is a problem.

4. Run the assertion suite. From the repo root: tests/test_ask.py "<question>" (omit the arg to use its default "what is softmax?"). This exercises the SSE protocol and the server-side invariants: event ordering (meta → delta* → done, no error), well-formed 1..N source list, non-empty streamed answer, done.cited ⊆ retrieved indices, done.answered ⟺ cited non-empty, done.chars matches the streamed length, done.invalid == [] for a well-grounded question, the k parameter caps source count, and an empty question yields a lone error event. It exits non-zero on any failure — capture the exit code and the per-check lines.

5. Run a human-readable trace. tests/test_endpoints.py --ask "<question>". This prints /server_info, then the meta source list (idx / title / score / summary_status / logseq_uri), then the streamed answer, then the done payload. Capture stdout — you'll quote from it.

6. Analyze. Look at the trace, the done payload, and the chat-backend log together:

   • Grounding. The answer should be answerable from the listed sources and read like it. Red flags: confident facts that aren't in any source's summary/excerpt (hallucination — the cardinal sin for a notes tool); the model ignoring the sources and answering from general knowledge; an answer that contradicts a source.
   • Citations. done.invalid should be [] — a non-empty value means the model cited a source number that wasn't retrieved (the server kept the answer but flagged it; call it out). done.cited should be a non-empty subset of 1..len(sources) for a question the corpus can answer. done.answered should track that. If the corpus genuinely doesn't cover the question, answered: false with an "I don't have notes on that"-style answer is the correct outcome, not a failure — but the sources list and scores should make that plausible.
   • Source quality. Same red flags as the /query smoke test: top-ranked sources should be on-topic; watch for everything clustered just above the 0.35 floor (weak retrieval), or journal-template / stub pages crowding in. Quote the actual titles + scores.
   • Streaming. tests/test_ask.py checks there's ≥1 delta and that done.chars matches the concatenated deltas — if those pass, streaming is wired. If the whole answer arrived as a single delta, note it (works, but suggests the chat backend isn't actually streaming).
   • Chat-backend log. grep -iE 'error|panic|warn|context|n_ctx|truncat' /tmp/fire_seq_search.chat.stderr.log /tmp/fire_seq_search.chat.stdout.log and grep -iE 'error|panic|/ask' /dev/shm/fsq_ask_debug.log. Surface anything non-routine. Specifically watch for context-overflow / prompt-truncation warnings from llama-server — /ask packs K×(summary+chunk) and the chat backend runs at -c 8192; if the prompt is getting truncated, the answer is built on partial context and that's a real bug. Also watch for HTTP 500s from either backend.