lexical-kb

lexical-kb — query an embedding-free knowledgebase

This KB has no semantic search and no embedding model. Retrieval is pure lexical BM25 over a precomputed inverted index. That design moves one job onto you: bridging the gap between how the user phrases a question and how the corpus phrases the answer. An embedding model would do this with a vector; here you are the semantic layer — you expand the query into terms before searching.

Corpus: {{SOURCE}} ({{CHUNK_COUNT}} chunks).

The retrieval protocol — follow every step

A raw user question fed straight to BM25 underperforms: it matches only the exact words the user happened to use. The expansion step is what makes lexical retrieval competitive with embeddings. Do not skip it.

1. Read the question. Extract core terms — the essential nouns, proper nouns, and identifiers the answer MUST contain. These carry full weight.

2. Generate expand terms — synonyms, morphological variants (plural/verb forms), acronym expansions and contractions, and adjacent concepts. These carry lower weight. This is the work the missing embedding model would have done. Be generous: 5–15 expansion terms is normal.

3. Run the searcher. It ships in this bundle in two equivalent runtimes — node search.js or python3 search.py, identical flags and identical results. Use whichever your environment has. Pass the user's original question via --query AND your term groups — expansion is additive, it never replaces the user's words:

   node search.js \
     --query "how does centered simhash differ from random projection?" \
     --core "simhash" --core "centered" \
     --expand "random projection" --expand "hyperplane" --expand "LSH" \
     --expand "binary quantization" --expand "hamming distance" \
     --k 5
   

   --core/--expand are repeatable; pass phrases, the searcher tokenizes them. The --query terms contribute at a low floor weight so a curated synonym can lift a result but can never drop a doc the literal question would have matched. Defaults: core 1.0, expand 0.4, query-floor 0.25, top-k 5. Keep expansion targeted — terms too generic ("system", "process") leak into unrelated chunks and blur the ranking. A precise word can mislead too if it is polysemous: prefer the disambiguating phrase as one --core term (e.g. --core "centered simhash") over a bare ambiguous word (--core "centered", which also matches "centered around …" in unrelated chunks). The passage extractor is lexical too, so it will highlight the wrong sense rather than correct it.