search-vector-lord

[SKILL] search-vector-lord [OBJ] Build full-text and vector retrieval systems for search and RAG. [RULES]

1. [CMD] IDs: Elasticsearch /websites/elastic_co_guide_en_elasticsearch_reference_8_19, OpenSearch /opensearch-project/documentation-website, Meilisearch /websites/meilisearch, pgvector /pgvector/pgvector, Pinecone /websites/pinecone_io, Milvus /milvus-io/milvus-docs.
2. [REQ] Pillar coverage: full-text search, vector search, hybrid retrieval, schemas/mappings, indexing/ingestion, scaling/operations, observability, security.
3. [REQ] Query engine ID with full question + topic (vector search, mapping, indexing, scaling).
4. [REQ] Distinguish managed vs self-hosted operational concerns.
5. [REQ] RAG designs combine vector + metadata filtering + keyword/rerank unless user asks pure vector.
6. [REQ] Retrieval quality is measured: build an eval set (queries + expected hits) BEFORE tuning; track recall@k, MRR, NDCG; every retrieval change is an experiment with a metric, not a hunch.
7. [REQ] Hybrid is the default: dense vectors miss exact terms (SKUs, names, error codes); BM25 misses semantics. Combine via RRF or weighted fusion, then a reranker (cross-encoder/Cohere-style) on the top-N for production quality.
8. [REQ] Chunking is design: chunk by semantic boundaries (headings, functions, sections) not fixed token windows; overlap ~10-15%; keep chunk metadata (source, section, path, timestamp) — filtering and citations depend on it.
9. [REQ] Embeddings are a contract: model choice fixes dimension + index forever — re-embedding = re-indexing. Version the embedding model in the index name; multilingual/domain data may need domain-specific models; eval the model on YOUR data.
10. [REQ] Index structure: HNSW params (m, ef_construction, ef_search) are recall-vs-latency levers — tune ef at query time per SLO; filtered search (pre- vs post-filtering) materially changes recall, especially high-selectivity filters.
11. [REQ] Full-text specifics: analyzer choice (stemming, synonyms, stopwords) is per-language and per-field; mappings immutable → reindex on change; fuzzy/typo-tolerance tuned (Meilisearch ships it sane; ES needs explicit config).
12. [REQ] Metadata filtering: always design filters into the schema (tenant_id, doc_type, date) — post-filtering retrieved results is a recall bug; tenant isolation enforced at query construction, not application hopes.
13. [REQ] Freshness & ops: index updates are async (refresh interval, vector index rebuilds) — document staleness windows; monitor index lag, query latency p95, cache hit rates; reindex/rollback plans before schema changes.
14. [REQ] RAG answer discipline: retrieved context is untrusted (injection surface); cite chunk sources; relevance threshold below which you say "no answer" rather than hallucinate; context budget allocation (most relevant in, dedupe near-identical chunks).
15. [PROHIBIT] Pure-vector search for term-heavy corpora, changing embedding models without reindexing, metadata filtering bolted on after retrieval, unbounded top-k ("k=1000" is a smell), or shipping retrieval without an eval set.