evo-parallel-tfidf

Parallel TF-IDF Search Engine Skill

Overview

This skill parallelizes a sequential TF-IDF document search engine using Python's multiprocessing.Pool. The key design decisions:

1. Two-phase index building: Phase 1 parallelizes tokenization/TF/DF computation. Phase 2 parallelizes inverted index + doc vector construction using global IDF.
2. Minimized serialization: Each worker gets only its document chunk + the shared IDF dict (not all doc data).
3. Initializer-based search workers: The index is sent once via Pool(initializer=...) rather than per-query.
4. Batched queries: Queries are grouped into batches for amortized IPC cost.

Key Architecture

Index Building Strategy

• Split documents into chunks (respecting chunk_size param, but ensuring >= num_workers chunks)
• Phase 1 workers: tokenize docs, compute TF, compute local DF counts
• Main process: merge DFs, compute global IDF
• Phase 2 workers: given their doc TFs + global IDF, compute partial inverted index + doc vectors + norms
• Main process: merge partial inverted indices, sort posting lists

Search Strategy

• Use Pool(initializer=_init_search_worker, initargs=(index, documents)) to send index once
• Split queries into batches (num_workers * 4 for load balancing)
• Each worker processes its batch using the sequential search_sequential function
• Reassemble results by query index

Usage

import sys
sys.path.insert(0, '/app/environment/skills/evo-parallel-tfidf/scripts')
from parallel_tfidf import build_tfidf_index_parallel, batch_search_parallel, ParallelIndexingResult

# Also need sequential module in path
sys.path.insert(0, '/root/workspace')
from document_generator import generate_corpus

# Generate or load documents
documents = generate_corpus(5000, seed=42)

# Build index in parallel
result = build_tfidf_index_parallel(documents, num_workers=4, chunk_size=500)
print(f"Built index in {result.elapsed_time:.3f}s")

# Search in parallel
queries = ["machine learning", "data analysis"]
results, elapsed = batch_search_parallel(queries, result.index, top_k=10, num_workers=4, documents=documents)
print(f"Search completed in {elapsed:.3f}s")
for i, query_results in enumerate(results):
    print(f"Query '{queries[i]}': {len(query_results)} results")