kb-ingest-bulk

Bulk Ingestion (map-reduce)

Load a large source set into the knowledge base in parallel. Four phases: map (parallel extract) → route (Python) → reduce (parallel synthesis, one writer per file) → finalize (one rebuild + one log entry).

Arguments

Preflight

1. Resolve library_path: if --library given, use it; else read CLAUDE.md ## Knowledge Base section. Derive shelf_index_path = <library>/_shelf-index.md, log_path = <library>/log.md, extracts_dir = <library>/.extracts.
2. Verify knowledge-extractor and agent-knowledge-updater agents are available.
3. Clamp --parallel to [1, 64].

Module bootstrap

Use this CLAUDE_PLUGIN_ROOT importlib loader block to import the sdlc_knowledge_base_scripts.kb_ingest_bulk helpers. It is self-contained — no other file needs to be opened.

python3 -c "
import sys, os, importlib.util, json
PLUGIN_ROOT = os.environ.get('CLAUDE_PLUGIN_ROOT', '')
SCRIPTS = os.path.join(PLUGIN_ROOT, 'scripts')
INIT = os.path.join(SCRIPTS, '__init__.py')
if os.path.isfile(INIT) and 'sdlc_knowledge_base_scripts' not in sys.modules:
    spec = importlib.util.spec_from_file_location(
        'sdlc_knowledge_base_scripts', INIT, submodule_search_locations=[SCRIPTS])
    if spec and spec.loader:
        mod = importlib.util.module_from_spec(spec)
        sys.modules['sdlc_knowledge_base_scripts'] = mod
        spec.loader.exec_module(mod)
from sdlc_knowledge_base_scripts.kb_ingest_bulk import (
    discover_sources, build_bulk_manifest, load_manifest, save_manifest,
    retry_failed, persist_extract, slug_for_source, mark_source_extracted,
    mark_source_failed, route_extracts, format_extract_prompt,
    format_reduce_prompt, ReduceDispatchRequest, mark_target_reduced,
    mark_target_failed, summarize_run, write_log_entry, ExtractDispatchRequest
)
"