index

Index PDFs, markdown, or source code into Qdrant collections for semantic search.

Note: For most users, /arc:corpus sync is recommended as it indexes to both semantic and full-text search. Use /arc:index when you only need semantic search in a single collection.

IMPORTANT: You must specify a subcommand (pdf, code, markdown, or text) before the path.

Subcommands (required):

• pdf: Index PDF documents into Qdrant (with OCR support)
• markdown: Index markdown files into Qdrant (with frontmatter extraction)
• code: Index source code repositories into Qdrant (git-aware)
• text: Index content to MeiliSearch for full-text search (has its own pdf/code/markdown subcommands; advanced — prefer /arc:corpus sync or /arc:indexes for full-text)

Common Options:

• --collection: Target collection (required)
• --from-file: Read file paths from list (one per line, or "-" for stdin)
• --model: Embedding model (auto-selected by content type)
• --workers: Parallel workers (default: 4)
• --force: Force reindex all files
• --randomize: Randomize file processing order (useful for parallel indexing)
• --gpu: Opt into accelerator embedding (CPU is the stable default)
• --streaming: Stream embeddings to Qdrant immediately (lower memory usage)
• --verbose: Show detailed progress (suppress library warnings)
• --debug: Show all library warnings including transformers
• --json: Output in JSON format

PDF Indexing Options:

• --no-ocr: Disable OCR (enabled by default for scanned PDFs)
• --ocr-language: OCR language code (default: eng)
• --ocr-workers: Parallel OCR workers (default: cpu_count)
• --normalize-only: Skip markdown conversion, only normalize whitespace
• --preserve-images: Extract images for multimodal search
• --process-priority: Process scheduling priority (low, normal, high)
• --embedding-batch-size: Batch size for embeddings (auto-tuned if not specified)
• --offline: Use cached models only (no network)

Markdown Indexing Options:

• --chunk-size: Target chunk size in tokens (overrides model default)
• --chunk-overlap: Overlap between chunks in tokens
• --recursive/--no-recursive: Search subdirectories recursively (default: recursive)
• --exclude: Patterns to exclude (e.g., node_modules, .obsidian)
• --offline: Use cached models only (no network)

Source Code Indexing Options:

• --depth: Git discovery depth (traverse subdirectories)

Examples:

# Basic indexing (CPU-first stable defaults)
/index pdf ~/Documents/Research --collection PDFs --model arctic-m
/index markdown ~/notes --collection Notes --model arctic-m
/index code ~/projects/myapp --collection MyCode --model jina-code

# Index from file list
/index pdf --from-file /path/to/pdf_list.txt --collection PDFs
/index markdown --from-file /path/to/md_list.txt --collection Notes

# Index from stdin (pipe file paths)
find ~/Documents -name "*.pdf" | /index pdf --from-file - --collection PDFs
ls ~/notes/*.md | /index markdown --from-file - --collection Notes

# With options
/index markdown ~/docs --collection Docs --chunk-size 512 --verbose
/index pdf ~/scanned-docs --collection Scans --no-ocr --offline

# Opt into accelerator embedding
/index pdf ~/Documents/Research --collection PDFs --model stella --gpu

# Streaming mode (lower memory for large collections)
/index pdf ~/Documents/Research --collection PDFs --streaming
/index code ~/projects/myapp --collection MyCode --streaming

# Parallel indexing from multiple terminals (randomize order)
/index pdf ~/Documents/Research --collection PDFs --randomize
/index markdown ~/notes --collection Notes --randomize

# Debug mode (show all warnings)
/index pdf ~/Documents/Research --collection PDFs --model stella --debug

Execution:

arc index $ARGUMENTS

File List Format (--from-file):

When using --from-file, provide a text file with one file path per line:

# Comments are supported (lines starting with #)
/absolute/path/to/file1.pdf
relative/path/to/file2.md
/another/file3.pdf

# Empty lines are ignored