Fetchium: Instructions file for Codex

AGENTS.md

Guidelines for AI agents (Claude Code, OpenAI Codex, Gemini CLI, etc.) working on the Fetchium codebase.

Build, Lint, and Test Commands

# Check compilation (fast, no linking)
cargo check

# Build the fetchium binary
cargo build -p fetchium-cli

# Build release binary
cargo build -p fetchium-cli --release

# Run all tests (currently 563, target: 0 failures)
cargo test

# Run tests for a specific crate
cargo test -p fetchium-core

# Run a single test by name
cargo test -p fetchium-core segment_type_roundtrip

# Lint (zero warnings policy — enforced in CI)
cargo clippy -- -D warnings

# Format code
cargo fmt

# Run the binary
./target/debug/fetchium --help
./target/debug/fetchium doctor
./target/debug/fetchium provider list

Current Status

• Tests: 563 passing, 0 failing, 0 clippy warnings
• Phases complete: 0–8 + Multi-provider AI system
• Binary: ./target/debug/fetchium with 26 commands

Architecture

crates/
├── fetchium-core/     # All algorithms: search, extract, rank, validate, cache, AI, intelligence
├── fetchium-cli/      # Binary: clap derive CLI, one file per command in commands/
├── fetchium-mcp/      # Manual JSON-RPC 2.0 stdio MCP server (5 tools)
└── fetchium-api/      # axum 0.7 REST API server

Data flow: CLI args → FetchiumConfig → fetchium-core pipeline → formatted output

Key Module Map (fetchium-core/src/)

ai/
  credentials.rs      Subscription OAuth detection (Claude Code, Gemini CLI, Codex CLI)
  provider_client.rs  Multi-provider chat client with SSE streaming + fallback chain
  providers.rs        ProviderKind enum, ProvidersConfig, ProviderEntry
  pipeline.rs         run_ai_pipeline() — search → extract → sandwich → provider
  types.rs            AiConfig (providers: ProvidersConfig, fast_model: Option<String>)
  router.rs           select_model(), select_fast_model()
  sandwich.rs         Ms-PoE sandwich layout
  ollama.rs           OllamaClient (local Ollama server)
  prompt.rs           System prompts for synthesis/factual/fallback modes
  setup.rs            DeviceSpec, recommend_models(), format_setup_guide()

search/
  orchestrator.rs     Parallel backend dispatch + BM25 rerank + dedup
  duckduckgo.rs       DDG HTML scraper
  fallback.rs         FallbackChain async executor

extract/
  layer1.rs, layer2.rs  CEP CSS+readability extraction
  pipeline.rs         Speculative parallel extraction
  boilerplate.rs      QADD pre-filter (strips script/style/svg)
  cep_predictor.rs    Decision tree ML predictor for CEP layer selection

token/
  qatbe.rs            BM25 + hybrid embedding ranking, greedy knapsack
  scs.rs              8 segment types, token-efficient JSON
  pds.rs              4-tier progressive streaming
  counter.rs          Heuristic tokenizer + TokenBudget

rank/
  bm25.rs             tantivy-backed Bm25Scorer
  signals.rs          ScoringContext (batch embeddings), HyperFusion 8-signal ranking
  fusion.rs           hyperfusion_rank()

validate/
  cross_source.rs     V4 bigram-Jaccard clustering, negation-aware contradiction
  temporal.rs         V3 exponential decay, intent classification
  authority.rs        V1 domain tiers, SSL/redirect penalties
  rar.rs              5 reflection checkpoints R1-R5

intelligence/
  pie/               STM (source trust), FPM (failure patterns), QPM (query prediction), PKG
  edf.rs             Evidence decay function, domain half-lives
  cce.rs             Confidence calibration (isotonic interpolation)
  acs.rs             Adversarial content shield (shadow/active mode)
  crp.rs             Contradiction resolution protocol
  sgt.rs             Source genealogy tracker (bigram-Jaccard mutation detection)
  totr.rs            Tree-of-Thoughts research

export/
  pandoc.rs          PDF: typst (~1s) → xelatex → default; check_typst() for doctor
  bibtex.rs          Pure Rust BibTeX generator

embeddings/
  engine.rs          fastembed-rs singleton, embed() + embed_batch()
  cache.rs           SHA-256 keyed SQLite cache

qadd/pipeline.rs     5-step DOM pruning, chunked embed_batch (EMBED_BATCH_SIZE=128)