harness-evolver CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this is

Claude Code plugin for LangSmith-native autonomous agent optimization. Uses LangSmith Datasets, Experiments, and Evaluators as the backend. Proposers modify the user's real code in isolated git worktrees. Distributed via npm (npx harness-evolver@latest) and the Claude Code plugin marketplace.

Dependencies

pip install langsmith                    # Python SDK — used by all tools
uv tool install langsmith-cli            # CLI — used by evaluator agent for reading runs and writing feedback

The SessionStart hook (hooks/session-start.sh) auto-creates a venv and installs both on each session.

Running tools locally

All Python tools live in tools/ and require the langsmith SDK. They auto-load the API key from the langsmith-cli credentials file if LANGSMITH_API_KEY is not in the environment.

# Setup — creates dataset, evaluators, baseline, writes .evolver.json
python tools/setup.py --project-name my-agent --entry-point "python main.py" --framework langgraph --goals accuracy

# Run evaluation for a candidate in a worktree (canary preflight runs 1 example first)
python tools/run_eval.py --config .evolver.json --worktree-path /tmp/wt --experiment-prefix v001a
# Default concurrency is 3. Use --concurrency 1 for agents that can't run in parallel (shared files, fixed ports, etc)
# Use --sample 10 to evaluate a random subset (used by light mode)
# Use --sample 10 --sample-split train to sample from train only + always eval all held_out
# Use --no-canary to skip preflight check

# Compare experiment results
python tools/read_results.py --experiments v001a,v001b --config .evolver.json --output comparison.json

# Trace analysis from an existing experiment
python tools/trace_insights.py --from-experiment "v003-2026-04-01" --output trace_insights.json

# Import production traces as test data
python tools/seed_from_traces.py --project my-prod-project --output-md production_seed.md --output-json production_seed.json

# AST-based architecture analysis (stdlib-only, no langsmith needed)
python tools/analyze_architecture.py --harness path/to/agent -o output.json

# Evolution progress chart (stdlib-only, no langsmith needed)
python tools/evolution_chart.py --config .evolver.json

# Integrated preflight check (API key + schema + state + health + canary)
python tools/preflight.py --config .evolver.json

# Archive evolution candidate (stdlib-only, no langsmith needed)
python tools/archive.py --config .evolver.json --version v001 --experiment v001-abc --worktree-path /tmp/wt --score 0.85 --won
python tools/archive.py --config .evolver.json --list

# Promote proven evolution learnings to CLAUDE.md (compound learning)
python tools/promote_learnings.py --memory evolution_memory.md --target CLAUDE.md --threshold 5 --dry-run
python tools/promote_learnings.py --memory evolution_memory.md --target CLAUDE.md --threshold 5

# Log evolution iteration to LangSmith (creates traceable run per iteration)
python tools/log_iteration.py --config .evolver.json --action start --version v001
python tools/log_iteration.py --config .evolver.json --action end --run-id <id> --score 0.85 --merged true

# Update config after merge (handles backup/restore/update atomically)
python tools/update_config.py --config .evolver.json --action backup
python tools/update_config.py --config .evolver.json --action restore
python tools/update_config.py --config .evolver.json --action update --winner-experiment v001-abc --winner-score 0.85

# Clean up orphan worktrees after eval
python tools/cleanup_worktrees.py --dry-run
python tools/cleanup_worktrees.py

# Constraint validation for proposals (stdlib-only, no langsmith needed)
python tools/constraint_check.py --config .evolver.json --worktree-path /tmp/wt --baseline-path .

# Secret detection in text (stdlib-only, pipe text to stdin)
echo "text with API keys" | python tools/secret_filter.py

# Mine Claude Code sessions for eval data (stdlib-only)
python tools/mine_sessions.py --agent-description "my agent" --output mined.json

# Dataset health diagnostic (size, difficulty, coverage, splits)
python tools/dataset_health.py --config .evolver.json --output health_report.json

# Validate state before evolution
python tools/validate_state.py --config .evolver.json --output validation.json

# Check iteration gates
python tools/iteration_gate.py --config .evolver.json --output gate_result.json

# Track regressions between iterations
python tools/regression_tracker.py --config .evolver.json --previous-experiment v001a --current-experiment v002c --output regression_report.json

# Consolidate cross-iteration memory
python tools/consolidate.py --config .evolver.json --output evolution_memory.md

# Synthesize evolution strategy document
python tools/synthesize_strategy.py --config .evolver.json --output strategy.md

# Add evaluator to config
python tools/add_evaluator.py --config .evolver.json --evaluator factual_accuracy --type llm

# Inject adversarial examples
python tools/adversarial_inject.py --config .evolver.json --experiment v003a --inject

# Extract a consolidated capability contract (AGP Contract Generation)
# AST-walk the agent source tree and emit contracts.md (signatures + docstrings
# + @tool decorators). Reduces prompt bloat and keeps prompts synced with code.
python tools/extract_contracts.py --root . --output contracts.md
python tools/extract_contracts.py --config .evolver.json --output contracts.md  # use entry_point

# Build the RSPL-lite resource registry (AGP Layer 1)
# Classifies project files into prompt/tool/agent/environment/memory, hashes and
# versions them. Proposers can target a single type via the evolvable flag.
python tools/resource_registry.py --root .
python tools/resource_registry.py --root . --list --type tool
python tools/resource_registry.py --root . --freeze prompts/system.md  # mark non-evolvable

# TF-IDF retrieval over the evolution archive (AGP retrieve operator)
# Returns top-k historically similar candidates for a lens/failure description.
python tools/archive_search.py --config .evolver.json --query "retry rate limit"
python tools/archive_search.py --config .evolver.json --query-file lenses.json --winners-only --top 3
python tools/archive_search.py --config .evolver.json --query "..." --losers-only --format json