harness:health

/harness:health

Check eval dataset quality and auto-correct issues. Can be run independently or is invoked by /harness:evolve before the iteration loop.

Prerequisites

.evolver.json must exist. If not, tell user to run /harness:setup.

Resolve Tool Path and Python

TOOLS="${EVOLVER_TOOLS:-$([ -d ".evolver/tools" ] && echo ".evolver/tools" || echo "$HOME/.evolver/tools")}"
EVOLVER_PY="${EVOLVER_PY:-$([ -f "$HOME/.evolver/venv/bin/python" ] && echo "$HOME/.evolver/venv/bin/python" || echo "python3")}"

1. Run Health Diagnostic

$EVOLVER_PY $TOOLS/dataset_health.py \
    --config .evolver.json \
    --production-seed production_seed.json \
    --output health_report.json 2>/dev/null

Print summary:

python3 -c "
import json, os
if os.path.exists('health_report.json'):
    r = json.load(open('health_report.json'))
    print(f'Dataset Health: {r[\"health_score\"]}/10 ({r[\"example_count\"]} examples)')
    for issue in r.get('issues', []):
        print(f'  [{issue[\"severity\"]}] {issue[\"message\"]}')
    if not r.get('issues'):
        print('  No issues found.')
"

2. Auto-Correct Issues

If health_report.json has corrections, apply them automatically:

CORRECTIONS=$(python3 -c "
import json, os
if os.path.exists('health_report.json'):
    r = json.load(open('health_report.json'))
    for c in r.get('corrections', []):
        print(c['action'])
" 2>/dev/null)

For each correction:

If create_splits: Assign 70/30 train/held_out splits:

$EVOLVER_PY -c "
from langsmith import Client
import json, random
client = Client()
config = json.load(open('.evolver.json'))
examples = list(client.list_examples(dataset_name=config['dataset']))
random.shuffle(examples)
sp = int(len(examples) * 0.7)
for ex in examples[:sp]:
    client.update_example(ex.id, split='train')
for ex in examples[sp:]:
    client.update_example(ex.id, split='held_out')
print(f'Assigned splits: {sp} train, {len(examples)-sp} held_out')
"