manzanas: Skill for Claude Code

Run the eval harness

eval/ is a pure protocol client: it runs YAML scenarios against any daemon N times (fresh lease + reset per run) and reports pass/fail, determinism rate, flaky steps, and per-step latency percentiles.

Build and run

go build -o bin/manzanas-eval ./eval/cmd/manzanas-eval

bin/manzanas-eval \
  --daemon http://<mac-host>:7433 \
  --runs 3 \
  --out eval-out \
  eval/scenarios/*.yaml

Outputs: eval-out/report.md (human), report.json (machine), <scenario>-<name>-run<N>.png screenshot artifacts. Exit code is non-zero on any failing run or cross-run tree-hash drift, so it drops straight into CI.

Requirements & caveats (verified)

• The shipped scenarios need a real Mac daemon: they use reset: erase and dispatch launch_app/observe. Against a --mock daemon the acquire fails with 501 not_implemented: reset is not implemented in this build — that is expected, not a harness bug. For Linux/CI coverage of the harness itself use go test ./eval/... (httptest fake daemon).
• Each run acquires a fresh lease with the scenario's reset policy, so eval runs erase their target on release — point evals at a host/label where that is acceptable, and prefer a device-specific label (e.g. iphone-air) so the reset can't select another agent's simulator.
• The runner renews the lease during long runs and deletes any snapshots the run created at teardown.
• Boots pass the daemon's safety gates; on a loaded host expect 503 overloaded retries to stretch run times.

Writing a scenario

YAML (strict parsing — unknown fields rejected; step names unique):

name: my-check
lease:
  labels: [iphone-air]     # device-specific label on shared hosts
  reset: erase             # none | erase | snapshot:<name>
  ttl_seconds: 600
default_timeout: 120s
steps:
  - op: boot
    timeout: 300s
  - op: action
    kind: launch_app
    payload: {bundle_id: com.apple.Preferences, terminate_running: true}
  - op: assert
    assert:
      element_exists: {label: General}