marin: Skill for Claude Code

A/B test Zephyr changes

Signals

The coordinator writes one zephyr.stage row per completed stage and execution_id. Use these fields:

Field	Aggregation across executions	Interpretation
cpu_time_total	sum	Primary efficiency and compute-cost signal
elapsed	sum, labeled as summed stage elapsed	Secondary latency signal; sensitive to scheduling and stragglers
items	sum	Workload-equivalence check
bytes_processed	sum	Workload-equivalence check
mem_peak_bytes_max	max	Worst observed shard RSS and OOM guardrail
mem_bytes_avg	weighted interpretation only	Typical shard RSS context
cpu_pct_avg	weighted interpretation only	CPU saturation context
item_rate, byte_rate	do not aggregate	Derived from noisy elapsed time

cpu_time_total sums process user and system CPU-seconds across completed shards, so worker count and queue delay do not directly change it. Use it as the primary efficiency signal and normalize per item or byte when accepted workload sizes differ. elapsed measures the stage barrier and includes startup, I/O, concurrency, queueing, and stragglers; repeat an elapsed-only result under comparable scheduling conditions.

Keep CPU and elapsed time as separate outcomes:

• CPU flat or lower and elapsed lower: latency or topology win without added compute cost.
• CPU higher and elapsed lower: faster and more expensive.
• CPU lower and elapsed higher: cheaper and slower.
• Wall-only change from one comparison: inconclusive until repeated.
• Topology or batching change: report the latency/compute tradeoff; do not describe wall-time gains as equivalent per-core efficiency gains.

Calibrate thresholds from same-code repeats for the selected sample and pool shape. A new OOM, application failure, or memory peak above the worker limit is a regression regardless of CPU.

Choose the comparison

Existing runs

Start at Collect execution IDs when control and treatment jobs already exist. Confirm that the control and every treatment used the same immutable sample, stage range, sources, worker resources, concurrency, parallelism, cluster, region, and priority.