collect-node-data

Collect node-store data from specific hardware

The autotune node table (SearchDB, default ~/.cache/emmy/autotune.db) is a cross-hardware dataset of search-tree value-of-position rows — the leaf measurements the future offline prior trains on, with the full feature dict the prior sees. It is read by emmy eval prior --dataset nodes (Spearman + regret per card and compile regime) and feeds prior diagnostics. Because your dev box (and most of the fleet) has no local CUDA GPU, the data for any given card must be measured on that card and brought back.

This skill does exactly that for one GPU, in a single budgeted run on the box: get a server (an existing one the user provides, or rent one) → set up emmy → the golden sweep (scripts/remote_node_collect.py, driving scripts/golden_neighbor_bench.py) → merge the new card's node rows into the local DB → back the DB up → tear the server down (only if this run rented it). Everything long-running on the remote — the apt/make setup phase and the sweep itself — runs inside named tmux sessions (emmy-node-setup / emmy-node-sweep), so a dropped ssh connection never kills the work and a human can attach to watch it.

Why a budgeted sweep (and not a search-driven tune). The old flow ran an ε-greedy emmy tune --dataset golden first: its wall time grew with the golden set (patience-stopped search per shape, no budget knob), and even at ε=0.25 three quarters of its benches followed the incumbent prior — the collected rows over-sampled the branches the prior already liked (a censoring feedback loop). The sweep replaces it: a fixed --budget-s wall clock, and points drawn from the enumeration itself rather than a search trajectory, so the future offline prior gets clean leaf rows whose selection didn't depend on the incumbent's opinions. emmy tune --explore-eps still exists for interactive tuning; it is just no longer the collection vehicle.

Why three slices. Every golden shape's candidate pool (all kinds — matmul, reduce, rms_norm, softmax, attention, the fused norm_linear / mlp_geglu, …) is split by distance to the recorded golden anchors, and the budget is spent across the slices at configurable shares (default 60/25/15):

• own — rows within --max-dist of the live card's OWN goldens: dense label support (and fine ranking resolution) exactly where this card's deploys land.
• cross — rows near OTHER cards' goldens that realize on this card: every such point is verified-excellent somewhere, so measuring it here yields either transfer signal or an arch-disagreement row — the data the prior's arch × knob interactions (and a leave-one-card-out evaluation) need. An anchor that doesn't enumerate here is skipped, which is itself the realizability filter.
• tail — a capped, hash-ordered (seed-independent, so stable across sessions) subsample of the rest of the enumeration: landscape support so the prior also learns what bad looks like.

Kind-stratified within each slice. After the slice draw, each batch picks a golden KIND uniformly among the kinds that still have points in that slice, and only then a shape within the kind (proportional to its remaining pool). Without this, matmul — ~87% of the selectable points on a 4090 — soaked up nearly the whole budget and small kinds (softmax, attention, linear_norm) could end a 4 h run with zero benches. Small kinds exhaust their pools early and their share flows back to the big ones, so a run guarantees every kind data first and spends the rest where the pool actually is.

Bench in the deployable regime. The offline prior trains on deployable records, and a sweep now measures there by default (emmy run --bench --ab, node recording on), so no pairing step is needed: every point collected is already a deployable measurement. Historical stores also hold -O1 rows from the era when sweeps ranked at that level; those are inert — the prior does not train on them and no deploy reads them. Do NOT pin --nvcc-flags to a non-deployable level to collect "cheaper" data: its error is biased along tile size, so it mis-ranks the wide register-tile family it would most matter for.