neuron-framework-profile-analysis-vllm-neuron

Analyzing vllm-neuron Profiles

Note: The analysis methodology (steps 2–5) is adapted from the NKI profiling skill and is not fully validated for framework-compiled NEFFs. Performance bounds, collective partitioning, and investigation results should be treated as approximate. Step 1 (ingestion) is stable.

Verification discipline (BLOCKING — read and satisfy before any analysis)

Profile analysis is where confident-sounding wrong answers are easiest to produce. These rules are a hard gate, not advice — distilled from repeated real failures on this workflow:

1. Device time comes from the LIVE nc_exec_running system trace, never from an isolated neuron-profile / neuron-explorer capture NEFF replay. A standalone replay undercounts the real multi-rank collective traffic (measured: replay ~12.9 ms vs live ~18 ms for the same TP=16 graph; another run mis-quoted a 14.8 ms replay as "the device floor" and built a wrong "serving overhead" conclusion on it). Always cross-check against the live trace (references/system-trace-queries.md §1f).
2. Query already-ingested profile data before writing any custom timing script. The parquet tables already hold per-op, per-engine, ns-resolution timestamps. If you catch yourself about to write time.perf_counter() around a model call, or a benchmark/timing harness for a device question — STOP and query the profile. Reach for a hand-rolled timer only when the question is genuinely host-side and never touches the device — and even then, check the system trace first.
3. Never compare two numbers captured from different runs. Subtracting a device total_time from one capture and a wall-clock from another ("19.5 − 12.9 = 6.6ms overhead") is arithmetic dressed up as measurement. Numbers compared must come from the same profile / same execution.
4. Validate concurrency claims with overlapping measurement windows, not overlapping process lifetimes. Two replicas whose processes run concurrently can still have timed windows that don't overlap (compile-time skew). Use an explicit barrier and confirm the windows overlap before reporting a scaling number.