torch-profile

How is the time distributed? Where is the bottleneck? What is not the problem? Every finding gets chased to a file:line.

torch-profile

• Profile the script at the current commit with torch.profiler.
• Add the profiler hooks yourself when the script has none.
• What you produce is a diagnosis, not a patch. Those hooks are the only edit you make: nothing here changes what the job computes or how long it takes, and nothing here measures a fix.

① capture                ② views                ③ findings
   one plain run +    →     how the wall      →    ranked, anchored,
   a profiled window        actually splits        costed, handed off

① Capture

Two runs.

First, unprofiled: torch.profiler adds overhead, so take the honest clock with the cheap APIs instead, such as torch.cuda.Event(enable_timing=True) and torch.cuda.max_memory_allocated(). This is also the only run that sees startup, which no profiling window covers: dataset preprocessing, checkpoint loading, first-step compilation, and anything else paid once. Time every step, and save both that series and those startup numbers under commits/<sha>/.

Then profiled, over about 3 steady-state steps, with the window chosen from the step-time series the first run produced. Steps 3-4-5 is a default, not a law: if records/execution_schedule.json marks a periodic phase, the window must not straddle it, and if step time trends, the window has to sit in the regime the run actually spends its steps in. Traces go under commits/<sha>/trace/, which workspace.py candidate <ws> <sha> creates.

② Views

One profiled step can hold a million events, and nobody reads a million events. A view is a reshaping that makes one kind of anomaly stand out and hides the rest. tools/view/ loads the traces into a store, then renders all five from it, one numbered file each:

export PYTHONPATH=<plugin root>   # you are standing in the target, not here
python -m tools.view load   <ws>/commits/<sha>/trace/*.json -o <ws>/commits/<sha>/store.db
python -m tools.view render <ws>/commits/<sha>/store.db -o <ws>/commits/<sha>/views --capacity-gb <GB>