torch-xpu-profile

torch-xpu-profile

torch.profiler with ProfilerActivity.XPU captures CPU + XPU op timeline on Kineto. Use for "which aten::* op is slow?", "is the GPU idle?", "is decode kernel-bound?". For server-side profiling use vllm-xpu-profile; for SYCL kernel level use xpu-profile-unitrace.

CUDA -> XPU translation

Quickstart

import torch
from torch.profiler import profile, ProfilerActivity, schedule
from transformers import AutoModelForCausalLM, AutoTokenizer

mid = "Qwen/Qwen2.5-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(
    mid, dtype=torch.bfloat16, device_map="xpu"
).eval()
inp = tok("Tell me a joke.", return_tensors="pt").to("xpu")

# Warmup so first-run kernel compile doesn't pollute the trace
with torch.no_grad():
    model.generate(**inp, max_new_tokens=8, do_sample=False,
                   pad_token_id=tok.eos_token_id)

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.XPU],
    record_shapes=True,
    profile_memory=True,
    with_stack=False,
) as prof:
    with torch.no_grad():
        model.generate(**inp, max_new_tokens=64, do_sample=False,
                       pad_token_id=tok.eos_token_id)

print(prof.key_averages().table(sort_by="xpu_time_total", row_limit=20))
prof.export_chrome_trace("trace.json")

Output: trace.json (drag into https://ui.perfetto.dev) plus a top-20 ops table sorted by XPU time.

Reading the trace

1. Top of key_averages table — first 5 rows are usually 70–90% of XPU time. Top = matmul/attention -> GPU-bound (good). Top = aten::copy_ / H2D-D2H transfers -> bandwidth-bound or paying for unnecessary moves.
2. Gaps in XPU timeline — wide white bands = GPU waiting for host. Causes: tokenizer on CPU, Python overhead, per-step sync. Decode TPOT spikes correlate with these gaps.
3. Per-step decode structure — same kernel sequence per token; divergence often means KV reallocation or SWA edge effects.
4. Memory peaks (with profile_memory=True) — peak before first generated token is prefill; subsequent peaks are KV growth. Compare against model-can-it-fit prediction.