trace-cmd

Trace analysis

Analyze kernel trace captures to support the patch-measure-analyze development cycle. Operate on trace-cmd .dat files or pre-generated report output.

When the user provides a trace file path, begin analysis immediately. If no specific question is asked, produce a general summary first, then offer deeper analysis.

trace-cmd .dat files are binary captures and trace-cmd report output streams through pipelines rather than landing in a normal text file. awk, grep, head, and similar shell tools are the correct way to extract fields and summaries from this data; any general preference for dedicated file-reading tools does not apply here.

CRITICAL: trace-cmd argument order

The input file MUST be specified with -i BEFORE any filter flags. Positional file arguments after -F are misinterpreted as filter files.

Correct:  trace-cmd report -i /tmp/trace.dat -F 'event_name'
Wrong:    trace-cmd report /tmp/trace.dat -F 'event_name'
Wrong:    trace-cmd report -F 'event_name' /tmp/trace.dat

Output format

Each event line follows this layout:

        TASK-PID  [CPU] FLAGS  TIMESTAMP: EVENT:  field=val ...

The TASK-PID column has variable-width leading whitespace and may contain slashes. Positional field numbers ($1, $4) are unreliable. Match patterns instead:

# Extract timestamp
awk '{match($0, /[0-9]+\.[0-9]+:/); ts=substr($0,RSTART,RLENGTH-1)+0}'

# Extract process name (strip trailing PID)
awk '{gsub(/^[ \t]+/,""); sub(/-[0-9]+.*$/,""); print}'

Pipeline reliability

trace-cmd report writes a multi-line header to stderr. Some builds intermix event lines on stderr. Two rules:

1. Always use 2>&1 when piping report output, then filter out non-event lines with grep for the event name.
2. Save filtered output to a temp file before running multi-stage extraction pipelines.

# Save each event type to its own file early
for ev in svc_stats_latency svc_xprt_dequeue svc_xprt_enqueue; do
  trace-cmd report -i <file> -F "$ev" 2>&1 | \
    grep "$ev" > /tmp/${ev}.txt
done