compare-llm-d-configurations

Compare llm-d Configurations

Purpose

Orchestrate two benchmark runs against different llm-d stack configurations, then compare results side by side. Each new run follows the sequence: deploy → benchmark → save state → teardown. If one configuration was already benchmarked in a previous session, its results can be loaded directly — only the other configuration goes through the full deploy/benchmark/teardown cycle. The comparison report is written to disk and displayed inline. Note that this skill uses three other skills: deploy-llm-d, run-llm-d-benchmark, and teardown-llm-d.

---

Phase 0: Pre-flight Setup

0.1 Create Comparison Workspace

Create a timestamped local directory to hold state for both runs:

COMPARISON_DIR="llm-d-comparison-$(date +%Y%m%d-%H%M%S)"
mkdir -p $COMPARISON_DIR/run-a/results
mkdir -p $COMPARISON_DIR/run-b/results
echo "Comparison workspace: $COMPARISON_DIR"

0.2 Check for Pre-existing Run

Ask the user: "Do you have results from a previous benchmark run you'd like to compare against, or are both configurations being run fresh?"

• Both fresh — proceed normally through Phases 1 and 2.
• One pre-existing — ask which one (A or B) is pre-existing, then collect its details now (see below). Skip the deploy/benchmark/teardown phases for that run; only execute those phases for the new configuration.

Collecting pre-existing run details:

Ask the user for:

1. A short label for the pre-existing run (e.g. "baseline-qwen2.5-7b")
2. The path to its results directory (will be used for metric extraction in Phase 3)
3. A brief summary of its configuration — guide used, model, hardware, harness, workload (whatever they remember; used to populate the config table in the report)

Write a run_state.json for the pre-existing run directly into the comparison workspace, using the information provided:

{
  "label": "<user-provided label>",
  "timestamp": "<timestamp of original run, or 'unknown'>",
  "stack": {
    "guide": "<guide or 'unknown'>",
    "namespace": "<namespace or 'unknown'>",
    "hardware": "<hardware or 'unknown'>",
    "gateway": "<gateway or 'unknown'>",
    "model": "<model or 'unknown'>"
  },
  "benchmark": {
    "harness": "<harness or 'unknown'>",
    "workload": "<workload or 'unknown'>",
    "load_stages": "<load stages or 'unknown'>"
  },
  "results_path": "<absolute path to existing results directory>"
}