chat-perf

Chat Performance Testing

When to use

• Before/after modifying chat rendering code (chatListRenderer.ts, chatInputPart.ts, markdown rendering)
• When changing the streaming response pipeline or SSE processing
• When modifying disposable/lifecycle patterns in chat components
• To compare performance between two VS Code releases
• In CI to gate PRs that touch chat UI code

Quick start

# Run perf regression test (compares local dev build vs VS Code 1.115.0):
npm run perf:chat -- --scenario text-only --runs 3

# Run all scenarios with no baseline (just measure):
npm run perf:chat -- --no-baseline --runs 3

# Compare two local builds (apples-to-apples):
npm run perf:chat -- --build /path/to/build-A --baseline-build /path/to/build-B --runs 5

# Build a local production package and compare against a release:
npm run perf:chat -- --production-build --baseline-build 1.115.0 --runs 5

# Run memory leak check (10 messages in one session):
npm run perf:chat-leak

# Run leak check with more messages for accuracy:
npm run perf:chat-leak -- --messages 20 --verbose

Perf regression test

Script: scripts/chat-simulation/test-chat-perf-regression.js npm: npm run perf:chat

Launches VS Code via Playwright Electron, opens the chat panel, sends a message with a mock LLM response, and measures timing, layout, and rendering metrics. By default, downloads VS Code 1.115.0 as a baseline, benchmarks it, then benchmarks the local dev build and compares.

You don't always need a baseline. A baseline exists only for comparison (regression detection). If you just want the current build's numbers — profiling a single change, capturing traces/heap snapshots, or iterating on a scenario — pass --no-baseline to skip downloading and benchmarking the baseline entirely (roughly halves runtime). Baseline comparison is what turns raw measurements into a pass/fail verdict; without it you still get all the metrics, just no verdict.

Key flags

Flag	Default	Description
--runs <n>	5	Runs per scenario. More = more stable. Use 5+ for CI.
--scenario <id> / -s	all	Scenario to test (repeatable). See common/perf-scenarios.js.
--build <path|ver> / -b	local dev	Build to test. Accepts path or version (1.110.0, insiders, commit hash).
--baseline <path>	—	Compare against a previously saved baseline JSON file.
--baseline-build <path|ver>	1.115.0	Version or local path to benchmark as baseline.
--no-baseline	—	Skip the baseline entirely — just measure the test build (no download, no comparison, ~2× faster). Use when you only need raw numbers, not a regression verdict.
--save-baseline	—	Save results as the new baseline (requires --baseline <path>).
--resume <path>	—	Resume a previous run, adding more iterations to increase confidence.
--threshold <frac>	0.2	Regression threshold (0.2 = flag if 20% slower).
--production-build	—	Build a local bundled package via gulp vscode for comparison against a release baseline.
--no-cache	—	Ignore cached baseline data, always run fresh.
--force	—	Skip build mode mismatch confirmation prompt.
--ci	—	CI mode: write Markdown summary to ci-summary.md (implies --no-cache, --heap-snapshots, --cleanup-diagnostics).
--heap-snapshots	—	Take heap snapshots after each run (slow; auto-enabled in --ci mode).
--gc-object-stats	—	GC deep-dives only. Enables V8 gc_stats tracing (per-type heap object dump on every GC). ⚠️ Corrupts all timing metrics — a major GC landing mid-request adds ~550ms — so never use it for benchmarking. Off by default; prefer heap snapshots for memory analysis.
--cleanup-diagnostics	—	Delete heap snapshots, CPU profiles, and traces to save disk. During runs, only the latest run's files are kept; after comparison, files for non-regressed scenarios are deleted. Auto-enabled in --ci mode.
--setting <k=v>	—	Set a VS Code setting override for all builds (repeatable).
--test-setting <k=v>	—	Set a VS Code setting override for the test build only.
--baseline-setting <k=v>	—	Set a VS Code setting override for the baseline build only.
--verbose	—	Print per-run details including response content.