nemo-mbridge-perf-moe-vlm-training

MoE VLM Training

Stable docs: @docs/training/moe-optimization.md Card: @skills/nemo-mbridge-perf-moe-vlm-training/card.yaml

FSDP vs 3D Parallel

Approach	Strength	Best fit
FSDP	Simplest path to a working multimodal run	first bring-up, memory-first tuning, awkward PP boundaries
3D parallel	Higher ceiling after tuning	stable models with a clean PP layout and time for deeper sweeps

For MoE VLMs, the practical workflow is usually:

1. get the first reliable run with FSDP
2. stabilize real-data input, recompute, and memory behavior
3. move to 3D parallel only if the throughput headroom is worth the extra work

Rounded Findings From Recent VLM Runs

Qwen3-VL class models

The main patterns were consistent across the tracker:

• FSDP on GB200-class systems can already reach healthy high-teens utilization with a comparatively simple setup
• B200 FSDP runs are viable, but more sensitive to recompute choice and frozen vision settings
• 3D parallel can recover to a similar or better operating point, but only after tuning MBS, recompute, and the real vision path together

Real data vs mock data

Mock-data VLM runs are not trustworthy performance proxies. In the experiments, image-free mock runs looked closer to "roughly twice as fast" than "slightly optimistic" when compared with real multimodal input.

Use real or realistic image payloads before drawing any conclusion about VLM throughput.

Smaller multimodal MoE runs

The smaller Qwen3.5-style multimodal experiments reinforce the same lessons:

• HybridEP is a solid default on GB200
• TE-scoped CUDA graphs help once the training loop is stable
• larger MBS can pay off, but only if the vision encoder does not become the next bottleneck

Decision Guide

Choose FSDP when

• you are bringing up a new VLM for the first time
• the model has awkward stage boundaries across embedding, vision, and decoder
• memory fit matters more than absolute throughput
• you may freeze the vision stack during decoder-focused tuning