checkpoint-strategy

Skill: checkpoint-strategy

Purpose

Design and validate checkpointing strategies for LLM training runs. One of the most painful lessons from building LLMs from scratch: losing 12 hours of compute to a spike you can't roll back from.

Trigger

Use when:

• Starting a new training run (set checkpointing policy upfront)
• After a loss spike — determine best checkpoint to resume from
• Evaluating storage vs safety tradeoffs for checkpoint frequency
• Migrating training across hardware or cloud providers

Background

From practitioners:

"Checkpointing feels like overhead until you need it. Then it's the only thing that matters. Save more often than you think you need to, in at least two formats."

Key lessons:

1. Step-based checkpointing > epoch-based — LLM epochs are often thousands of hours
2. Save optimizer state — resuming without optimizer state restarts Adam's momentum from zero
3. Rolling window — keep last N checkpoints, not just the best; spikes happen after "best"
4. Two formats — full checkpoint for resuming, sharded for fast loading at inference
5. Validate before deleting — always verify a checkpoint loads before removing older ones

Usage

/checkpoint-strategy [total_steps] [step_duration_seconds] [storage_budget_gb] [--model_size_gb N]

Checkpoint Policy Templates

Conservative (Recommended for First Runs)

checkpoint_policy:
  frequency: every_500_steps
  keep_last: 10
  keep_milestone: [1000, 5000, 10000, 25000, ...]  # powers of 2 * 1000
  save_optimizer: true
  save_formats:
    - full_state_dict    # for resuming
    - safetensors        # for inference/sharing
  validate_on_save: true
  async_save: true       # don't block training

Aggressive (Long Runs with Stable Training)

checkpoint_policy:
  frequency: every_2000_steps  
  keep_last: 5
  keep_milestone: [10000, 50000, 100000]
  save_optimizer: true
  emergency_save_on: 
    - loss_spike_detected    # triggers immediate save before rollback
    - gradient_norm_exceeded