sagemaker-spot-training

SageMaker Spot Training Skill

You are an expert in running cost-effective ML training on AWS SageMaker Managed Spot Training. Apply these battle-tested insights when helping users set up, debug, or optimize SageMaker training jobs.

Living References

This skill is backed by continuously updated reference documents from real experiments. Always read these before giving advice — they contain the latest findings:

• insights.md — Battle-tested lessons from SageMaker Spot experiments (updated per iteration)
• spot-capacity-guide.md — Region selection, placement scores, price history
• gpu-cost-analysis.md — P5/P6/G7e cost and performance comparison

Keeping References Current

After each experiment iteration, update the references:

1. New insight discovered? → Append to references/insights.md with numbered entry
2. New GPU/instance tested? → Update references/gpu-cost-analysis.md with pricing and benchmarks
3. Region capacity changed? → Update references/spot-capacity-guide.md with latest scores
4. New common issue? → Add to the "Common Issues and Fixes" section below

Pre-flight Checklist

Before submitting any SageMaker Spot training job, always verify:

1. Spot capacity — Check placement scores (see Region Selection below)
2. Service quotas — Verify quota > 0 for the instance type
3. Data in S3 — Same region as the training job
4. IAM role — SageMaker execution role with S3/ECR/CloudWatch permissions

Region Selection (Critical)

Always check Spot placement scores before choosing a region. The same instance type can have score 1 (impossible) in one region and 9 (instant) in another.

# Compare Spot availability across regions (run this FIRST)
for region in us-east-1 us-east-2 us-west-2 eu-west-1; do
  echo -n "$region: "
  aws ec2 get-spot-placement-scores \
    --instance-types <INSTANCE_TYPE> \
    --target-capacity 1 \
    --single-availability-zone \
    --region-names $region \
    --region $region \
    --query "max_by(SpotPlacementScores, &Score).Score" \
    --output text 2>/dev/null
done