spark-training-gotchas

spark-training-gotchas is a skill for Claude Code from wshobson/agents. It costs 63 tokens per session (2,241 once invoked), scanned A, original, MIT.

A checklist for the ten recurring problems that can affect machine-learning training on NVIDIA DGX Spark. It covers startup errors, shared-memory limits, heat, bandwidth, and numerical precision.

In plain words
What is it for?
Use it before or during Spark training to investigate startup failures, out-of-memory errors, performance drops, multi-Spark setups, or choices between FP8 and NVFP4 precision.
Why use it?
It helps catch failures before a multi-hour training run and provides a named check for problems such as software crashes, unexpected memory errors, or mid-run slowdowns.

Skill for Claude Code

Written for Claude Code: shipped in a Claude Code plugin.

Part of the dgx-spark-ops plugin — 3 skills, 1 command, 1 agent shipped together

Good fit Use it before or during Spark training to investigate startup failures, out-of-memory errors, performance drops, multi-Spark setups, or choices between FP8 and NVFP4 precision.

Compare 6 skills from other repositories ↓
Install with agentmods
npx agentmods add skills/wshobson/agents/spark-training-gotchas
About the project

Agentic Plugin Marketplace is a collection of reusable plugins, agents, skills, commands, and rules for coding-agent tools including Claude Code, Codex CLI, Cursor, OpenCode, Antigravity CLI, and GitHub Copilot. It is for developers assembling agentic workflows across multiple harnesses from shared Markdown sources, and the catalogue entries are examples or subsets of those workflow components.

wshobson/agents · 39,476 stars · on GitHub · sethhobson.com

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent
npx skills add wshobson/agents --skill spark-training-gotchas
Clone the repo
git clone --depth 1 https://github.com/wshobson/agents

Made for: Claude Code.

Or install dgx-spark-ops, the plugin that ships this one along with the rest of its 3 skills, 1 command, 1 agent.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

agentmods badge for spark-training-gotchas

README.md
[![agentmods](https://agentmods.dev/badge/skills/wshobson/agents/spark-training-gotchas/github.svg)](https://agentmods.dev/skills/wshobson/agents/spark-training-gotchas)
Your own site
<a href="https://agentmods.dev/skills/wshobson/agents/spark-training-gotchas"><img src="https://agentmods.dev/badge/skills/wshobson/agents/spark-training-gotchas/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

agentmods 80×15 button for spark-training-gotchas

Your own site · 80×15
<a href="https://agentmods.dev/skills/wshobson/agents/spark-training-gotchas"><img src="https://agentmods.dev/badge/skills/wshobson/agents/spark-training-gotchas.svg" alt="Reviewed on agentmods" width="80" height="20"></a>
Per session 63 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.
When invoked 2,241 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe. Third-party audits
  • Socket pass 14 Jul 2026
  • Snyk warn 14 Jul 2026
  • NVIDIA SkillSpector pass 7 Sept 2026
How audits are shown
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5.1 $0.00063 $0.02241
Opus 5 $0.00032 $0.01120
Sonnet 5 $0.00013 $0.00448
Haiku 4.5 $0.00006 $0.00224

Measured 5d ago against content hash 4ba2e6510c32, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-08, from the pricing page.

Security

Grade A, and why

spark-training-gotchas scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 5d ago.

The scan reads SKILL.md. This mod also ships 1 executable file (assets/preflight.sh), listed below but not scanned — reading those needs a real analyzer, not pattern matching.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

plugins/dgx-spark-ops/skills/spark-training-gotchas/SKILL.md · 201 lines

How it starts

The opening of the file, as written. The whole thing — 201 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Spark Training Gotchas

DGX Spark's GB10 chip (Grace Blackwell, SM121, 128GB unified memory, aarch64) has ten recurring failure modes across launch, memory, thermals, bandwidth, and precision. Each is named G1–G10 so it can be checked by number — the numbering is load-bearing for tooling that runs these checks. Read this before a long run, not after hour six.

When to Use This Skill

  • A training run fails to start, with an import error or a segfault that doesn't point at the real cause.
  • A run OOMs while nvidia-smi still shows headroom.
  • Throughput degrades partway through a run that started fine.
  • Before any multi-hour or multi-epoch job on GB10.
  • Wiring two Sparks together, before picking a parallelism strategy.
  • Choosing between FP8 and NVFP4 for a Spark-hosted run.

Common Issues Quick Reference

# Symptom Fix
G1 undefined symbol / segfault cu130 wheel or container
G2 flash-attn wrong backend used skip pip build; monkeypatch on NGC
G3 OOM despite headroom drop page cache
G4 throughput drop / reboot expect ~100W sustained cap
G5 memory-bound step slow budget 180–192 GB/s
G6 cache evicted mid-run one GPU server at a time
G7 NVFP4 slower than FP8 stay FP8 unless sm_121a
G8 playbook fails outright check upstream issues
G9 env breaks after install use a container
G10 2-Spark TP hangs DDP/FSDP only, never TP

The Ten Gotchas

G1: CUDA 12/13 ABI Mismatch

  • SYMPTOM: ImportError: undefined symbol naming a CUDA function, or a segfault on the first .cuda() call.
  • CAUSE: most PyPI wheels link libcudart.so.12; Spark ships CUDA 13. pip never checks CUDA ABI, so it surfaces only at import or first kernel launch.
  • CHECK: references/gotcha-checks.md G1 — the wheel's CUDA build tag.
  • FIX: reinstall from download.pytorch.org/whl/cu130 or use a matched container.

G2: flash-attn — Skip the pip Build, Watch Unsloth's Auto-Detect

Read the full file on GitHub · 201 lines

Files

What ships with it

2 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 5d ago First seen · 201 lines · 63 tokens per session scan A 4ba2e6510c32

Subscribe to this mod's changes

spark-training-gotchas is a skill published in the GitHub repository wshobson/agents (39,476 stars, last pushed yesterday), licensed MIT. It adds 63 tokens to every session and 2,241 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-03.

Related

Other skills, from other repositories

technical-troubleshooting

Provide setup, troubleshooting, and maintenance guidance. Use when the user reports a device that won't power on, connectivity issues, setup questions, overheating, or maintenance concerns.

strands-agents/samples · 38 tokens

llama-cpp

Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware. Use for edge deployment, M1/M2/M3 Macs, AMD/Intel GPUs, or when CUDA is unavailable. Supports GGUF quantization (1.5-8 bit) for reduced memory and 4-10× speedup vs PyTorch on CPU.

davila7/claude-code-templates · 76 tokens

amc-run-rtsp-calibration

Calibrate a new dataset from live RTSP camera streams via the AutoMagicCalib REST API. Use when the user provides RTSP URLs or asks to calibrate live cameras; VIOS records clips, AMC ingests them, then runs calibration.

NVIDIA/skills · 59 tokens

jetson-inference-mem-tune

Pick the serving stack and per-runtime memory flags (vLLM, SGLang, llama.cpp, TensorRT Edge-LLM) for an LLM/VLM workload on any NVIDIA Jetson.

NVIDIA/skills · 50 tokens

returns-policy

Answer return, refund, and warranty questions for electronics. Use when the user mentions returns, refunds, RMAs, warranty coverage, damaged items, or opened packaging.

strands-agents/samples · 36 tokens

geepers-data

Fetch structured data from 17 authoritative APIs — arXiv, Census Bureau, GitHub, NASA, Wikipedia, PubMed, news, weather, finance, FEC, and more — through a single endpoint. Use when you need real data from authoritative sources for research, visualizations, or analysis.

InternLM/WildClawBench · 65 tokens