training-stability-monitor

training-stability-monitor is a skill for Claude Code, Codex from strikersam/autonomous-ai-agency. It costs 42 tokens per session (802 once invoked), scanned A, original, MIT.

A checklist and command for finding instability during language-model training. Training instability includes problems such as sudden loss increases, exploding gradients, or model outputs collapsing into repetition or nonsense.

In plain words
What is it for?
Reviewing training logs, gradient sizes, learning-rate schedules, batch-size choices, loss plateaus, and checkpoint recovery options.
Why use it?
It helps identify warning signs before a long training run fails and points to possible recovery actions such as changing the learning rate or restoring a checkpoint.

Skill for Claude CodeCodex

Written for no agent in particular: nothing here depends on one.

Good fit Reviewing training logs, gradient sizes, learning-rate schedules, batch-size choices, loss plateaus, and checkpoint recovery options.

Compare 6 skills from other repositories ↓
Install with agentmods
npx agentmods add skills/strikersam/autonomous-ai-agency/training-stability-monitor
Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent
npx skills add strikersam/autonomous-ai-agency --skill training-stability-monitor
Clone the repo
git clone --depth 1 https://github.com/strikersam/autonomous-ai-agency

Made for: Claude Code, Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

agentmods badge for training-stability-monitor

README.md
[![agentmods](https://agentmods.dev/badge/skills/strikersam/autonomous-ai-agency/training-stability-monitor/github.svg)](https://agentmods.dev/skills/strikersam/autonomous-ai-agency/training-stability-monitor)
Your own site
<a href="https://agentmods.dev/skills/strikersam/autonomous-ai-agency/training-stability-monitor"><img src="https://agentmods.dev/badge/skills/strikersam/autonomous-ai-agency/training-stability-monitor/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

agentmods 80×15 button for training-stability-monitor

Your own site · 80×15
<a href="https://agentmods.dev/skills/strikersam/autonomous-ai-agency/training-stability-monitor"><img src="https://agentmods.dev/badge/skills/strikersam/autonomous-ai-agency/training-stability-monitor.svg" alt="Reviewed on agentmods" width="80" height="20"></a>
Per session 42 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.
When invoked 802 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe. Third-party audits
  • NVIDIA SkillSpector pass 7 Sept 2026
How audits are shown
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5.1 $0.00042 $0.00802
Opus 5.5 $0.00017 $0.00321
Sonnet 5.5 $0.00008 $0.00160
Haiku 4.5 $0.00004 $0.00080

Measured 8d ago against content hash b4b979734c0b, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade A, and why

training-stability-monitor scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 8d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

harness/skills/training-stability-monitor/SKILL.md · 89 lines

How it starts

The opening of the file, as written. The whole thing — 89 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Skill: training-stability-monitor

Purpose

Monitor and diagnose LLM/ML training instability — loss spikes, exploding gradients, dead neurons, and learning rate mismatches. Surfaces actionable fixes before they derail a training run.

Trigger

Use when:

  • A training loss curve shows spikes or divergence
  • Gradients are exploding or vanishing
  • Model outputs collapse (repetition, gibberish, empty)
  • You want a pre-flight check before a long training run

What It Does

  1. Scans training logs for loss spike signatures (sudden >2x jump in loss)
  2. Checks gradient norms — flags if norm exceeds configured threshold (default: 1.0)
  3. Validates LR schedule — warns if no warmup is configured, or if LR is too high for model size
  4. Checks batch size vs model size ratio — common source of instability
  5. Detects loss plateau — flat loss for N steps may indicate dead optimizer state
  6. Recommends recovery steps — rollback checkpoint, reduce LR, increase gradient clipping

Usage

/training-stability-monitor [log_file_or_directory] [--threshold 1.0] [--window 100]

Output Format

=== Training Stability Report ===
[PASS/WARN/FAIL] Loss Spike Detection
[PASS/WARN/FAIL] Gradient Norm
[PASS/WARN/FAIL] LR Schedule
[PASS/WARN/FAIL] Batch Size Ratio
[PASS/WARN/FAIL] Loss Plateau

CRITICAL ISSUES: N
WARNINGS: N

Recommended Actions:
1. ...

Key Lessons (from LLM-from-scratch practitioners)

  • Loss spikes are normal but recoverable — the fix is gradient clipping + LR warmup, not stopping the run
  • Gradient norm logging is non-negotiable — without it you're flying blind
  • Warmup is mandatory — jumping to peak LR on step 0 destabilizes attention weights
  • Save checkpoints every N steps — not just at epoch boundaries; spikes need rollback points
  • BF16 > FP16 for stability on modern hardware; overflow is silent in FP16

Integration Points

  • Works alongside debug-tracer for step-level tracing
  • Works alongside insights for surfacing patterns across runs
  • Can feed into session-handoff to carry stability context across sessions

Read the full file on GitHub · 89 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 8d ago Changed · +4 lines · +42 tokens per session b4b979734c0b
  2. 1mo ago First seen · 85 lines · 0 tokens per session scan A 67cf24f26750

Subscribe to this mod's changes

training-stability-monitor is a skill published in the GitHub repository strikersam/autonomous-ai-agency (11 stars, last pushed today), licensed MIT. It adds 42 tokens to every session and 802 once invoked, about $0.0002 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.

Related

Other skills, from other repositories

digital-health-clinical-asr-finetune

Stage 4 of the Clinical ASR Flywheel. Use when priority KER is above 0.3 to run stock NeMo SFT on Parakeet TDT v2 and offline cycle N+1 re-eval. NOT for generic word boosting (use /finetune-asr).

NVIDIA/skills · 72 tokens

digital-health-clinical-asr-eval

Stage 3 of Clinical ASR Flywheel. Score a NeMo manifest, produce the five-section KER leaderboard (by-ipasource diagnostic). Not for ASR auth (/riva-asr).

NVIDIA/skills · 51 tokens

kermt-continue-pretrain

Continue KERMT pretraining on a custom SMILES corpus with a groverbase, cmim, or hybrid checkpoint. Use a local checkpoint or optionally download a pinned Hugging Face model bundle using HFTOKEN if configured. Run containerized training and write model bundles, prepared data, logs, and checkpoints to user-selected…

NVIDIA/skills · 75 tokens

kermt-finetune

Finetune a pretrained KERMT encoder on a labeled CSV. Validate the checkpoint and data, prepare features, and run containerized training. Use a local checkpoint or optionally download a pinned Hugging Face model bundle using HFTOKEN if configured. Write model bundles, prepared data, logs, and trained models to…

NVIDIA/skills · 74 tokens

cdf-curation-and-retrieval

NVIDIA CDF curation and retrieval. Cosmos Curator video and image pipelines (split, dedup, shard, annotate) with filtering, captioning, and embedding. Cosmos Dataset Search ingest, search, and capability qualification. TAO data mining embeddings and neighbor matching. Ships domain event catalogs, including workplace…

NVIDIA/skills · 117 tokens

nemo-mbridge-perf-expert-parallel-overlap

Validate and use MoE expert-parallel communication overlap in Megatron-Bridge, including overlapmoeexpertparallelcomm, delaywgradcompute, and flex dispatcher backends such as DeepEP and HybridEP.

NVIDIA/skills · 56 tokens