monitor-experiment

monitor-experiment is a skill for Claude Code, Codex from EricYuan2007/GPUPlane. It costs 60 tokens per session (926 once invoked), scanned A, original, Apache-2.0.

A monitoring procedure for a running GPU training job, where a graphics processor trains a machine-learning model. It checks progress, convergence, warnings, and serious events at regular intervals.

In plain words
What is it for?
Use it to watch training runs, report healthy progress, inspect logs, cancel poisoned jobs, and respond to critical or warning events.
Why use it?
It helps detect failures such as running out of GPU memory, invalid loss values, overfitting, stalled progress, low disk space, or a disconnected agent, then follows the documented response.

Skill for Claude CodeCodex

Part of the gpu-training plugin — 3 skills, 1 MCP server shipped together

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

agentmods
npx agentmods add skills/ericyuan2007/gpuplane/monitor-experiment
Any agent
npx skills add EricYuan2007/GPUPlane --skill monitor-experiment
Clone the repo
git clone --depth 1 https://github.com/EricYuan2007/GPUPlane

Made for: Claude Code, Codex.

Or install gpu-training, the plugin that ships this one along with the rest of its 3 skills, 1 MCP server.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

agentmods badge for monitor-experiment

README.md
[![agentmods](https://agentmods.dev/badge/skills/ericyuan2007/gpuplane/monitor-experiment.svg)](https://agentmods.dev/skills/ericyuan2007/gpuplane/monitor-experiment)
Your own site
<a href="https://agentmods.dev/skills/ericyuan2007/gpuplane/monitor-experiment"><img src="https://agentmods.dev/badge/skills/ericyuan2007/gpuplane/monitor-experiment.svg" alt="Measured on agentmods" height="20"></a>
Per session 60 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.
When invoked 926 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. Scan, not verified.
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5 $0.00060 $0.00926
Opus 5 $0.00030 $0.00463
Sonnet 5 $0.00012 $0.00185
Haiku 4.5 $0.00006 $0.00093

Measured 3d ago against content hash c924751639b7, method: parsed. Prices are Anthropic first-party input rates as of 2026-08-30, from the pricing page.

Security

Grade A, and why

monitor-experiment scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 3d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

.claude/skills/gpu-training/monitor-experiment/SKILL.md · 70 lines

How it starts

The opening of the file, as written. The whole thing — 70 lines — stays where its author put it; the contents beside it link to each section on GitHub.

monitor-experiment

Watch a training run and intervene on anomalies. The control plane's EventDetector already classifies problems (OOM, loss-NaN, loss-spike, overfitting, GPU-underutilization, disk-low, agent-disconnect); this skill is about consuming those signals and acting, not re-deriving them.

Default watch loop

Poll at minute cadence (not seconds — training is slow and the user is often away). Each tick:

  1. get_run_summary(run_id) — read progress.percent, convergence of the primary metric (trend/improving/stalled), health.warnings.
  2. list_events(run_id=…, severity=critical, limit=20) — anything new since the last tick needs attention. info events (CHECKPOINT_CREATED, EVALUATION_FINISHED) are progress, not problems.

If everything is healthy, report a one-line status to the user and stop — do not spam. Only escalate on warning/critical.

Anomaly runbook

signal (from diagnose_run / events) action
LOSS_NAN (critical) the run is poisoned — cancel_job(confirm=True) after telling the user; then explain_failure for the cause (bad LR, amp, data).
OOM (critical) tail_logs to confirm, then recommend a smaller batch / gradient checkpointing / accumulation; cancel_job and retry_job once the user fixes the config.
OVERFITTING_SUSPECTED don't cancel — flag it; compare_checkpoints to see if an earlier checkpoint is better, and suggest early-stopping.
GPU_UNDERUTILIZED (warning, >10min) likely a data-loader bottleneck or CPU-bound step; tell the user to check num_workers / pin_memory, not a crash.
AGENT_DISCONNECTED / JOB_LOST infra, not the model — check the agent host is up; retry_job once the agent reconnects.
DISK_LOW checkpoints will fail to write — tell the user to clean checkpoints/ or move the run.

Failure triage

When get_run_summary shows status=FAILED (or an OOM/NaN event fires):

  1. explain_failure(run_id) — structured likely_cause + suggested_fixes. It already read the exit code, failure reason, classified events, and the log tail. Read its suggested_fixes before doing anything else.
  2. tail_logs(job_id, stream=stderr, tail=200) only if explain_failure's cause is "unknown" — to grep the traceback yourself.
  3. Present the cause + fixes to the user. Do not auto-retry after an OOM or NaN without the user fixing the config — it will fail the same way.
  4. Once fixed, retry_job(job_id) (same id, new attempt) and resume watching.

Read the full file on GitHub · 70 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 3d ago First seen · 70 lines · 60 tokens per session scan A c924751639b7

Subscribe to this mod's changes

monitor-experiment is a skill published in the GitHub repository EricYuan2007/GPUPlane (0 stars, last pushed 8d ago), licensed Apache-2.0. It adds 60 tokens to every session and 926 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.

Related

Other skills, from other repositories

exploratory-data-analysis

Perform bounded, local exploratory analysis of explicitly supported scientific files. Use for redacted CSV/TSV/JSON profiles; optional NumPy, HDF5, FASTA/FASTQ, and basic image metadata inspection; missingness/leakage audits; outlier and transformation sensitivity; and rigorous EDA report scaffolds. Other domain…

K-Dense-AI/scientific-agent-skills · 83 tokens

nature-statistics

Audit, revise, or draft manuscript statistical reporting for Nature / high-impact journal submissions. Use when the user asks to check statistical analysis sections, p values, confidence intervals, sample size, biological versus technical replicates, randomization, blinding, multiple-comparison correction, model…

Yuan1z0825/nature-skills · 139 tokens

evaluating-with-leakage-gates

Evaluate an OpenMed de-identification or clinical NER model against the leakage-first release gates G1a through G8, which gate releases on residual PHI leakage rather than on F1. Use when the user wants to run the OpenMed eval harness on a synthetic golden set, decide whether a de-id model is RELEASABLE or…

maziyarpanahi/openmed · 158 tokens

mapping-to-snomed

Maps clinical concept spans extracted by OpenMed to SNOMED CT concepts through a USER-SUPPLIED terminology server (the user's own Ontoserver, Snowstorm, or UMLS/UTS), never a bundled vocabulary. Use when the user wants to code findings, disorders, procedures, body structures, or substances to SNOMED CT, run an ECL…

maziyarpanahi/openmed · 205 tokens

mixed-precision

Use FP16/BF16 mixed precision to accelerate training and reduce memory. Use when optimizing GPU performance.

aiming-lab/AutoResearchClaw · 25 tokens

indication-dossier

Build a source-backed biomedical indication dossier. Use when a research task asks for disease biology, target rationale, patient segmentation, biomarkers, trials, drugs, competitive landscape, or translational evidence.

companion-inc/feynman · 43 tokens