vllm-ascend-workspace: Skill for Codex

.agents/skills/vllm-ascend-graph-debug/SKILL.md

vllm-ascend-graph-debug is a skill for Codex from maoxx241/vllm-ascend-workspace. It costs 97 tokens per session (3,088 once invoked), scanned A, original, MIT.

A troubleshooting guide for vLLM Ascend, a system for running language models on Ascend hardware. It compares normal execution with graph mode to locate compilation, capture, replay, hanging, or accuracy problems.

In plain words
What is it for?
Use it to create debugging cases, run controlled experiments, compare intermediate tensors, identify the first mismatch, and document the final fix.
Why use it?
It gives a recorded, step-by-step way to find where graph mode first differs from normal execution instead of relying on scattered notes or repeated guesses.

Skill for Codex

Written for Codex: agents/openai.yaml present. Also seen: installed under .agents/ (shared by several agents).

This is maoxx241/vllm-ascend-workspace's own configuration. It tells Codex how to work on vllm-ascend-workspace itself, so it is not a mod to install elsewhere. Copy it as a starting point and replace the rules that are about this project. Everything vllm-ascend-workspace configures →

Reuse

Borrowing it

Nothing to install: this file belongs to maoxx241/vllm-ascend-workspace. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.

Copy the file
curl -O https://raw.githubusercontent.com/maoxx241/vllm-ascend-workspace/main/.agents/skills/vllm-ascend-graph-debug/SKILL.md
Clone the repo
git clone --depth 1 https://github.com/maoxx241/vllm-ascend-workspace

Made for: Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

agentmods badge for vllm-ascend-graph-debug

README.md
[![agentmods](https://agentmods.dev/badge/skills/maoxx241/vllm-ascend-workspace/vllm-ascend-graph-debug/github.svg)](https://agentmods.dev/skills/maoxx241/vllm-ascend-workspace/vllm-ascend-graph-debug)
Your own site
<a href="https://agentmods.dev/skills/maoxx241/vllm-ascend-workspace/vllm-ascend-graph-debug"><img src="https://agentmods.dev/badge/skills/maoxx241/vllm-ascend-workspace/vllm-ascend-graph-debug/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

agentmods 80×15 button for vllm-ascend-graph-debug

Your own site · 80×15
<a href="https://agentmods.dev/skills/maoxx241/vllm-ascend-workspace/vllm-ascend-graph-debug"><img src="https://agentmods.dev/badge/skills/maoxx241/vllm-ascend-workspace/vllm-ascend-graph-debug.svg" alt="Reviewed on agentmods" width="80" height="20"></a>
Per session 97 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.
When invoked 3,088 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe. Third-party audits
  • NVIDIA SkillSpector pass 7 Sept 2026
How audits are shown
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5.1 $0.00097 $0.03088
Opus 5 $0.00048 $0.01544
Sonnet 5 $0.00019 $0.00618
Haiku 4.5 $0.00010 $0.00309

Measured 11d ago against content hash 88bb6088bfc5, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-10, from the pricing page.

Security

Grade A, and why

vllm-ascend-graph-debug scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 11d ago.

The scan reads SKILL.md. This mod also ships 2 executable files (scripts/graph_debug_case.py, tests/test_graph_debug_case.py), listed below but not scanned — reading those needs a real analyzer, not pattern matching.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

.agents/skills/vllm-ascend-graph-debug/SKILL.md · 234 lines

How it starts

The opening of the file, as written. The whole thing — 234 lines — stays where its author put it; the contents beside it link to each section on GitHub.

NPU Graph Debug

用于排查 vLLM Ascend 图模式下的编译、捕获、重放和精度问题。核心方法是:记录已知信息,先定性问题阶段,再控制变量缩小范围;只有范围足够小时,才插入预分配 buffer,通过图内 copy_ 和图外落盘对比 graph/eager 的中间状态。

结构化入口

从仓库根目录使用 scripts/graph_debug_case.py

  1. init 创建 .vaws-local/graph-debug/<case-id>/case.json 和 Run Manifest v1。
  2. 每轮单变量实验后立即用 record 追加假设、预期、观测、结论和下一步。
  3. 需要中间状态对拍时,用 compare 对齐 eager/graph JSONL snapshot 并找到首个分叉。
  4. 修复后用 finalize 同时记录最小复现、原始复现和 instrumentation 清理状态。

按需读取:

  • Behavior contract:case 生命周期、snapshot schema、比较语义和 Run Manifest 集成。
  • Command recipes:可复制的 init、record、compare、finalize 命令。
  • Acceptance:结案前必须逐项满足的验收条件。

工作原则

  1. 每轮实验只改变一个变量,实验前写明假设,实验后记录结论。
  2. 优先证明“已经排除什么”和“问题首次出现在哪里”,避免重复回到已验证路径。
  3. Eager 模式必须先正常;如果 Eager 本身失败,先修 Eager,不进入图模式排查。
  4. 精度排查优先固定随机性和输入;性能、吞吐、并发压力只在问题需要时引入。
  5. 图内只做设备侧、可 capture 的操作;同步、CPU 读回、文件写入统一放到图外。

记录要求

不要使用散落的临时笔记作为唯一事实来源。以 case 目录中的 case.json 为结构化记录;issue 或 PR comment 可以引用它的摘要。每次实验完成后立即运行 record,并保持“已知事实 / 已排除 / 当前嫌疑 / 下一步”与 case 内容一致。

总流程

  1. 建立基线:同输入、同 seed、同采样参数分别跑 Eager 和 Graph。
  2. 固定确定性:启动环境设置 HCCL_DETERMINISTIC=true;model runner 初始化时调用 torch.use_deterministic_algorithms(True)
  3. 阶段定性:先判断问题发生在 compile、capture、replay 还是 accuracy。
  4. 控制变量缩小范围:模型规模、并行策略、请求 shape、capture size、特性开关、部署方式逐项收敛。
  5. 静态审计:检查 graph replay 依赖的 metadata、padding、dummy run 与真实请求路径、自定义算子 capture/replay 约束。
  6. 动态打点:在候选模块插入预分配 debug buffer,图内 copy_,图外 flush。
  7. 对齐比较:按 step/layer/rank/tag 先比统计量,再比局部样本,定位首个分叉点。
  8. 修复验证:关闭 debug 代码和同步逻辑,重跑最小复现和原始复现,更新记录。

阶段定性

阶段 典型现象 先问什么 下一步
Compile 启动、加载或 torch.compile 阶段报错/卡住 是否是 Python/Dynamo/FX/算子前端问题 看堆栈;缩小到具体模块或算子;必要时图外打印
Capture dummy run、capture model 阶段报错/卡住 编译后静态图是否能在固定 shape 下执行 隔离 compile backend 与 graph capture;查同步、CPU 读回、自定义算子
Replay capture 成功,真实请求时报错/卡住 真实请求和捕获 shape/metadata/通信序列是否一致 查 padding、固定地址 metadata、event 配对、rank 间状态
Accuracy graph 输出与 Eager 非预期差异 是良性数值差异还是功能性错误 无 padding 对照;静态审计;中间 tensor 对比

Read the full file on GitHub · 234 lines

Files

What ships with it

6 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 11d ago First seen · 234 lines · 97 tokens per session scan A 88bb6088bfc5

Subscribe to this mod's changes

vllm-ascend-graph-debug is a skill published in the GitHub repository maoxx241/vllm-ascend-workspace (36 stars, last pushed 6d ago), licensed MIT. It adds 97 tokens to every session and 3,088 once invoked, about $0.0005 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

Related

Other skills, from other repositories

gke-ai-troubleshooting-tpu-dynamic-slices-monitoring

Monitors, troubleshoots, and manages GKE TPU Dynamic Slices custom resources. Use when checking TPU slice lifecycle states, troubleshooting slice provisioning failures, validating single-slice or multi-slice (JobSet) workload manifests, or safely patching stuck finalizers and disabling the slice controller. Don't use…

google/skills · 107 tokens

gke-ai-troubleshooting-tpu-vbar-oom

Diagnoses and prevents vbarcontrolagent segfaults, out-of-memory (OOM) errors, and TPU device initialization failures on TPU v6e nodes in GKE caused by race conditions during TPU device resets or high-frequency metrics polling. Use when troubleshooting vbarcontrolagent crashes, memory cgroup OOMs in serial console…

google/skills · 125 tokens

competition-firmware-layout

Internal downstream skill for ctf-sandbox-orchestrator. CTF-sandbox workflow for firmware images, partition tables, boot chains, update packages, extracted filesystems, embedded configs, and device-facing trust boundaries. Use when the user asks to unpack firmware, map partition layout, inspect bootloader or init…

zhaoxuya520/reverse-skill · 110 tokens

doca-flow

Build and debug DOCA Flow applications on supported NVIDIA NICs/DPUs: define match/action pipes, initialize ports and representors, choose forwarding targets, validate pipes before hardware programming, read counters, match the Flow version to the installed DOCA release, and diagnose Flow API errors. Trigger on DOCA…

NVIDIA/skills · 140 tokens

diagnose-driver-install

Diagnose NVIDIA driver installation failures on DeepOps-managed nodes — nvidia-smi errors, "No devices were found", DKMS build failures, or GPU pods crash-looping. Use before reinstalling anything.

NVIDIA/deepops · 47 tokens

catc-troubleshoot

Catalyst Center troubleshooting workflows - device unreachable investigation, client connectivity issues, interface down analysis, site-wide outage triage, wireless roaming problems, integration with pyATS for CLI-level diagnostics. Use when a device is unreachable, a user reports connectivity problems, an interface…

automateyournetwork/netclaw · 74 tokens