Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add decodingai-magazine/building-a-coding-agent-from-scratch-course --skill manual-e2e-qagit clone --depth 1 https://github.com/decodingai-magazine/building-a-coding-agent-from-scratch-courseWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/decodingai-magazine/building-a-coding-agent-from-scratch-course/manual-e2e-qa)<a href="https://agentmods.dev/skills/decodingai-magazine/building-a-coding-agent-from-scratch-course/manual-e2e-qa"><img src="https://agentmods.dev/badge/skills/decodingai-magazine/building-a-coding-agent-from-scratch-course/manual-e2e-qa/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/decodingai-magazine/building-a-coding-agent-from-scratch-course/manual-e2e-qa"><img src="https://agentmods.dev/badge/skills/decodingai-magazine/building-a-coding-agent-from-scratch-course/manual-e2e-qa.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector warn
SkillSpector: 3 findings, up to high
These are SkillSpector’s own severities. On a checked sample its high-severity flags on skills were ~96% false positives — a documented command, a public API, a “never do X” rule — so we show them as a caution to read, not a verdict. Why →
- high Privilege Escalation · line 15 Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.Fix: Remove references to credential paths. Use environment variables or secrets managers. For docs, use placeholder paths (e.g., /path/to/config). Never load .env or token files in production code paths.
- high Privilege Escalation · line 19 Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.Fix: Remove references to credential paths. Use environment variables or secrets managers. For docs, use placeholder paths (e.g., /path/to/config). Never load .env or token files in production code paths.
- high Tool Misuse · line 49 Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).Fix: Validate all tool parameters against an allowlist. Reject dangerous parameter values (shell=True, --force, -rf /) and use safe defaults.
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00120 | $0.06043 |
| Opus 5.5 | $0.00048 | $0.02417 |
| Sonnet 5.5 | $0.00024 | $0.01209 |
| Haiku 4.5 | $0.00012 | $0.00604 |
Grade C, and why
manual-e2e-qa scanned grade C with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Recursive force deletehighDestructive command
rm -rf with a variable or a broad path is one typo away from removing the wrong tree.
| **sandbox git token (`SANDBOX_GIT_TOKEN`, both backends)** | `rm -rf .decode/sandbox` (**required** — see below), then `SANDBOX_GIT_TOKEN=<fine-grained-PAT> SANDBOX_MODE=docker decode run --repo <url> "create NOTES.md How it starts
The opening of the file, as written. The whole thing — 68 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Testing E2E
Automated M1 proof: capstone tests/integration/test_milestone1_capstone.py — six-step conversation (read → gated write approve → gated write deny → todo_write → ask_user → web_fetch) through real build_agent() + Runner + render_event + session log + memory write-back; only network boundary swapped (FunctionModel for model, httpx.MockTransport for web tool). Run: make integration-tests or make ci. No API key, no network.
Below: manual e2e vs real Gemini — exercise each surface, then try to break it (adversarial half = Tester's job).
Launch (one env var, no service):
export GEMINI_API_KEY=… # the only required secret (see .env.example); or put it in .env
uv run decode # the REPL: a "> " prompt + a footer hint render
No GEMINI_API_KEY → one friendly stderr line — decode: set GEMINI_API_KEY in your environment or .env to start (see .env.example). — exit non-zero, no traceback (task-004 guard in cli.py).
Per surface: what to type + what "working" looks like.
| Surface | Type this | Working looks like |
|---|---|---|
| Plain chat | what can you do? |
answer streams token-by-token above prompt; prompt pinned at bottom. |
| Read (gated) | read pyproject.toml |
permission? read … prompt; y → panel with numbered file contents; n → model told denied, adapts. |
| Write (gated, approve) | create a file hello.txt that says hi |
permission? write … → y → file appears on disk (cat hello.txt), model confirms. |
| Write (gated, deny) | repeat write, answer n |
file not created (ls hello.txt → absent); model told denied (doesn't pretend it wrote). |
| Bash | run the tests with make unit-tests |
permission? bash … → y → panel with stdout/stderr (truncated past cap); runaway command bounded by bash_timeout_s. |
| Todo checklist | make a 3-step plan to add a CLI flag and track it |
blue tasks panel renders checklist ([ ] / [~] / [x]), re-renders as statuses update. |
| web_fetch | fetch https://example.com and summarize it |
permission? web_fetch … → y → page as Markdown (HTML stripped), model summarizes. |
| ask_user | deploy my app (underspecified) |
model calls ask_user; ask: … question + type your answer: cue; next typed line is the answer, turn resumes. |
| lsp (code intelligence) | where is build_agent defined? |
model calls lsp (definition); auto-allows (read-only, no prompt); answer cites src/decode/agent/factory.py:68:5. Then write a broken bad.py with a syntax error → approved write's result carries appended LSP diagnostics (ty) — fix these: block; model corrects. |
| agent (Explore subagents) | explore how permissions and sandboxing each work across the repo, in parallel |
ONE agent(prompts=[…]) call — one prompt per angle — auto-allows (READ_ONLY — "can only cause reads"); result panel = ONE labelled aggregate, a ## Subagent i — "<prompt>" section per child, in prompt order (no synthesis LLM call — the parent model synthesizes). Child = read-only Explore subagent (read/glob/grep/lsp only — no write/edit/bash/web_fetch). The harness guarantees the parallelism (asyncio.gather inside the tool, not the model volunteering N calls): width cap 6 prompts per call (ModelRetry → "consolidate" above it; empty list → ModelRetry), concurrency capped subagent_max_parallel (default 4 — a 6-wide fan-out runs 4, then 2). Child bounded subagent_max_requests (default 25); reports share the fold budget — each truncated to subagent_result_max_bytes // len(prompts) (default 16000 total, so the fold costs the same at any width). Input guarded pre-spawn: a prompt under 8 words (explore the repo) nags the model back (ModelRetry naming Prompt i ("…") as too terse) and spawns NOTHING — rewrite and it fans out (a nag costs one retry leg; agent registers retries=3, so a stubborn model is coached, not aborted). Output validated: a report that is empty or backed by zero tool calls (child answered from memory) buys that child exactly one re-spawn with a nudge — two attempts, never three — then its section carries The subagent returned no usable report.; a child that raises gets This subagent failed before producing a report. Either way the siblings still fold. Every result ends with the Synthesis Footer — the harness telling the parent to compile the sections into ONE answer: prose + a text diagram (ASCII/box-drawing; Mermaid only for a genuine graph), so a good fan-out answer shows you a diagram in the terminal. Children silent-until-done (no-op event sink), transcripts ephemeral — --resume carries only the one spawn call + the folded aggregate (ADR-0013, ADR-0017). |
| Opik tracing (observability) | export OPIK_API_KEY=<comet-key> (free at comet.com), relaunch uv run decode, run any turn; open it in Opik UI |
on launch Decode - Opik tracing on (project 'decode-local'). prints once, before banner — the project is DERIVED decode-<DECODE_ENV> unless OPIK_PROJECT_NAME is set (ADR-0015 §8), so DECODE_ENV=prod files traces under decode-prod; unset key → byte-identical (zero spans, no network, no line). One REPL turn = one Trace (root Span chat_turn); session's traces = one Thread keyed on session id — gated tool's approve/resume leg + follow-ups ride the same trace (latency honestly includes gate wait). Every LLM + tool call = Span: inputs/outputs, latency, tokens (gen_ai.usage.*), cost for models Opik prices (Gemini yes; OpenRouter/Modal may be tokens-only). Memory write-back + compaction ride the one global instrument_pydantic_ai(); subagent agent(...) child nests inside parent turn's trace, child tokens visible (closes ADR-0013 §9). --resume mints fresh session id → new Thread. Headless: a decode run = one Trace (root span decode_run), Thread = the run's session id; activation surfaces only in LOG — stdout stays exactly the answer (pipe-clean), stderr carries only guards/notices. Opik and Kitaru are independent instrumentation (OTLP vs the recording adapter) and compose — recording a run changes no span (ADR-0019 §6). Evals/experiments = M13 (ADR-0014). |
| decode run (headless) | decode run "list the python files" (separate command, not in REPL) |
agent tool-loops headlessly on a plain asyncio.run around the same build_agent() — every tool inline under bypass, no prompt, ask_user a no-op — prints the answer on stdout (pipe-clean: | pbcopy is safe), exit 0. No durability, no pause, no wait — a crash is a re-run (ADR-0019 §1). RUNTIME_ENABLED=false → one friendly stderr line, non-zero exit, nothing built. A missing TASK argument (and no Worker Task env) → one line naming both ways to supply one. |
| Recording Seam (Kitaru session) | export KITARU_API_URL=https://<workspace>.cloudinfra.zenml.io; export KITARU_AGENT_ID=<agent-uuid>, then decode run "explain this repo" — A/B: re-run with env -u KITARU_AGENT_ID |
recorded run → LOG line [kitaru] recording this run on <url> (agent_id=…, session_name=<run session id>), and uv run kitaru session get <id> shows every LLM + tool call as nodes; the REPL records the same way with session_name = decode session id, so a conversation's turns group. B side (unset): byte-identical run, no kitaru module imported at all. KITARU_API_URL only in .env (never exported) = NOT configured — decode never reads it, the adapter's client does (set -a && . .env && set +a). Degrade: point KITARU_API_URL at a closed port → ONE stderr line [kitaru] not recording this run: … continuing on the bare agent, the run still answers and exits 0 (recording is an observer, never an availability dependency). Hard-fail A/B: the same with KITARU_TASK_ID=<uuid> in the env (a Kitaru Worker Task) → non-zero exit instead, because an unrecorded replay is a lying experiment (ADR-0019 §3). |
| decode run --model (model override) | decode run --model gemini-2.5-pro "list the python files" |
same bypass run; Model Override swaps only the active provider's model id for this run (provider stays LLM_PROVIDER-selected — no cross-provider swap). Answer → stdout, stderr carries only guards/notices. Presence, not correctness — a wrong model id for the provider fails at the first model request. A replay-time model swap is Kitaru's own --override, NOT this flag. |
| Worker replay (baseline) | uv run python scripts/bootstrap_kitaru.py --dry-run then without it; set -a && . .env && set +a && kitaru worker start in a second shell; then kitaru replay create <SESSION_ID> --agent decode@<VERSION> --tool-policy '{"default":{"type":"history","scope":"baseline","on_miss":"error_result"}}' |
--dry-run prints the exact kitaru agent version register argv and registers nothing; a real run adds a version to the EXISTING decode agent (never a second agent). kitaru worker list → one live Worker. kitaru replay get <REPLAY_ID> → status: completed + result_session_id; kitaru session get <RESULT> → origin: replay, the node counts, and inputs = the baseline's own recorded payload ({"input": …} or a bare prompt string — all three shapes resolve to the prompt, anything else hard-fails; ADR-0019 Amendments §1). Isolation is the point: the spawn's cwd is Harness Home ~/.decode-kitaru-worker (sessions, logs, .decode/sandbox clone all land there — docker ps during the run, tail -f ~/.decode-kitaru-worker/.decode/logs/decode.log), and git status in YOUR checkout stays clean. Provider keys come from the Worker's shell, not the run spec (secret_ids: []) — a Worker started without them fails at the first model request. Full runbook: 06_evals_replays.md. |
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago Changed 9844d93a3917
- 1mo ago Changed · +7 lines · +8 tokens per session 2de43090808d
- 1mo ago Changed · +4 tokens per session 0a36d81abb92
- 1mo ago First seen · 61 lines · 108 tokens per session scan C ef49f4c0a433
manual-e2e-qa is a skill published in the GitHub repository decodingai-magazine/building-a-coding-agent-from-scratch-course (683 stars, last pushed yesterday), licensed Apache-2.0. It adds 120 tokens to every session and 6,043 once invoked, about $0.0005 per session on Opus 5.5. A static security scan graded it C with 1 finding (recursive force delete). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
write-e2e-test
Write an end-to-end (e2e) test for vix. Use when the user asks to add, create, or modify an e2e test that drives the real vix TUI and vixd daemon against the mock LLM server.
e2e-testing
Guide for running end-to-end tests of the Qwen Code CLI, including headless mode, MCP server testing, and API traffic inspection. Use this skill whenever you need to verify CLI behavior with real model calls, reproduce user-reported bugs end-to-end, test MCP tool integrations, or inspect raw API request/response…
terminal-capture
Automates terminal UI screenshot testing for CLI commands. Applies when reviewing PRs that affect CLI output, testing slash commands (/about, /context, /auth, /export), generating visual documentation, or when 'terminal screenshot', 'CLI test', 'visual test', or 'terminal-capture' is mentioned.
tmux-real-user-testing
A real-user test workflow for Qwen Code's terminal interface, using tmux to drive the program and capture readable screen snapshots. tmux is a tool for running and observing terminal sessions.
agent-reproduce-align
Use after a Codex or Claude Code feature has been implemented in Qwen Code to run the selected reference agent and Qwen Code under the same scenario, capture HTTP and terminal traces, compare request bodies, tool/function schemas, outputs, and iterate until the reproduced behavior is close enough.
qa-testing
Verify your work by actually operating the app or website you changed, instead of assuming it works. Strongly recommended whenever you build, modify, or debug a web app, website, or desktop GUI app. Drive real browsers with the agent-browser CLI and native desktop apps with the cua-driver CLI. These are installed on…