Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add GoogleCloudPlatform/vertex-ai-samples --skill quality-flywheelgit clone --depth 1 https://github.com/GoogleCloudPlatform/vertex-ai-samplesWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/googlecloudplatform/vertex-ai-samples/quality-flywheel)<a href="https://agentmods.dev/skills/googlecloudplatform/vertex-ai-samples/quality-flywheel"><img src="https://agentmods.dev/badge/skills/googlecloudplatform/vertex-ai-samples/quality-flywheel/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/googlecloudplatform/vertex-ai-samples/quality-flywheel"><img src="https://agentmods.dev/badge/skills/googlecloudplatform/vertex-ai-samples/quality-flywheel.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector pass
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00128 | $0.01887 |
| Opus 5.5 | $0.00051 | $0.00755 |
| Sonnet 5.5 | $0.00026 | $0.00377 |
| Haiku 4.5 | $0.00013 | $0.00189 |
Grade A, and why
quality-flywheel scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 193 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Quality Flywheel Skill
You are the Quality Flywheel — an expert in GenAI evaluation. Your
mission is to help users evaluate and iteratively improve their GenAI
models and agents using the Google GenAI Evaluation SDK
(google.genai / vertexai).
When to use this skill
- Evaluating GenAI agents or models using
client.evals.evaluate() - Creating synthetic datasets or ingesting session traces
- Selecting, configuring, or writing custom evaluation metrics
- Analyzing rubric verdicts and loss patterns
- Suggesting concrete code/prompt improvements based on eval results
Workflow
Follow this workflow sequentially when assisting users:
Step 0. Setup & Project Initialization
- CRITICAL: Before generating or executing any scripts, obtain the
GCP Project ID and Location (e.g.,
global,us-central1). Check environment variables first (GOOGLE_CLOUD_PROJECT,GOOGLE_CLOUD_LOCATION). If not found, ask the user. - Newer Gemini models may only be available in the
globalregion — uselocation="global"if the user wants to use them.
Step 1. Dataset Creation & Formatting
-
Parse Inputs: Convert user-provided descriptions into the SDK formats (
EvalCase,AgentData,ConversationTurn,EvaluationDataset). See references/dataset_schema.md for the full type hierarchy and examples. -
Single-Turn (Model Eval): Create
EvalCaseobjects withpromptstrings. Useclient.evals.run_inference(model=..., src=dataset)to populate model responses if needed. -
Multi-Turn (Agent Eval): If the user wants to test a multi-turn agent but lacks data:
- Generate Scenarios: Use
client.evals.generate_user_scenarioswith aUserScenarioGenerationConfigspecifyinguser_scenario_count,simulation_instruction, andenvironment_data. - Run Inference: Use
client.evals.run_inferencewith auser_simulator_configto simulate interactions up tomax_turn.
- Generate Scenarios: Use
What ships with it
9 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
- EVAL.yaml 3.8 KB
- references/dataset_schema.md 8.7 KB
- references/failure_patterns.md 5.1 KB
- references/metric_registry.md 9.0 KB
- references/sdk_patterns.md 8.0 KB
- scripts/generate_eval_code.py 9.1 KB runs code
- scripts/parse_adk_traces.py 7.8 KB runs code
- scripts/validate_dataset.py 11 KB runs code
- TEST.md 2.6 KB
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 193 lines · 128 tokens per session scan A 6d059789b5dc
quality-flywheel is a skill published in the GitHub repository GoogleCloudPlatform/vertex-ai-samples (790 stars, last pushed 3d ago), licensed Apache-2.0. It adds 128 tokens to every session and 1,887 once invoked, about $0.0005 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
examples-qa
Verify Instructor behavior through the repository's ./examples/ suite in pass, live record, or hermetic replay mode. Use when running selected examples or the corpus, capturing and reusing recorded LLM HTTP responses, diagnosing hub results, or distinguishing real errors, assertion failures, skipped examples, and…
AI/ML Model Testing
Testing machine learning models including accuracy validation, bias detection, drift monitoring, A/B testing, and model regression testing.
deep-learning-pytorch
Use when writing or changing PyTorch code — model, Dataset/DataLoader, training and evaluation loops, determinism, mixed precision, torch.compile, checkpoints with safetensors, Hugging Face models, ONNX export, and tests that run on CPU.
leakage-safe-feature-engineering
Use when building or changing features, preprocessing, encoders, resampling or train/validation/test splits — keep every fitted transform inside a Pipeline fitted on training folds only, and build time and group features without seeing the future or the same entity twice.
ml-verify
Use when the user wants to verify code, config, or math before running — or proactively before any expensive training job or deployment.
add-llm-evals
Use this when adding evaluation to an LLM/agent app - measuring output quality (correctness, faithfulness, relevance, safety) rather than just watching traces. Trigger on "add evals", "test my prompt", "is my RAG accurate", "catch regressions", "score outputs", or setting up an eval suite in CI. Covers offline (CI)…