Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/awslabs/agent-plugins/model-evaluationnpx skills add awslabs/agent-plugins --skill model-evaluationgit clone --depth 1 https://github.com/awslabs/agent-pluginsWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00065 | $0.01186 |
| Opus 5 | $0.00032 | $0.00593 |
| Sonnet 5 | $0.00013 | $0.00237 |
| Haiku 4.5 | $0.00006 | $0.00119 |
Grade A, and why
model-evaluation scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 111 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Model Evaluation
Generate code that evaluates a SageMaker model.
Prerequisites
- The SDK environment has been verified (SDK version, region, execution role). If not done, activate the
sdk-getting-startedskill first.
Principles
- One thing at a time. Each response advances exactly one decision. Never combine multiple questions in a single turn.
- Confirm before proceeding. Wait for the user to agree before moving to the next step.
- Don't read files until you need them. Only read reference files when you've reached the step that requires them.
- Don't ask what you already know. If the answer is in conversation history, workflow_state.json, plan.md, or any file you've already read — use it. Confirm if unsure, but don't re-ask.
- No narration. Share outcomes and ask questions. Keep responses short.
- No repetition. If you said something before a tool call, don't repeat it after.
Scope
This skill supports the evaluation feature for SageMaker Serverless Model Customization. It can evaluate any base or fine-tuned model supported by SageMaker serverless model customization — both OSS models (Llama, Mistral, Qwen, etc.) and Nova models.
Tell the user when the skill is activated:
"I can help evaluate any base or fine-tuned model supported by SageMaker serverless model customization."
If the user requests help evaluating a model that isn't supported by SageMaker serverless model customization, explain that it is not supported by this skill.
Evaluation Types
There are two evaluation types:
- LLM-as-Judge — an LLM grades your model's responses. (OSS models only — not supported for Nova.)
- Custom Scorer — programmatic evaluation via Lambda function (includes built-in math and code scorers). Works with both OSS and Nova models.
Workflow
Step 1: Determine evaluation type
Do you already know which evaluation type to use?
Check conversation history, plan.md, workflow_state.json, or anything else you've already read.
What ships with it
14 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
- code_templates/custom_scorer_evaluator.py 2.5 KB runs code
- code_templates/llmaaj_evaluator.py 2.4 KB runs code
- references/code_output_guide.md 3.3 KB
- references/create-reward-function.md 3.0 KB
- references/custom-lambda-scorer.md 4.4 KB
- references/custom-scorer-evaluation.md 10 KB
- references/evaluation-type-guide.md 8.0 KB
- references/llmaaj-builtin-evaluation.md 5.0 KB
- references/llmaaj-custom-evaluation.md 2.3 KB
- references/llmaaj-evaluation.md 14 KB
- references/supported-judge-models.md 2.5 KB
- scripts/nova_reward_function_source_template.py 13 KB runs code
- scripts/reward_function_source_template.py 9.3 KB runs code
- scripts/validate_custom_metrics.py 3.6 KB runs code
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2d ago First seen · 111 lines · 65 tokens per session scan A 66e66b73d199
model-evaluation is a skill published in the GitHub repository awslabs/agent-plugins (876 stars, last pushed 5d ago), licensed Apache-2.0. It adds 65 tokens to every session and 1,186 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
redshift-support-specialist
Amazon Redshift domain expertise for query optimization, operational reviews, and cost optimization on provisioned clusters and Serverless workgroups. Use when a user asks about Redshift query tuning, slow queries, disk spill, distribution/sort key issues, a Redshift health check or operational review, or Redshift…
database-rds-devops
Database-level data-plane diagnostics for Aurora MySQL and Aurora PostgreSQL. Executes predefined read-only health check queries via RDS Data API to analyze buffer pool, connections, locks, replication, storage, performance, and index efficiency. Requires the rds-aidba MCP server for database-internal access beyond…
eks-operation-review
Comprehensive Amazon EKS operational review aligned with the AWS EKS Best Practices Guide. Use this skill when a user asks to review, audit, or assess EKS clusters for best practices compliance, operational readiness, security posture, cost optimization, reliability, networking, scalability, or upgrade readiness.…
investigation-cost-guardrail
Cost guardrail for AWS DevOps Agent that covers ALL AWS services and native agent tools. Before the agent makes any paid API call, this skill estimates cost, enforces budgets per investigation, detects expensive operations across all services (Athena queries, S3 scans, DynamoDB scans, SageMaker inference, PromQL…
cis-aws-foundations-3.2.3
Ensure that RDS instances are not publicly accessible.
bedrock-adoption-readiness
Amazon Bedrock production readiness assessment covering IAM governance, data retention (ZDR), quota and capacity headroom, and operational observability across Standard Bedrock and Mantle surfaces. Use this skill when a user asks to review Bedrock readiness, assess Bedrock security posture, evaluate quota headroom…