toolset-design

Toolset Design Guide

This skill codifies the methodology for designing MCP tools and toolsets in this repository. The core principle is eval-first, tools-second: prove the need before writing the code.

---

Phase 1: Validate the need with evals

Before writing any toolset code, write eval tasks that represent what a user would actually ask an LLM to do in the target domain. This serves two purposes:

1. Baseline: Run the evals with only the existing toolsets enabled (core, config, etc.). If the LLM can already accomplish the tasks using pods_exec, resources_list, or other generic tools, you may not need dedicated tools at all.

2. Gap identification: The tasks where the baseline fails (or produces poor results) reveal the actual gaps that new tools should fill.

How to write eval tasks

Create tasks under evals/tasks/<domain>/ using the mcpchecker/v1alpha2 format. See existing tasks in evals/tasks/ for examples at different difficulty levels.

Design tasks that represent real user workflows, not tool-shaped requests:

# BAD: This is testing a tool, not a workflow
prompt:
  inline: Run helm list --all-namespaces

# GOOD: This is testing what a user would actually ask
prompt:
  inline: What Helm releases are deployed across the cluster?

# BETTER: This tests whether the LLM can solve a real problem
prompt:
  inline: >
    The application in namespace "payments" was working yesterday but
    is failing after a recent Helm upgrade. Investigate what changed
    in the latest release and identify the issue.

Task difficulty should span the range:

• Easy: Simple queries (list resources, show config)
• Medium: Filtered/analyzed output (find specific entries matching a pattern, correlate data across sources)
• Hard: Multi-step diagnosis (investigate root cause, identify misconfiguration, suggest fix)

Running the baseline

# Run evals with only core+config toolsets to establish baseline
mcpchecker check evals/core-eval-testing/<agent>/eval-core.yaml --label-selector suite=<your-domain>