tool-audit

Tool Definition Audit Skill

This skill performs a systematic audit of MCP tool definitions to identify potential sources of confusion for agent decision-making.

Parameters Required

The user will provide the JSON file path in their request. If not provided, ask for:

• tools_json_path: Path to the JSON file containing tool definitions (e.g., tools.json)

Expected JSON Format

The skill expects a JSON structure with:

{
  "active": {
    "server-name": [
      {"name": "tool_name", "description": "Tool description"}
    ]
  },
  "deactivated": ["server-list"]
}

Note: Some servers may have ERROR entries (authentication/connectivity failures) that should be excluded from real tool counts.

Audit Process

Count active servers and tools

active_servers = len(data['active']) total = 0 for server, tools in data['active'].items(): count = len(tools) total += count print(f'{server}: {count} tools') print(f'TOTAL: {total} tools') print(f'Active servers: {active_servers}')

Count deactivated servers

deactivated = data.get('deactivated', []) deactivated_count = len(deactivated) print(f'Deactivated servers: {deactivated_count}') if deactivated_count > 0: print(f'Deactivated list: {deactivated}')

- **NEVER estimate** based on file size, line count, or manual counting
- **ALWAYS use Python** to count tools and servers programmatically
- Count both active AND deactivated servers
- Iterate through each server in "active" section
- Count tools per server: `len(tools_array)`
- Count deactivated servers: `len(data.get('deactivated', []))`
- Identify ERROR tools (name starts with "ERROR" or description contains "HTTP Error")
- Calculate: Real tools = Total entries - ERROR tools
- Verify total by summing all server counts
- Extract all tool names and descriptions from active servers
- Create inventory with server/product associations
- Calculate statistics:
- Total active servers: `len(data['active'])`
- Total deactivated servers: `len(data.get('deactivated', []))`
- List deactivated server names if any exist
- Total real tools (excluding ERROR placeholders)
- Tools per server breakdown with accurate percentages
- Description length distribution
- Quality score distribution
- **VALIDATION**: Before proceeding, verify that:
- Sum of all server tool counts equals reported total
- Percentages add up to 100%
- No server has 0 tools unless it's truly empty or has ERROR
- Deactivated server count matches the actual list length
</Step>