search-benchmark

Search Benchmark Skill

Measure semantic search quality against a deployed AI Registry. Generates a ground truth dataset from the registry's own assets, runs queries against the live API, evaluates results using standard information retrieval metrics (NDCG@10, MRR, Recall@10), and produces a markdown report.

Prerequisites

1. Registry URL - The base URL of the deployed registry (e.g., https://d2xl2zfuhgc4l0.cloudfront.net)
2. JWT Token - A valid admin token in .token file (get from "Get JWT Token" button in registry UI)
3. Registry must have assets indexed - At least some servers, agents, or skills registered

The .token file supports both raw JWT format and the full JSON response from the registry UI.

Input

/search-benchmark [REGISTRY_URL] [TOKEN_FILE]

• REGISTRY_URL - Base URL of the registry to benchmark (default: reads from user or uses http://localhost)
• TOKEN_FILE - Path to the token file (default: .token)

Workflow

Step 1: Check for Existing Ground Truth

Check if a ground truth dataset already exists:

ls tests/fixtures/search_dataset/ground_truth.json 2>/dev/null

If the file exists, report how many queries it contains and ask the user: "A ground truth dataset already exists (N queries). Do you want to use it or generate a new one from this registry?"

• If use existing: skip to Step 2
• If generate new: proceed to Step 1b

Step 1b: Generate Expert Ground Truth

This is NOT a simple programmatic generation. You must deeply analyze the registry's assets and craft queries like a search expert. Follow this process:

1b.1: Fetch all assets as JSON

TOKEN=$(python3 -c "
import json
with open('{TOKEN_FILE}') as f:
    raw = f.read().strip()
if raw.startswith('{'):
    data = json.loads(raw)
    print(data.get('tokens',{}).get('access_token') or data.get('access_token',''))
else:
    print(raw.replace('Bearer ',''))
")

curl -s -H "Authorization: Bearer $TOKEN" "{REGISTRY_URL}/api/servers?limit=2000" > /tmp/servers.json
curl -s -H "Authorization: Bearer $TOKEN" "{REGISTRY_URL}/api/agents?limit=2000" > /tmp/agents.json
curl -s -H "Authorization: Bearer $TOKEN" "{REGISTRY_URL}/api/skills?limit=2000" > /tmp/skills.json