online-evals

Config Online Evaluations

Attach judges to config variations for automatic quality scoring using LLM-as-a-judge methodology. Judges evaluate responses and return scores between 0.0 and 1.0.

Prerequisites

• LaunchDarkly account with AgentControl enabled
• API access token with write permissions
• Existing config with variations (use configs-create skill)
• For automatic metric recording and the consolidated judge-result API: Python AI SDK v0.20.0+ or Node.js AI SDK v0.20.0+

API Key Detection

1. Check environment variables - LAUNCHDARKLY_API_KEY, LAUNCHDARKLY_API_TOKEN, LD_API_KEY
2. Check MCP config - Claude: ~/.claude/config.json -> mcpServers.launchdarkly.env.LAUNCHDARKLY_API_KEY
3. Prompt user - Only if detection fails

Core Concepts

What Are Judges?

Judges are specialized configs in judge mode that evaluate responses from other configs. They use an LLM to score outputs and return structured results:

{
  "score": 0.85,
  "reasoning": "Answered correctly with one minor omission"
}

Built-in Judges

LaunchDarkly provides three pre-configured judges:

Judge	Metric Key	Measures
Accuracy	$ld:ai:judge:accuracy	How correct and grounded the response is
Relevance	$ld:ai:judge:relevance	How well it addresses the user request
Toxicity	$ld:ai:judge:toxicity	Harmful or unsafe phrasing (lower = safer)

Completion Mode Only

Judges can only be attached to completion mode configs in the UI. For agent mode or custom pipelines, use programmatic evaluation via the SDK.

Restrictions

• Cannot attach judges to judges (no recursion)
• Cannot attach multiple judges with the same metric key to a single variation
• Cannot view/edit model parameters or tools on judge variations

Workflow

Step 1: Create Custom Judges (Optional)

For domain-specific evaluation, create judge configs:

# Create judge config
curl -X POST "https://app.launchdarkly.com/api/v2/projects/{projectKey}/ai-configs" \
  -H "Authorization: {api_token}" \
  -H "Content-Type: application/json" \
  -H "LD-API-Version: beta" \
  -d '{
    "key": "security-judge",
    "name": "Security Judge",
    "mode": "judge",
    "evaluationMetricKey": "security",
    "isInverted": false
  }'