oatda-compare-models

OATDA Compare Models

Send one prompt to multiple LLM providers in a single parallel request and get back each model's response. Perfect for benchmarking, picking the best model for a task, or evaluating quality / speed / cost tradeoffs.

When to Use

Use this skill when the user wants to:

• Compare outputs from 2+ LLM models side-by-side
• Benchmark models on a specific prompt
• Pick the best model for a particular task before committing to one
• See how different providers (OpenAI, Anthropic, Google, DeepSeek, etc.) handle the same input

Prerequisites

The user needs an OATDA API key. Check in this order:

1. $OATDA_API_KEY environment variable
2. ~/.oatda/credentials.json config file

If neither exists, tell the user:

You need an OATDA API key. Get one at https://oatda.com, then set it: export OATDA_API_KEY=your_key_here

Step-by-Step Instructions

1. Resolve the API key

# Check env var first; if empty, auto-load from credentials file
if [[ -z "$OATDA_API_KEY" ]]; then
  export OATDA_API_KEY=$(cat ~/.oatda/credentials.json 2>/dev/null | jq -r '.profiles[.defaultProfile].apiKey' 2>/dev/null)
fi

# Verify key exists (show first 8 chars only)
echo "${OATDA_API_KEY:0:8}"

If the output is empty or null, stop and ask the user to configure their API key.

IMPORTANT:

• Never print the full API key. Only show the first 8 characters for verification.
• The key resolution script and subsequent curl commands must run in the same shell session. Either run all commands in one session, or chain them (e.g., export OATDA_API_KEY=... && curl ...).

2. Determine models to compare

Ask the user (or infer from context) which models they want. If they don't specify, use this sensible default set:

[
  {"provider": "openai", "modelId": "gpt-5"},
  {"provider": "anthropic", "modelId": "claude-sonnet-4-5-20250929"},
  {"provider": "google", "modelId": "gemini-3-pro-preview"},
  {"provider": "deepseek", "modelId": "deepseek-v4-pro"}
]