model-capability-proof

Model Capability Proof

Overview

Use this skill before trusting a model listing, launch post, provider doc, benchmark, or config file.

A model being listed is not the same as your agent being able to use it with your account, tools, context window, and runtime path.

When to Use

• Changing an agent's default model or provider.
• Verifying a new model announcement.
• Checking context length or reasoning modes.
• Debugging fallback to the wrong provider.
• Testing whether tool calls, streaming, images, audio, or structured output work.
• Comparing builder and reviewer model stacks.

Capability Dimensions

Check each separately:

1. Model availability: provider accepts the model ID.
2. Account entitlement: your account or subscription can run it.
3. Context length: runtime uses the claimed window, not registry folklore.
4. Reasoning controls: effort flags are accepted and included in request body.
5. Tool use: real client-side or server-side tools work.
6. Streaming: deltas arrive and UI/backend preserves them.
7. Structured output: schemas or JSON mode work if needed.
8. Media: image, audio, or video inputs/outputs work if claimed.
9. Fallback behaviour: no silent fallback to an old provider or smaller model.
10. Gateway/API path: the production agent path works, not just a direct SDK call.

Proof Ladder

Run proofs in this order:

1. Read config and provider registry.
2. Dry-run request construction if the harness supports it.
3. Direct no-tools smoke.
4. Normal agent chat smoke.
5. Tool-capable smoke.
6. Gateway/API smoke.
7. Real task smoke in the intended lane.

Stop at the first failure and label exactly which capability failed.

Report Format

Model:
Provider:
Account/auth path:
Context configured:
Context runtime:
Reasoning configured:
Reasoning request proof:
No-tools smoke:
Tool smoke:
Streaming proof:
Fallback providers:
Gateway/API proof:
Verdict: usable / partially usable / not usable for this lane