openrouter-provider-ranking

OpenRouter Provider Ranking

Rank the endpoint providers of a single OpenRouter model slug for a specific workload. Never call the order "globally optimal": it is optimal only against the stated constraints, weights, token profile and available telemetry.

Mandatory rules

1. Apply hard constraints first, score second. Never let high TPS or a low price compensate for an incompatibility.
2. For tool calling default to native-exacto: the :exacto model suffix, no provider.sort, no provider.order. Your own score in that mode is diagnostic, not a replacement for OpenRouter's private telemetry.
3. Use manual only when the user requires a deterministic order, when your own production telemetry outweighs Exacto, or when a failover chain has to be pinned explicitly.
4. Never combine :exacto with provider.sort — an explicit sort wins. Never add provider.order in native-exacto mode.
5. Price on the real token profile, not on headline input/output rates. Include cache read/write, per-request fees, conditional pricing overrides and the observed cache hit rate.
6. Do not invent missing Exacto, benchmark, cache or tool-success metrics. Apply a conservative prior and an uncertainty penalty, and name the gaps explicitly.
7. For multi-turn workloads pass a stable session_id. Remember that a manual provider.order disables OpenRouter's sticky provider routing, and session_id does not restore it.
8. Never store the API key in the skill, a config, a log or the output JSON. Read it only from OPENROUTER_API_KEY or the environment variable the user names.
9. A catalogue field is a hypothesis; a measurement is a fact. Do not hand back a ranking as the answer: always propose the minimal verification of the top candidates against the constraints that were asked for (step 5), and say plainly whether it was run.
10. The comparison row is endpoint × reasoning effort, not the endpoint. Changing effort moves price, latency and quality more than changing provider does — and "none" means "omit the field", not "no reasoning".
11. Compute price from components on the token profile (in, out, cache_read, cache_write, per-request fee). Headline and effective price swap places: the most expensive rate card produced the lowest cost per step at a 99.996 % cache hit.
12. Measure cache, latency and reliability on the profile that will ship. Cache hit on a repeated identical prompt (99 %) and in an agent run (68 %) are different numbers; TTFT on a short prompt is 4-6x optimistic; and fitness for a single request does not predict behaviour under sequential load.