ai-gateway AGENTS.md

AGENTS.md

Project Overview

Ferro Labs AI Gateway is a high-performance, open-source AI gateway written in Go. It acts as a unified routing layer between applications and 30 LLM providers (OpenAI, Anthropic, Gemini, Mistral, etc.), offering smart routing, plugin middleware, and API key management — all with an OpenAI-compatible API and transparent pass-through proxy.

• Module: github.com/ferro-labs/ai-gateway
• Go version: 1.25+
• License: Apache 2.0

Current Development Snapshot

• 30 provider subpackages — each provider lives in providers/<id>/<id>.go with its own test file, one per ProviderEntry in providers/providers_list.go. (providers/core/ and providers/capabilities/ are shared code, not providers.) No root-level constructor shims remain.
• Unified factory — providers/factory.go holds types/constants; providers/providers_list.go holds all built-in ProviderEntry records. Auto-registration via AllProviders() means main.go never needs editing for new providers.
• providers/core/ split — interfaces in contracts.go; shared types split into chat.go, stream.go, embedding.go, image.go, model.go, constants.go, errors.go.
• Single source of truth for name constants — providers/names.go re-exports NameXxx from each subpackage's const Name.
• Model discovery — providers/core exposes DiscoverOpenAICompatibleModels/DiscoverModelsWithHeaders for live /models enumeration, shared by many OpenAI-compatible providers (fireworks, xai, moonshot, nvidia-nim, novita, …).
• Provider coverage — OpenAI, Anthropic, Gemini, Groq, Bedrock, Vertex AI, Hugging Face, Cerebras, Cloudflare, Databricks, DeepInfra, Moonshot, Novita, NVIDIA NIM, OpenRouter, Qwen, SambaNova, and more.
• Built-in OSS plugins — word filter, max token, response cache, request logger, rate limit, budget.
• Admin API — dashboard, key management, usage stats, request logs, config history/rollback (internal/admin/handlers/).
• Metrics — Prometheus metrics exposed at /metrics (pkg/metrics/).
• Circuit breaker — per-provider circuit breaker in pkg/circuitbreaker/.
• Observability (v1.1.0) — OpenTelemetry tracing. Public observability/ package (stable Provider/Span/Exporter/Event seam + gen_ai.*/ferro.* attribute constants); internal/otel/ wires the OTLP exporter, W3C propagation, and a custom IDGenerator that unifies the OTel trace_id with the logging trace ID / X-Request-ID; internal/redact/ redacts error messages. Defaults to a zero-allocation NoOp when no OTLP endpoint and no exporter are configured.
• Capability matrix (v1.2.0) — providers/capabilities/matrix.go is the single source of truth for which OpenAI chat parameters each provider can express. Providers no longer keep private supported-parameter lists; core.EnforceUnsupportedParams reads the matrix and applies compatibility.on_unsupported_param (warn | drop | reject). Served by GET /v1/capabilities.
• Conformance suite (v1.2.0) — test/conformance/ builds every provider through its ProviderEntry, points it at an httptest stub returning that provider's native payload, and asserts the translated core.Response. No build tag, no network: it runs with make test. TestConformanceCoverage fails on any provider with neither a fixture nor an allowlist reason.
• Plugin failure policy (v1.2.0) — a plugin that denies (Context.Reject) and a plugin that breaks (error/panic) are distinct: RejectionError keeps its 4xx/429, FailureError is a 500. Logging and metrics plugins fail open; guardrail, auth, ratelimit, transform, and unknown types fail closed. Reject is honoured for every type. The type that decides this is the one the plugin reports from Type(), not plugins[].type in config.
• Env references (v1.2.0) — internal/envref resolves ${VAR} at component construction, never at config load, so the Config never carries a materialised secret into the config-history store, GET /admin/config, or a rollback. A bare $ is data ($100, pa$$w0rd survive); an undefined variable is an error.
• Per-target concurrency (v1.2.0) — targets[].concurrency bounds in-flight requests per provider with a queue; overflow returns ErrProviderSaturated → 429. The limiter decorates the provider at the call site (innermost, circuit breaker outermost) and never forges capability interfaces.
• Health split (v1.2.0) — /livez (process alive) and /readyz (ready to serve) replace the single /health semantics; /health is retained.
• Plugin short-circuiting (Unreleased) — Context.Skip is removed; code that sets or reads it no longer compiles. Context.SkipProvider replaces it and skips the provider call only: every remaining before_request plugin and the whole after_request stage still run, so a response-cache hit can no longer disable a guardrail, rate limiter or budget behind it. Only a rejection or a plugin failure ends the before_request stage, and it ends that stage alone — on_error still runs, so a request denied by policy is still recorded. SkipProvider stays set through after_request as a fact (cache-served vs provider-served) that cost recording keys off.
• Request attribution (Unreleased) — Context.Target names the routing target a request used: the virtual key that served it, or on a failure the last one attempted, empty when none was attempted (denied by a plugin, no target serves the model, or served from cache). It exists because a failure has no response to read a provider from, which left on_error rows naming no provider. Request-log rows also carry api_key_id — the credential's opaque id, never the credential — so usage is attributable, including a request served from cache, which reaches no provider but is still consumed by a credential. (A cached response is only ever served back to the credential that primed it — the response cache keys on api_key_id — so a row's credential is the one that made the call, not the one that filled the cache.) A cache-served request is priced at a known zero rather than as though the provider had been called.
• Unified request pipeline (Unreleased) — chat, streaming, embeddings and image generation all route through routeTargets in gateway_pipeline.go. Retry, circuit breaker, per-target concurrency, error classification, metrics and request logging live there once, so the four surfaces cannot drift. targets[].retry is honoured under every routing mode, not only fallback.
• One completion ceiling (Unreleased) — max_tokens and max_completion_tokens are resolved to a single value at every entry point, max_completion_tokens winning when both are present, so a guardrail and the provider read the same number. See Completion length.
• Declared models (Unreleased) — targets[].models lets an operator name the models a target serves, joined into the routing index alongside the catalog and live discovery and advertised by /v1/models. It is provider-agnostic and additive: it needs no /models endpoint, works offline, and never narrows what a target already serves. See Declared models.
• Strategy validation at load (Unreleased) — every strategy is validated by config.ValidateConfig, so an invalid one is a ferrogw validate / startup error rather than a request-time 500. Unknown conditions[].key and content_conditions[].type values are load errors; weight: 0 means zero traffic, a negative weight is an error, and an all-zero weight set is an error.