custom-client

Custom Client + Agent

Use this path to evaluate models from a different provider or write a custom tool-calling agent. STATE-Bench does not ship third-party adapters — provider integration is user-owned.

Custom runs use two extension points together:

• BaseLLMClient (state_bench/client.py) — your provider-specific client wrapper.
• BaseAgent (state_bench/agents/base.py) — your agent loop that calls that client and returns provider-neutral tool requests for the harness to execute.

File layout

Place your implementations under repo-root extension folders:

agents/
  my_agent.py
clients/
  my_client.py

Class names must be unique under their folder trees. The harness loads --agent-class from agents/ and --agent-client-class from clients/.

Provider environment variables

STATE-Bench does not interpret provider env vars; your BaseLLMClient.from_env() does. Add whatever your provider needs to .env:

MY_PROVIDER_API_KEY="<your provider api key>"
MY_PROVIDER_MODEL="<your model name>"

Custom client

Subclass BaseLLMClient and implement from_env(). The base class does not require a specific method shape; your custom agent decides which client methods to call.

# clients/my_client.py
import os

from state_bench.client import BaseLLMClient


class MyLLMClient(BaseLLMClient):
    def __init__(self, api_key: str, model: str):
        self.api_key = api_key
        self.model = model

    @classmethod
    def from_env(cls):
        return cls(
            api_key=os.environ["MY_PROVIDER_API_KEY"],
            model=os.environ.get("MY_PROVIDER_MODEL", "my-model"),
        )

    @property
    def model_name(self):
        return self.model

    def generate(self, *, system_prompt, conversation, tools):
        # Call your provider here. Convert STATE-Bench tool schemas to your
        # provider's format if needed, and return an object your agent understands.
        raise NotImplementedError