Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add agents/uipath/coder_eval/codexgit clone --depth 1 https://github.com/UiPath/coder_evalWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00035 | $0.03462 |
| Opus 5 | $0.00017 | $0.01731 |
| Sonnet 5 | $0.00007 | $0.00692 |
| Haiku 4.5 | $0.00003 | $0.00346 |
Grade A, and why
CODEX scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured yesterday.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 316 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Running OpenAI Codex in Coder Eval
Overview
Coder Eval can run OpenAI's Codex as the agent under evaluation, via the official Codex SDK. The CodexAgent mirrors the structure of ClaudeCodeAgent and plugs into the same sandbox, scoring, and telemetry pipeline — set agent.type: codex in a task and the rest of the framework works unchanged.
Setup
1. Install the Codex SDK
Install coder-eval with the codex extra:
pip install 'coder-eval[codex]'
This installs:
openai-codex- The official Codex Python SDK (from PyPI)openai-codex-cli-bin- Platform-specific Codex CLI binaries (pulled in transitively)
2. Authentication
Codex requires authentication. Options:
# Option 1: API Key (direct)
await codex_client.login_api_key("your-api-key")
# Option 2: ChatGPT (interactive)
await codex_client.login_chatgpt()
# Option 3: Device Code Flow
await codex_client.login_chatgpt_device_code()
CodexAgent.start() calls login_api_key automatically when CODEX_API_KEY is present in the environment. Without a key it falls back to any existing ChatGPT login. (Only CODEX_API_KEY is read — not OPENAI_API_KEY/AZURE_OPENAI_API_KEY; point CODEX_API_KEY at whichever endpoint's key you use.)
Endpoint routing
| Env var | Purpose |
|---|---|
CODEX_API_KEY |
Auth key/token for the selected endpoint (required for headless runs). |
CODEX_BASE_URL |
Route to a custom endpoint. Unset → standard OpenAI platform (api.openai.com). Set → custom provider (gateway or Azure). |
CODEX_MODEL |
Fallback model when agent.model is unset. On Azure this is the deployment name. |
CODEX_API_VERSION |
Azure only: the required api-version query param. Leave unset for OpenAI/gateways. |
Standard OpenAI: leave CODEX_BASE_URL unset, set CODEX_API_KEY to an OpenAI sk-… key and CODEX_MODEL to a Codex/Responses-capable model.
Azure OpenAI:
CODEX_BASE_URL=https://<your-resource>.openai.azure.com/openai
CODEX_API_VERSION=2025-04-01-preview # required by Azure
CODEX_MODEL=<your-deployment-name> # deployment, not the base model id
CODEX_API_KEY=<azure-openai-key>
This registers a custom Codex model provider (base_url + env_key=CODEX_API_KEY + query_params={api-version} + wire_api=responses). The Codex CLI only supports the Responses wire API (it rejects wire_api=chat as "no longer supported"), so the protocol is fixed. If your Azure deployment requires the key in an api-key header rather than Authorization: Bearer, that needs an additional provider http_headers/env_http_headers entry — open an issue if you hit that.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- yesterday First seen · 316 lines · 35 tokens per session scan A 6924b179cf82
CODEX is an agent published in the GitHub repository UiPath/coder_eval (119 stars, last pushed 3d ago), licensed Apache-2.0. It adds 35 tokens to every session and 3,462 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other agents, from other repositories
callbacks
Callbacks provide hooks into the lifecycle of an AnyAgent execution. Using callbacks, you can monitor, control, and extend agent behavior without modifying the core underlying agent logic.
design-interaction
Interaction Designer on the Atlas bench. Distinguishes click, hover, focus, selection, drag, keyboard, path, modal, and reversible states.
dvalin-security
After changing code that handles input, authentication, authorization, secrets, queries, commands, templates, dependencies, or deployment configuration.
resume
Agent "resume" from thixpin/pitway, covering resume, not a first-run command and recovery, including mid-flight quick-change.
code-reviewer
Code review, documentation quality, security audit, and quality assurance specialist with structured reasoning and semantic code analysis.
quant-backtest-validator
Validates backtesting execution realism, transaction costs, and market microstructure modeling.