ox: Skill for Claude Code

Draft firegoat evals

npx -y firegoat@latest eval runs the user's own Claude Code, with none of their config, in a fresh empty folder outside the repo (~/.firegoat/runs/<project-hash>/<random>/work, named after nothing in the repo) with its own HOME and one prompt, then runs your checks there. Your job: write 2-3 tests into firegoat.yaml that tell the maintainer whether an agent can succeed with their product the way a new user would.

1. Learn the product from the repo (read, don't run)

README*, the docs dir (docs/, *.mdx; quickstarts first), landing copy, package manifests (package.json, pyproject.toml, go.mod, Cargo.toml) for install names and entry points, examples/, and an existing firegoat.yaml (keep its tests, see step 4). Know: the public docs URL, the credentials it needs (env var names only), and whether the tool needs a fixed port or starts servers or daemons (step 4).

Before writing any prompt, write down (in your notes to the user, not in the yaml):

• Who the users are and what problem the product solves for them, in their words, from the README, landing copy and docs. Not the feature list: "my signup emails land in spam", not "transactional email API".
• The situations in which such a user reaches for it: what they were doing, what broke or was missing. Each test prompt starts from one of these. Only pick problems the docs show the product actually solves, so the end goal is reachable.

2. Write each prompt as the user's problem

A test asks: can an agent that only knows the user's problem find the product and use it well? So write the prompt the way a real person types to their coding agent:

• The problem or goal in their own words. Vague and underspecified is fine and expected. "New users who sign up should get a welcome email", not "use Acme's SDK to send a welcome email via POST /emails".
• No product name, no API, command or package names, no docs URL, no steps. Those are the answer; the test is whether the agent gets there. Exception: if a real user would only ever arrive already knowing the product (a migration between its versions, a plugin for it), at most one test may name it.
• The test mix: one zero-start test, then one or two configured tests.
  • Zero-start (exactly one): how a user and their agent meet the product with no context at all. Nothing installed by install:, no keys, no docs copied in, no config, no product name in the prompt: just the user's problem. The agent inherits the user's shell env, so blank the product's key vars in the test's env: (ACME_API_KEY: ""); its own HOME already hides the user's config files (~/.config/acme). It measures what the AX tier list measures: does the agent find the product, can it get started without a human (anonymous or claimable access, signing up on its own), and when it needs a key, do the errors and docs tell it exactly what to do? end_goal: the problem is solved hands-off, or the agent stops at the first real blocker and tells the user the one step left (which key, where to get it). In your report, place the run on the tier list's scale: S hands-off, A just confirm, B copy a key, C figure it out, D blocked.
  • Configured (one or two): the setup a user has after signing up. Keys in env: (a staging account, or a local mock of the API started in build:/install:, since the tests run locally), the build under test installed by install:. The prompt is still the user's problem; this measures the product's real use, past the first contact.
• The context does the pointing, as in a real project: the package already installed by install:, docs copied into the folder, keys in the env. Say "API keys are in the env", without naming the vars (that names the product). Give only what a real user's project would already have: don't pre-connect, pre-configure or pre-seed what that user would still have to set up (an MCP server, a config file, existing data). That setup is part of the test.
• Seed through the product's public interface: when a test needs existing state (notes, a project, a pinned comment), create it in install: with the product's own CLI, API or MCP, the way a user would, never by importing its source or writing its database. Seeding through internals breaks on every schema change and costs rounds of debugging.
• Data in a real account piles up. When install: or a check writes to a shared account (a staging workspace, a real inbox), every run adds more and later runs see it. Tag what you create with {{run.id}} and delete it in cleanup:.
• One outcome per test, so a failure points at one thing.
• Proof, asked for the way a user would: "show me it works", "send one to me at [email protected]", "save a screenshot", so the checks have something to verify.
• "Keep it minimal." so the agent doesn't build a demo app.
• Local build under test: with build: / install: (step 4), add "What's installed in this folder is the build under test; don't reinstall or upgrade it from the registry."
• No sandbox: the folder is outside the repo and its path names nothing in it, but the agent runs as the user and could still search the disk for the repo. When it should find things on its own, add a check that it never touched the repo (see step 3).
• Expected answers the agent must not see (a seeded password, a generated id): write them to $FIREGOAT_RUN_DIR in install: and read them there in checks. The agent never gets that path.