blackbox

Intervention: blackbox (delivered through the Skillberry proxy)

An intervention answers how a candidate reaches the model under test — as opposed to a capability, which is what gets edited. The two are independent: the same skill-package capability can be delivered by writing files where a runner reads them (intervention: direct) or by this intervention.

In blackbox mode:

benchmark runner
  └── the agent's LLM call ──► SPA (host:7000) ──► Skillberry Store (host:8000)
                                    │                  (resolves ONE skill:
                                    │                   prose -> prompt enrichment,
                                    │                   scripts/ -> callable tools)
                                    └──────────────► upstream LLM
  other LLM calls (user simulator, judge, verifier) ──► upstream LLM, NEVER SPA

The agent gets no skill files, no mounted directory, no visible prompt edit — which is the shape Skillberry actually ships, so optimizing here optimizes the real thing.

Requirements from the benchmark

Two things must already be true of the runner, and this intervention checks rather than supplies them. Onboard a runner that lacks either and the data is wrong, not missing.

• The runner must be SPA-aware. Three integrations have to be present for blackbox mode to produce correct data: Skillberry context headers on the agent's LLM calls, a merge of the proxy-side trajectory into the runner's own, and a disconnect at session end. A Skillberry-aware build carries them; a stock runner does not.
• The benchmark must front its environment over HTTP. Store-hosted tools execute in the store's process, not where the benchmark's state lives, so the skill's tools can only reach that state through a service the benchmark provides. This intervention only health-checks it, via SPA_REMOTE_ENV_URL.

Inputs / outputs (manifest tokens)

needs: [], provides: [] — deliberately. An intervention owns an out-of-process delivery stack, not a step in the run DAG, so it neither consumes nor produces a pipeline token. It is selected by the spec (intervention: blackbox), not sequenced by orchestrate.