foundry-evals

Foundry Agent Evaluations

Evaluate Foundry hosted agents using the two-phase invoke+score pattern with Foundry's built-in evaluators.

For the complete per-agent adoption and release-evidence workflow, see foundry-agentops. This skill remains authoritative for deep evaluator configuration and dataset design; AgentOps aggregates evidence and never replaces that evaluation contract.

When to Use

• After deploying a hosted agent, to measure quality
• Running batch evaluations against test scenarios
• Comparing agent performance across versions
• Validating that business rules (BR-XXX from SpecKit) are followed

Why Two Phases?

The Foundry SDK's azure_ai_agent target type does NOT correctly route to hosted agent endpoints — it sends requests to the project endpoint instead of the agent's dedicated endpoint. You must invoke the agent yourself, then score the results separately.

Phase 1: Invoke agent → collect responses
Phase 2: Score responses → Foundry evaluators

Critical requirement: MUST complete both phases sequentially. Skipping either phase will result in incomplete evaluations.

Phase 1: Invoke the Agent

from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential

project = AIProjectClient(
    endpoint="<project_endpoint>",
    credential=DefaultAzureCredential(),
    allow_preview=True,
)
oai = project.get_openai_client(agent_name="my-agent")

# Warm up — single-shot ping is INSUFFICIENT for hosted agents that
# scale to zero (15min idle). Use the retry loop pattern below for any
# eval likely to hit a cold container. A single ping can return
# server_error in ~9s and make every scenario fail before the agent
# has even spun up.
print("Warming up...")
oai.responses.create(input="Hello", stream=False)

# MUST invoke each query SEQUENTIALLY (never concurrent)
# Concurrent requests overwhelm cold-start containers → empty responses
results = []
for query in test_queries:
    response = oai.responses.create(input=query, stream=False)
    results.append({
        "query": query,
        "response": response.output_text,
    })
    print(f"✓ {query[:50]}...")