verify-samples-tool

verify-samples Tool

The verify-samples project (dotnet/eng/verify-samples/) is an automated tool that runs sample projects and verifies their output using deterministic checks and AI-powered verification.

Running verify-samples

Important: By default, samples must be pre-built before running verify-samples. Build the solution first, or pass --build to build samples during the run:

cd dotnet
dotnet build agent-framework-dotnet.slnx -f net10.0

Then run verify-samples:

# Run all samples across all categories
dotnet run --project eng/verify-samples -- --log results.log --csv results.csv

# Run a specific category
dotnet run --project eng/verify-samples -- --category 02-agents --log results.log

# Run specific samples by name
dotnet run --project eng/verify-samples -- Agent_Step02_StructuredOutput Agent_Step09_AsFunctionTool

# Control parallelism (default 8)
dotnet run --project eng/verify-samples -- --parallel 8 --log results.log

# Build samples during run (skips the need for a prior build step)
# This may cause build conflicts as multiple samples are built in parallel, so use with caution
dotnet run --project eng/verify-samples -- --build --log results.log

# Combine options
dotnet run --project eng/verify-samples -- --category 03-workflows --parallel 4 --log results.log --csv results.csv --md results.md

Required Environment Variables

The tool itself needs:

• AZURE_OPENAI_ENDPOINT — for the AI verification agent
• AZURE_OPENAI_DEPLOYMENT_NAME (optional, defaults to gpt-5-mini)

Individual samples require their own env vars (for example, FOUNDRY_PROJECT_ENDPOINT). The tool automatically checks and skips samples with missing env vars.

Output Files

• --log results.log — detailed per-sample log with stdout/stderr, AI reasoning, and a summary
• --csv results.csv — tabular summary with Sample, ProjectPath, Status, FailedChecks, and Failures columns
• --md results.md — Markdown summary with results table and collapsible failure details (suitable for GitHub PR comments)