benchmark-execute

Benchmark Command

Execute schema benchmark comparison between LiamDB and OpenAI models.

Arguments

• model: Target model to benchmark (LiamDB or OpenAI, case-insensitive)

Usage

/benchmark-execute LiamDB
/benchmark-execute openai

Execution

Important: Benchmark operations are time-intensive. The system now supports:

• Parallel dataset processing for faster execution
• Automatic input format standardization
• Improved error handling and progress reporting

Critical: When executing LiamDB benchmarks, use a 30-minute (1800-second) timeout to prevent premature termination. The deep modeling workflow can take 10+ minutes per test case.

First, I'll clean up any existing workspace and set up a fresh benchmark environment with multiple datasets:

rm -rf benchmark-workspace && pnpm --filter @liam-hq/schema-bench setupWorkspace

This will set up four benchmark datasets:

• default: Standard schema generation benchmark (3 complex cases)
• entity-extraction: Tests if specified table/column names appear in output (5 cases)
• ambiguous-recall: Measures recall of core tables from an ambiguous prompt. Uses the same input across 3 cases with different expected schemas (3/5/10 tables) to evaluate robustness.
• logical-deletion: Evaluates account deletion design without naive is_deleted. Focuses on PII separation, referential integrity for orders, legal retention/holds, closure reasons, and auditability.

The system features:

• Parallel Processing: Datasets are processed simultaneously for faster execution
• Smart Concurrency: Each dataset uses MAX_CONCURRENT=5 for stability
• Input Standardization: Entity-extraction inputs are automatically wrapped in {"input": "..."} format

Next, I'll execute the specified model with dataset selection:

{{#if (eq (lower model) "liamdb")}}

# Run LiamDB on all datasets in the workspace
pnpm --filter @liam-hq/schema-bench executeLiamDB -all

# Run LiamDB on a specific dataset
pnpm --filter @liam-hq/schema-bench executeLiamDB -entity-extraction

# Run LiamDB on the ambiguous-recall dataset only
pnpm --filter @liam-hq/schema-bench executeLiamDB -ambiguous-recall

# Run LiamDB on the logical-deletion dataset only
pnpm --filter @liam-hq/schema-bench executeLiamDB -logical-deletion

# Run LiamDB on multiple datasets
pnpm --filter @liam-hq/schema-bench executeLiamDB -default -entity-extraction -ambiguous-recall -logical-deletion

{{else if (eq (lower model) "openai")}}

# OpenAI currently targets the default dataset
pnpm --filter @liam-hq/schema-bench executeOpenai

{{else}} Error: Invalid model specified. Please use 'LiamDB' or 'OpenAI'. {{/if}}