create-workflow-benchmarks

Create workflow benchmarks and tasks

A benchmark defines repeatable work; its tasks have stable IDs. An experiment records attempts against a specific benchmark version. Creating either in the UI does not execute the workflow.

Author in run_chatbot

Start fastworkflow run_chatbot, select the application workflow, then open Benchmark setup. Use the printed authenticated local URL. Selecting a workflow normally starts its server; reuse an existing session when only authoring or reviewing. Follow the project's runtime permissions.

1. Choose New benchmark, enter a title and optional description, and add task rows. Each task's prompt is optional: a runner can supply the actual messages later.
2. Save benchmark assigns the benchmark ID, v1, and new task IDs. Keep those identities when adding execution code; do not derive task IDs from conversation topics.
3. Edit benchmark saves a new immutable version. Existing task IDs are retained; new rows receive new IDs. Removing a row removes it only from the new version.
4. Select the intended version, then New experiment. Copy the generated experiment ID and task IDs to the driver. This creates a registration, initially without an evidence store.
5. Once the driver records execution, use Open execution records to inspect it.

The left navigation is benchmark → experiment → conversation. Ad-hoc conversations appear under ad-hoc conversations → UTC date → conversation. Node clicks show details on the right and expand/collapse the left tree. Turns and trace components are inspected on the right; only the right pane shows the complete breadcrumb. Tasks are listed in benchmark/experiment details, not as another required left-tree level. Sidebar filters and the old Experiments/ Benchmarks navigation buttons are no longer part of this flow.

Author the same objects from Python

For the UI's simple title/prompt model, use the setup API:

from fastworkflow.benchmark import setup as benchmark_setup

version = benchmark_setup.save_benchmark(workflow_folderpath, {
    "title": "Order support regressions",
    "description": "Lookup, ambiguity and recovery using disposable test orders",
    "tasks": [{"prompt": "Show the delivery status of my most recent order."}],
})
registration = benchmark_setup.create_experiment(
    workflow_folderpath, version["benchmark_id"], version["version"]
)
experiment_id = registration["experiment_id"]
task_ids = registration["task_ids"]