running-livekit-simulations

Running LiveKit simulations

A simulation plays a scenario against the real agent using an LLM-driven simulated user, then a judge grades the transcript. A unit test asserts on one turn. A simulation tells you whether a whole conversation reached the right outcome.

Read lk agent simulate --help before running. Subcommands and flags change, a wrong flag wastes a paid run, and this skill doesn't restate them. reading-livekit-docs has the rest.

When to reach for a simulation

Use simulations to regression-test long-horizon behavior before deploying to production: whether a multi-turn conversation reaches the right outcome when the caller backtracks, whether details gathered early survive to the end, whether the agent holds to its instructions under pressure, and whether it ended in the right state. For a single turn, use testing-livekit-agents; to poke at behavior while editing, use debugging-livekit-agents.

Running

Run from the agent's project directory. The mode is a subcommand:

lk agent simulate text --scenarios scenarios.yaml    # see --help for the current flags

With no scenario file, the CLI generates scenarios from the agent's source. That uploads the code, and the CLI asks for confirmation first. Generation belongs to writing-livekit-scenarios.

By default the CLI starts the agent as a local worker, dispatches the scenarios to it, and stops it when the run ends. An option lets you grade an already-running agent by name instead. That needs a scenario file, since there's no local source to generate from.

Concurrency is limited per run and per project. The docs have the current limits.

Text or audio

Text is the default, and it's the right one. The simulated user exchanges text with the agent, so the run exercises the LLM, the tools and the conversation logic while the framework turns off STT, TTS and VAD. It's faster, cheaper and more deterministic. Use it for iteration and for anything automated.

Audio runs the same scenarios through the full speech pipeline. The simulated user speaks, listens and interrupts like a caller would, and the run scores what only speech exposes: