rasa-writing-e2e-tests

Writing E2E Tests in Rasa

E2E tests validate the assistant as a whole system — from user message to bot response, including flow execution, slot filling, custom actions, and generative responses. They live as YAML files in the tests/ directory and run via rasa test e2e.

Workflow

1. Identify the conversation scenario to test (happy path, edge case, error path).
2. Review existing tests — extend before creating new ones.
3. Create or open a YAML file under tests/ (see "File organization").
4. Write the test case with user steps and expected outcomes (see "Writing test cases").
5. Add assertions for detailed validation if needed (see "Assertions").
6. Set up fixtures if the test requires pre-filled slots (see "Fixtures").
7. Stub custom actions if needed to avoid external calls (see "Stubbing custom actions").
8. Run tests with rasa test e2e (see "Running tests").

Key principles

• One test case per scenario: happy path, cancellation, validation failure, etc.
• Start with a user step: the test runner begins evaluation after action_session_start.
• Alternate user and expected outcomes: avoid consecutive user steps.
• Use assertions for CALM-specific checks: flow_started, flow_completed, action_executed give stronger guarantees than matching bot text.
• Use bot or utter steps for simple cases: when you only need to verify the response text or domain response name.
• Use fixtures to avoid repetitive setup: shared context like authentication goes in conftest.yml.

File organization

Test files live under tests/ and are automatically discovered by Rasa. Subdirectories are supported — pass the path as a positional argument to rasa test e2e.

tests/
├── e2e_test_cases.yml           # default test file
├── conftest.yml                 # shared fixtures (visible to all tests)
├── booking/
│   ├── conftest.yml             # fixtures scoped to this directory
│   ├── test_book_trip.yml
│   └── test_cancel_booking.yml
└── payments/
    └── test_process_payment.yml