rasa-configuring-model-groups

Configuring Model Groups

Model groups are defined in endpoints.yml under the model_groups key. Pipeline components, the rephraser, and other features reference groups by their id. Each group contains one or more model deployments and an optional router for multi-deployment routing.

Workflow

1. Open endpoints.yml (create if it doesn't exist).
2. Add a model group for the LLM used by the pipeline's command generator.
3. Add a model group for embeddings if flow retrieval is enabled.
4. If multiple deployments are needed, add them to the same group and configure a routing strategy (see "Multi-deployment routing").
5. Reference the group id from config.yml pipeline components (see rasa-configuring-assistant skill).

Providers

Rasa provides dedicated client wrappers only for certain providers. The supported sets differ for LLM and embeddings.

LLM (Rasa wrappers):

• openai,
• azure,
• self-hosted,
• rasa.

Embeddings (Rasa wrappers):

• openai,
• azure,
• huggingface_local.

For any other provider (e.g. Anthropic, Cohere, Google), use the provider keys and options from LiteLLM's provider list, since Rasa's generic clients are built on LiteLLM.

Configuring single provider

The simplest setup — one deployment per group:

model_groups:
  - id: my_llm
    models:
      - provider: openai                  # or azure, self-hosted, etc.
        model: <your-llm-model>

  - id: my_embeddings
    models:
      - provider: openai                  # or azure, huggingface_local, etc.
        model: <your-embedding-model>

To switch providers, change provider and add any required provider-specific settings:

model_groups:
  - id: my_llm
    models:
      - provider: azure
        deployment: <your-deployment-name>
        api_base: https://my-azure-instance/
        api_version: "2024-02-15-preview"
        api_key: ${AZURE_API_KEY}

Configuring multi-deployment routing