rill-connector

Instructions for developing a connector in Rill

Introduction

Connectors are resources that contain credentials and settings for connecting to external systems. They are typically found at the root of the project's DAG, providing access to data sources and services that power downstream resources like models and metrics views.

Connectors are usually lightweight resources. When reconciled, they validate the connection to the external system but do not move or process data. The main exception is managed OLAP connectors (with managed: true), which trigger database provisioning.

Driver capabilities

Each connector uses a driver that implements one or more capabilities:

• OLAP database: Can power metrics views and dashboards (e.g., duckdb, clickhouse, druid, pinot, starrocks, snowflake, bigquery, databricks as live connectors)
• SQL database: Can run SQL queries as model inputs (e.g., postgres, bigquery, snowflake)
• Information schema: Can list tables and their schemas (e.g., duckdb, bigquery)
• Object store: Can list, read, and write flat files (e.g., s3, gcs)
• Notifier: Can send notifications and alerts (e.g., slack)
• AI: Can generate embeddings or responses (e.g., openai, claude, gemini)

Core Concepts

Naming conventions

Connectors are typically named after their driver (e.g., a file connectors/duckdb.yaml creates a connector named duckdb). Use descriptive names when you have multiple connectors of the same type:

• connectors/s3_data.yaml and connectors/s3_staging.yaml
• connectors/clickhouse_prod.yaml and connectors/clickhouse_dev.yaml

Secrets management

Always store sensitive credentials in .env and reference them using template syntax:

type: connector
driver: s3
aws_access_key_id: "{{ .env.aws_access_key_id }}"
aws_secret_access_key: "{{ .env.aws_secret_access_key }}"

NOTE: Some legacy projects use the deprecated .vars instead of .env.