rill-development

Instructions for developing a Rill project

This document is intended for data engineering agents specialized in developing projects in the Rill business intelligence platform.

Introduction to Rill

Rill is a business intelligence platform built around the following principles:

• Code-first: configure projects using versioned and reproducible source code in the form of YAML and SQL files.
• Full stack: go from raw data sources to user-friendly dashboards powered by clean data with a single tool.
• Declarative: describe your business logic and Rill automatically runs the infrastructure, migrations and services necessary to make it real.
• OLAP databases: you can easily provision a fast analytical database and load data into it to build dashboards that stay interactive at scale.

Project structure

A Rill project consists of resources that are defined using YAML and SQL files in the project's file directory. Rill supports different resource types, such as connectors, models, metrics views, explore dashboards, and more.

Here is an example listing of files for a small Rill project:

.env
connectors/duckdb.yaml
connectors/s3.yaml
models/events_raw.yaml
models/events.sql
metrics/events.yaml
dashboards/events.yaml
rill.yaml

Let's start with the project-wide files at the root of the directory:

• rill.yaml is a required file that contains project-wide configuration. It can be compared to package.json in Node.js or dbt_project.yml in dbt.
• .env is an optional file containing environment variables, usually secrets such as database credentials.

The other YAML and SQL files define individual resources in the project. They follow a few rules:

• The YAML files must contain a type: property that identifies the resource type. The other properties in the file are specific to the selected resource type.
• SQL files are a convenient way of creating model resources. They are equivalent to a YAML file with type: model and a sql: property.
• Each file declares one main resource, but may in some cases also emit some dependent resources with internally generated names.
• The main resource declared by a file gets a unique name derived from the filename by removing the directory name and extension. For example, connectors/duckdb.yaml defines a connector called duckdb.
• Directories are ignored by the parser and can be used to organize the project as you see fit. Small projects often have one directory per resource type.
• Resources can reference other resources, which forms a dependency graph (DAG) that informs the sequence they are executed.
• Resource names are unique within a resource type. For example, only one model can be named events (regardless of directory), but it is possible for both a model and a metrics view to be called events.
• Clear resource names are important as they are widely used as unique identifiers throughout the platform (e.g. in CLI commands, URL slugs, API calls). They are usually lowercase and snake case, but that is not enforced.
• Any resource can carry a top-level tags: list (free-form labels) for organizing and filtering resources across a project. Distinct from dimension/measure tags inside a metrics view.