implement-sqrl

Implement SQRL scripts

SQRL is an extension of Flink SQL that adds support for table functions and convenience syntax to build reactive data processing and serving applications.
The “R” in SQRL stands for Reactive and Relationships.

This document focuses only on features unique to SQRL; when SQRL accepts Flink SQL verbatim we simply refer to the upstream spec.

Reference Templates

Review project examples in /opt/datasqrl-examples/README.md before implementing new features to identify related projects and copy features from those reference implementations.

Script Structure

A SQRL script is an ordered list of statements separated by semicolons (;).
Only one statement is allowed per line, but a statement may span multiple lines.

Typical order of statements:

IMPORT ...            -- import other SQRL scripts
CREATE TABLE ...      -- define internal & external sources
MyTable := SELECT ... -- define tables or functions (with hints)
EXPORT MyTable TO ... -- write table data to sinks

At compile time the statements form a directed-acyclic graph (DAG).
Each node is then assigned to an enabled execution engine according to the optimizer and the compiler generates the data processing code for that engine.

Flink SQL

SQRL inherits full Flink SQL grammar for

• CREATE {TABLE | VIEW | FUNCTION | CATALOG | DATABASE}
• SELECT queries inside any of the above
• USE ...
• INSERT INTO

SQRL currently tracks Flink 2.3; use its compatible SQL syntax. Refer to the Flink SQL documentation for the detailed specification.

Type System

In SQRL, every table and function has a type based on how the table represents data. The type determines the semantic validity of queries against tables and how data is processed by different engines.

SQRL assigns one of the following types to tables based on the definition:

• STREAM: Represents a stream of immutable records with an assigned timestamp (often referred to as the "event time"). Streams are append-only. Stream tables represent events or actions over time.
• VERSIONED_STATE: Contains records with a natural primary key and a timestamp, tracking changes over time to each record, thereby creating a change-stream.
• STATE: Similar to VERSIONED_STATE but without tracking the history of changes. Each record is uniquely identified by its natural primary key.
• LOOKUP: Supports lookup operations using a primary key against external data systems but does not allow further processing of the data.
• STATIC: For data that does not change over time, such as constants.