malloy-materialization

Materialization (Malloy Persistence)

Materialize an expensive source once so queries read a pre-built warehouse table instead of recomputing it every time. You tag a source #@ persist, a materialization run builds it into a physical table, and queries against it are rewritten to read that table.

The #1 gotcha, up front: if a persist source isn't materializing, it is almost always one of two things - a .malloy file in the package missing the ##! experimental.persistence flag (which aborts the whole package's build plan), or no build ever ran (a standalone Publisher does not build on publish - see Building and refreshing). Jump to Debugging a no-op build.

The recipe (get this right and it just works)

1. ##! experimental.persistence on EVERY .malloy file in the package - not only the file that declares the persist source. Either form enables it:

   • ##! experimental.persistence, or
   • ##! experimental { access_modifiers, sql_functions, persistence } (add persistence to the existing list).

   Why every file: the build plan is computed by asking every .malloy file in the package for its persist sources, and that call throws on any file whose model lacks the flag (Model must have ##! experimental.persistence). One unflagged helper or import file, even one with no persist source of its own, aborts the whole package's build plan, so every persist source in the package drops out. This is the most common cause of a no-op build.

2. #@ persist name="..." on a query-based source, with the name quoted:

   #@ persist name="my_dataset.my_table"
   source: my_rollup is some_source -> { group_by: ...; aggregate: ... }
   

   • Only query_source and sql_select sources are persistable - a source whose definition has a -> { ... } pipeline or a conn.sql("..."). This includes one refined by a trailing extend { ... }. What is not persistable is a plain extend over a bare conn.table(...); a #@ persist on such a source is silently ignored (its annotation is never read) - that one source just won't materialize, and the rest of the package still builds.
   • Quote the name. name="my_table" (or a path name="dataset.table" / name="project.dataset.table") is required. A bare name=my_table always fails the build/publish with persist annotation name must be quoted (a raw-source scan that hard-stops); it never silently no-ops.
   • name= is the target table name. In a standalone Publisher this is the physical table (rebuilt in place); a hosted (control-plane) deployment builds it under a content-addressed generation name. In both, the source's identity for reuse is a content address of its connection and canonical SQL (its sourceEntityId), so republishing unchanged persist logic reuses the existing table and changing the logic builds fresh.