derivation-python

derivation-python

Estuary derivation implemented as a Python class with async-generator transforms. Runtime is uv, types come from Pydantic models generated at flowctl generate time, and the whole thing is validated in pyright strict mode at publish.

Prereq: read derivation-basics first for concepts, project layout, stateless-vs-stateful, and the universal workflow.

Hard constraint: Python derivations run on private or BYOC data planes only — never on shared public planes. Reach out to your Estuary account manager if you're interested in a private or BYOC data plane.

Docs: concept page · walkthrough · upstream examples (stateful.flow.py, pipeline.flow.py).

When to reach for Python (vs SQL / TypeScript)

• ML / embeddings / tokenisers — model2vec, fastembed, transformers, tiktoken
• Async I/O — per-document await httpx.get(...), external APIs with bounded concurrency
• Heavy Python ecosystem deps — pandas, pyarrow, custom parsers
• Rich type safety — Pydantic validators; pyright-strict at publish

Reach for other skills when plain filter/transform fits SQL (derivation-filter-transform), the logic is reducible (derivation-aggregate-metrics), or you want stateful logic without a private DP (derivation-stateful-logic, SQLite).

How it works

You declare derive.using.python.module: <file>.flow.py. flowctl generate introspects your collection schemas and emits IDerivation, Document, and Request symbols you import. Your Derivation(IDerivation) class has one async generator method per transform — named as snake_case of the YAML transforms[].name. Each transform method consumes Request.Read<TransformName> (typed Pydantic read.doc) and yields Document(...) values. Dependencies in derive.using.python.dependencies are resolved by uv at publish time and at runtime. Optional state is persisted across restarts via __init__ / start_commit / reset lifecycle methods.