schema-define-fields

Define Fields in the Read Schema

Add or refine specific field definitions in a collection's readSchema. The readSchema controls what materializations and derivations see when reading from a collection.

Docs: https://docs.estuary.dev/guides/customize-materialization-fields/#pruned-fields

When to Use

• Pruned fields — Fields missing from materialization field selection because schema inference hit x-complexity-limit (1,000 fields default, 10,000 per binding for captures that emit a SourcedSchema — most SQL captures, plus Salesforce-native, HubSpot-native, Kinesis, and other CDK connectors that opt in)
• Type refinement — Inference saw bad data and widened a field's type (e.g., [string, null, array]); you want to narrow it to [string, null] for downstream consumers
• Define specific fields — Ensure certain fields are always present and typed when materializations read the collection
• Protect fields from inference changes — Proactively define fields so future inference updates can't prune or retype them. Also prevents dataflow reset failures: a reset clears the inferred schema, and if a downstream materialization has required fields that only existed in inference, the publish will fail with required projection X does not exist in collection Y
• Derivation shuffle key — Change a field's effective read-side type so it works as a shuffle key

Prerequisites

• Know the exact JSON path of the field(s)
• Know the field's data type
• flowctl authenticated

Key Concepts

• The readSchema is what materializations and derivations see. Captures write to the writeSchema (use schema-field-redaction for write-side changes).
• Don't edit flow://connector-schema or flow://inferred-schema directly — auto-discover and inference will overwrite your changes. Add a custom entry alongside them using allOf.
• allOf is an intersection (AND). You can narrow types and add required, but you can't widen what inference already narrowed.