schema-field-redaction

Protect Sensitive Data with Field Redaction

Add redaction annotations to your collection schema to protect sensitive data at the source. Redacted fields are blocked or hashed before documents are stored or materialized.

When to Use

• PII Protection: Hash emails, SSNs, phone numbers for GDPR/HIPAA/CCPA compliance
• Credential Security: Block passwords, API keys, tokens from ever leaving the capture
• Internal Fields: Remove internal-only metadata before materialization
• Data Minimization: Ensure only necessary data reaches downstream systems

How Redaction Works

Redaction is applied at write time — sensitive data never reaches Estuary storage, logs, or downstream materializations. This is the earliest point to remove fields, before derivations or materialization field selection.

Redaction only applies to documents captured after publishing the annotation. To redact historical data, trigger a dataflow reset backfill on the capture.

Redaction Strategies

Strategy	Effect	Use When
block	Removes the field entirely	Field should never exist downstream
sha256	Replaces value with salted hash	Need to join/match on field (hash is deterministic per task)

Prerequisites

• flowctl installed and authenticated
• Write access to the collection you want to modify
• Know which fields contain sensitive data

Key Constraints

1. Cannot redact key fields — Would break collection semantics.
2. block on required fields — Validation fails with `block` redact strategy cannot be applied at '/path/to/field' because it must exist (the path is a JSON Pointer to the field) UNLESS the read schema references the inferred schema (flow://inferred-schema), in which case the validator suppresses that error automatically. For explicit schemas, remove the field from required first (see Troubleshooting).
3. sha256 on non-string-typed fields — At runtime the hasher accepts any type. But at publish time, validation fails with /path/to/field has 'sha256' redact strategy but cannot be a string (types are "<type>") if the field's declared type set has no overlap with string. A union like ["string","null"] or ["string","integer"] passes — only a field that cannot be a string at all (e.g., pure integer or pure object) is rejected.