data-quality

Data Quality

Purpose

Catch bad data before it reaches a dashboard, a model, or a customer. A pipeline that silently propagates corrupt data is worse than one that fails, because the failure is discovered downstream, later, by someone who trusts the number.

When to Use

• Ingesting data from a source you do not control.
• Building quality gates into a pipeline.
• Investigating a metric that looks wrong.
• Auditing a dataset before it is used for analysis or training.

Capabilities

• Profiling: distributions, cardinality, null rates, outliers.
• Schema validation and type enforcement.
• Constraint checks: uniqueness, referential integrity, ranges, formats.
• Freshness, completeness, and volume anomaly detection.
• Quarantine and alerting patterns.

Inputs

• The dataset and its expected schema.
• The business rules the data must satisfy.
• Historical volume and distribution, for anomaly baselines.

Outputs

• A profile of the data as it actually is, not as documented.
• Validation checks that run on every load.
• A quarantine path for rows that fail, and an alert when they do.

Workflow

1. Profile before you trust — Row count, null rate, cardinality, min/max, and the distribution of every column. The documented schema and the actual data disagree more often than not.
2. Validate the schema at the boundary — Column presence, types, and nullability, checked on ingest. A silently added column or a type change upstream is the most common pipeline break.
3. Assert the business rules — Uniqueness on keys, referential integrity, valid ranges, and formats. total_cents >= 0 is a rule; assert it.
4. Check freshness and volume — Is the data recent, and is there roughly as much of it as usual? A pipeline that runs successfully on an empty file is the failure that is hardest to notice.
5. Quarantine, do not drop — Failing rows go to a quarantine table with the reason. Dropping them silently destroys the evidence needed to fix the source.
6. Fail loudly and stop — A pipeline that continues past a failed quality gate has published bad data. Stop, alert, and keep the previous good version live.