great-expectations

Great Expectations Data Quality AI Skill Guide

Overview & Engine Architecture

Great Expectations (GX) validates batches of data against Expectation Suites. Datasources/connectors describe how to read data; Checkpoints run validation and emit Data Docs / action notifications. Agents encode measurable contracts (null rates, ranges, set membership, row counts), fail pipelines on critical expectations, and keep suites versioned with code.

Datasource -> Batch -> Expectation Suite
                         -> Checkpoint / Validation Result
                         -> Data Docs / actions (Slack, store)

When to use this skill

• Blocking bad data before warehouse loads or model training
• Documenting statistical contracts for analytics tables
• CI checks on sample fixtures plus scheduled prod validations

Operational directives

1. Start with critical expectations only (nulls, uniqueness, ranges) - grow suites deliberately.
2. Bind suites to stable batch definitions (table + time partition), not one-off CSVs.
3. Fail the job on success=False for producer pipelines; warn-only for exploratory.
4. Keep expectation thresholds grounded in measured baselines, not vibes.
5. Commit suites/checkpoints; treat Data Docs as build artifacts.

Fluent GX sketch (modern API)

import great_expectations as gx

context = gx.get_context()
datasource = context.data_sources.add_pandas(name="orders_df")
# Or connect SQL / filesystem per project docs for your GX version

batch = datasource.read_dataframe(df, asset_name="orders")
suite = context.suites.add(gx.ExpectationSuite(name="orders.critical"))

suite.add_expectation(gx.expectations.ExpectColumnValuesToNotBeNull(column="order_id"))
suite.add_expectation(gx.expectations.ExpectColumnValuesToBeUnique(column="order_id"))
suite.add_expectation(
    gx.expectations.ExpectColumnValuesToBeBetween(column="amount", min_value=0, max_value=1_000_000)
)

result = batch.validate(suite)
assert result.success, result