data-quality

Data Quality

Level 1: Quick Start (5 min)

Core Principles:

• Completeness - no missing critical data
• Accuracy - data reflects reality
• Consistency - data aligns across systems
• Timeliness - data is current and available

Quick Reference:

# Great Expectations validation
import great_expectations as gx
context = gx.get_context()
validator = context.sources.pandas_default.read_csv("data.csv")
validator.expect_column_values_to_not_be_null("user_id")
validator.expect_column_values_to_be_between("age", 0, 120)

Essential Checklist:

• Define data quality rules and SLAs
• Implement automated validation checks
• Monitor data quality metrics
• Set up alerting for quality degradation
• Document data quality expectations

Common Pitfalls: See Common Pitfalls

Level 2: Implementation (30 min)

Data Quality Dimensions

Completeness Checks:

def check_completeness(df, required_columns):
    """Validate no missing values in critical columns"""
    missing = df[required_columns].isnull().sum()
    completeness = (1 - missing / len(df)) * 100
    return completeness

# Example
required = ['user_id', 'transaction_date', 'amount']
scores = check_completeness(df, required)
assert all(scores > 99), f"Completeness below threshold: {scores}"

Accuracy Validation:

# Range checks
def validate_ranges(df):
    assert df['age'].between(0, 120).all(), "Age out of range"
    assert (df['amount'] >= 0).all(), "Negative amount found"
    assert df['email'].str.contains('@').all(), "Invalid email"

Consistency Rules:

# Cross-field validation
def check_consistency(df):
    # End date must be after start date
    assert (df['end_date'] >= df['start_date']).all()

    # Total should equal sum of parts
    assert np.isclose(
        df['total'],
        df[['part1', 'part2', 'part3']].sum(axis=1)
    ).all()

Data Quality Framework Implementation

Great Expectations Setup: