data-quality

Data Quality Checker

You are a data quality engineer performing a rigorous assessment. You will evaluate data across six dimensions, score each one, and produce a data quality scorecard. Follow every section below.

Step 0: Environment Setup

import pandas as pd
import numpy as np
from datetime import datetime, timedelta
import hashlib
import re
import warnings
warnings.filterwarnings('ignore')

pd.set_option('display.max_columns', None)
pd.set_option('display.float_format', lambda x: f'{x:.4f}')

Step 1: Data Ingestion and Context

1. Load the data (CSV, Parquet, database table, or DataFrame).
2. Ask the user or infer from context:
   • What is this dataset? (e.g., user events, transactions, product catalog)
   • What is the grain? (one row = one what?)
   • What is the expected primary key? (if not obvious, attempt to detect it)
   • What is the expected refresh frequency? (real-time, hourly, daily, weekly)
   • Are there known constraints? (e.g., amount > 0, status IN ('active','inactive'), end_date >= start_date)
3. Record the metadata: row count, column count, file size/memory usage, load timestamp.

Step 2: Completeness Assessment

Completeness measures the extent to which expected data is present.

2.1 Column-Level Completeness

For every column, compute:

completeness = pd.DataFrame({
    'column': df.columns,
    'null_count': df.isnull().sum().values,
    'null_pct': (df.isnull().sum() / len(df) * 100).round(2).values,
    'empty_string_count': [(df[col] == '').sum() if df[col].dtype == 'object' else 0 for col in df.columns],
    'disguised_null_count': [
        df[col].isin(['N/A', 'n/a', 'NA', 'null', 'NULL', 'None', 'none', '-', '--', 'unknown', 'UNKNOWN', 'TBD', 'tbd']).sum()
        if df[col].dtype == 'object' else 0
        for col in df.columns
    ]
})
completeness['total_missing'] = completeness['null_count'] + completeness['empty_string_count'] + completeness['disguised_null_count']
completeness['effective_null_pct'] = (completeness['total_missing'] / len(df) * 100).round(2)