data-analysis

Data Analysis Skill

Overview

Perform structured exploratory data analysis (EDA) on tabular datasets. Covers loading, cleaning, profiling, statistical analysis, grouping, and communicating insights clearly.

Step-by-Step Process

Step 1: Load and Inspect the Data

import pandas as pd
import numpy as np

df = pd.read_csv("data.csv")

# Basic shape
print(f"Rows: {len(df):,}, Columns: {df.shape[1]}")
print(df.dtypes)
print(df.head(10))

For Excel: pd.read_excel("data.xlsx", sheet_name=0) For JSON: pd.read_json("data.json") For large files: use pd.read_csv("data.csv", chunksize=10000)

Step 2: Profile the Dataset

# Missing values
missing = df.isnull().sum()
print(missing[missing > 0])

# Descriptive statistics (numeric)
print(df.describe())

# Cardinality (categorical)
for col in df.select_dtypes("object").columns:
    print(f"{col}: {df[col].nunique()} unique values")
    if df[col].nunique() <= 20:
        print(df[col].value_counts())

Step 3: Clean the Data

# Drop duplicate rows
df = df.drop_duplicates()

# Handle missing values
df["column"].fillna(df["column"].median(), inplace=True)  # numeric
df["category"].fillna("Unknown", inplace=True)              # categorical

# Fix data types
df["date"] = pd.to_datetime(df["date"])
df["price"] = df["price"].str.replace("$", "").astype(float)

# Strip whitespace in strings
df["name"] = df["name"].str.strip()

Step 4: Compute Key Statistics

# Central tendency and spread
df["revenue"].agg(["mean", "median", "std", "min", "max"])

# Percentiles
df["revenue"].quantile([0.25, 0.5, 0.75, 0.9, 0.99])

# Correlation matrix
corr = df.select_dtypes("number").corr()

Step 5: Group and Aggregate

# Group by one dimension
summary = df.groupby("region")["revenue"].agg(["sum", "mean", "count"])

# Group by multiple dimensions
pivot = df.groupby(["year", "product_category"])["sales"].sum().unstack()

# Top N
top_products = df.groupby("product")["revenue"].sum().nlargest(10)