data-analysis

Data Analysis

Workflow

1. Load & inspect — read data, check shape, types, nulls
2. Clean — handle missing values, fix types, remove duplicates
3. Explore — summary stats, distributions, correlations
4. Analyze — answer the specific question
5. Visualize — create clear, labeled charts
6. Report — summarize findings in plain language

Quick Start: Data Profiling

python scripts/profile.py data.csv                    # print profile to stdout
python scripts/profile.py data.xlsx --output report.md  # save to file
python scripts/profile.py data.xlsx --sheet "Sales"     # specific sheet

The profiler auto-detects file format and generates: row/column counts, types, null percentages, numeric statistics, and top categorical values.

Loading Data

import pandas as pd

# Auto-detect format
df = pd.read_csv("data.csv")
df = pd.read_excel("data.xlsx")
df = pd.read_json("data.json")
df = pd.read_csv("data.tsv", sep="\t")

# Handle encoding issues
df = pd.read_csv("data.csv", encoding="latin-1")

# Large files — read in chunks
for chunk in pd.read_csv("large.csv", chunksize=10000):
    process(chunk)

Inspection

df.shape                    # (rows, cols)
df.dtypes                   # column types
df.head(10)                 # first 10 rows
df.describe()               # numeric statistics
df.describe(include='all')  # include categorical
df.isnull().sum()           # missing values per column
df.nunique()                # unique values per column
df.duplicated().sum()       # duplicate rows

Cleaning

# Drop duplicates
df = df.drop_duplicates()

# Handle missing values
df['col'].fillna(df['col'].median(), inplace=True)  # fill with median
df = df.dropna(subset=['critical_col'])               # drop rows missing critical data

# Fix types
df['date'] = pd.to_datetime(df['date'])
df['amount'] = pd.to_numeric(df['amount'], errors='coerce')
df['category'] = df['category'].astype('category')

# Clean strings
df['name'] = df['name'].str.strip().str.lower()

# Rename columns
df.columns = df.columns.str.strip().str.lower().str.replace(' ', '_')