data-analysis

Data Analysis Methodology

Prioritize correctness over speed — a wrong insight is worse than no insight.

Step 1: Data Ingestion

Uploaded files are located at /home/user/uploads/. Always list that directory first to discover available files:

import os
for f in os.listdir('/home/user/uploads/'):
    print(f)

Tool selection: Use code_run as the primary execution tool — it is universally supported across all sandbox providers. Use code_interpret only when you need rich output capture (e.g., inline dataframes, rendered plots); note that code_interpret may not be available in all environments.

Examine the raw file (first 20-30 lines) to understand format, delimiter, encoding, headers, and obvious quality issues before loading.

Load by file type — always specify dtypes for known columns and parse_dates for date columns:

Extension	Loader
.csv	pd.read_csv('/home/user/uploads/file.csv', parse_dates=[...])
.tsv	pd.read_csv('/home/user/uploads/file.tsv', sep='\t', parse_dates=[...])
.xlsx / .xls	pd.read_excel('/home/user/uploads/file.xlsx', engine='openpyxl')
.json	pd.read_json('/home/user/uploads/file.json')
.parquet	pd.read_parquet('/home/user/uploads/file.parquet')

Step 2: Mandatory EDA

Run this diagnostic block on every dataset before any analysis:

print(f"Shape: {df.shape}")
print(f"\nDtypes:\n{df.dtypes}")
print(f"\nMissing values:\n{df.isnull().sum()[df.isnull().sum() > 0]}")
print(f"\nDuplicate rows: {df.duplicated().sum()}")
print(f"\nNumeric summary:\n{df.describe()}")

for col in df.select_dtypes(include='object').columns:
    n_unique = df[col].nunique()
    print(f"\n{col}: {n_unique} unique values")
    if n_unique <= 20:
        print(df[col].value_counts())

Do not skip this step. Report findings before proceeding to analysis.

After completing EDA, send a progress update via user_message summarizing the dataset shape, quality issues found, and your planned analysis approach.