skill-008

Requirements for Outputs

General CSV File Handling

Cleanliness Standards

• All CSV files must be delivered free of duplicate entries and with consistent formatting.
• Ensure all string values are trimmed of whitespace and standardize case (e.g., all lowercase).

Standardization Rules

• Dates should be formatted to YYYY-MM-DD.
• Numerical values should not contain commas or currency symbols.
• Replace any missing values with "N/A" or appropriate placeholders.

Data Cleaning Techniques

Deduplication

• Implement algorithms to detect and remove duplicate rows based on key columns.
• Example code snippet:

import pandas as pd

def remove_duplicates(file_path):
    df = pd.read_csv(file_path)
    df_cleaned = df.drop_duplicates()
    return df_cleaned

Formatting Strings

• Normalize string values by removing leading or trailing whitespace and converting to lowercase before analysis.
• Example code snippet:

def format_strings(df):
    df['column_name'] = df['column_name'].str.strip().str.lower()
    return df

Handling Missing Data

• Replace missing values with specified placeholders or use interpolation if appropriate.
• Example code snippet:

def handle_missing_data(df):
    df.fillna('N/A', inplace=True)
    return df

Documentation Requirements

Data Source Citation

• Ensure all cleaned data is accompanied by a citation of the original data source: "Source: [System/Document], [Date], [Specific Reference]."

Change Log

• Maintain a change log documenting any alterations made during the cleaning process, including date and reason for changes.