file-processing

File Processing Skill

This skill provides utilities and guidance for building robust file processing applications.

Purpose

Use this skill when your task involves:

• Reading and parsing CSV, JSON, or text files
• Data validation and cleaning
• File format conversions
• Batch processing of multiple files
• Generating reports from data files

Key Capabilities

1. Data Reading

• CSV parsing with header detection
• JSON file handling (single object or array)
• Text file processing line-by-line
• Error handling for malformed files

2. Data Validation

• Check for required fields
• Validate data types
• Handle missing values
• Report data quality issues

3. Data Transformation

• Filter rows based on conditions
• Calculate statistics (sum, avg, count)
• Format conversions
• Data aggregation

4. Output Generation

• Write processed data to new files
• Generate summary reports
• Create multiple output formats

Best Practices

Project Structure for File Processing

project/
├── main.py              # Entry point with CLI
├── file_reader.py       # File I/O operations
├── data_processor.py    # Core processing logic
├── validator.py         # Data validation
├── config.py            # Configuration constants
└── utils.py             # Helper functions

Error Handling Pattern

def read_file_safely(filepath):
    """Read file with proper error handling"""
    try:
        if not os.path.exists(filepath):
            raise FileNotFoundError(f"File not found: {filepath}")

        with open(filepath, 'r', encoding='utf-8') as f:
            return f.read()
    except Exception as e:
        print(f"Error reading file: {e}")
        return None

CSV Processing Template

import csv

def process_csv(input_file, output_file):
    """Process CSV with header detection"""
    with open(input_file, 'r', encoding='utf-8') as f:
        reader = csv.DictReader(f)

        processed = []
        for row in reader:
            # Transform each row
            processed_row = transform_row(row)
            processed.append(processed_row)

    # Write results
    with open(output_file, 'w', encoding='utf-8') as f:
        if processed:
            writer = csv.DictWriter(f, fieldnames=processed[0].keys())
            writer.writeheader()
            writer.writerows(processed)