211-python-clean-code

python-clean-code.mdc

• Purpose: Ensure maintainable, consistent, and robust Python code, with special emphasis on data analytics workflows. Research shows: Clean code practices reduce maintenance time by 40-60% and bug density by 15-50% (Martin, 2008).

• General Python Standards:

  • Conform to PEP 8; enforce with Black/flake8 and pre-commit hooks.
  • Use type hints for all public functions/methods; validate with mypy.
  • Document each function with a one-sentence summary and Google-style Args/Returns/Raises.
  • Use the logging module (structured logs); avoid print for diagnostics.
  • Validate inputs; raise specific exceptions. Wrap external calls in try/except, log context.
  • No plaintext secrets or credentials—use environment variables or a secrets manager.
  • Prefer explicit imports; avoid wildcard (*) imports.
  • Respect a max line length of 88 characters (Black standard).
  • Cognitive Load Optimization: Keep functions under 20 lines (CLT index <0.6)
  • Chain of Code Pattern: Include executable examples in docstrings for complex logic

• Data Analytics Specific Patterns:

  1. Data Transformation Functions:

     • Include explicit input validation for required columns
     • Use proper data types (e.g., datetime for date fields, not strings)
     • Handle null values explicitly and document handling strategy
     • Implement defensive copying to prevent unintended mutations
     • Add logging at appropriate points in the transformation process
     • Example pattern:

       def transform_revenue_data(df: pd.DataFrame) -> pd.DataFrame:
           """Transform revenue data for analysis.
           
           Args:
               df: DataFrame containing raw revenue data
                   Required columns: ['date', 'customer_id', 'amount']
                   
           Returns:
               DataFrame with transformed data and additional metrics
               
           Raises:
               ValueError: If required columns are missing
           """
           # Validate input
           required_cols = ['date', 'customer_id', 'amount']
           if not all(col in df.columns for col in required_cols):
               raise ValueError(f"Missing required columns. Required: {required_cols}")
               
           # Convert date to datetime if needed
           if not pd.api.types.is_datetime64_any_dtype(df['date']):
               df['date'] = pd.to_datetime(df['date'])
               
           # Add calculated columns
           result = df.copy()  # Defensive copy
           result['month'] = result['date'].dt.to_period('M')
           
           # Log transformation details
           logger.info(f"Transformed revenue data shape: {result.shape}")
           
           return result
       

  2. Performance Best Practices:

     • Use vectorized operations instead of loops for pandas operations
     • Avoid multiple GroupBy operations on the same dimensions
     • Implement chunking for large dataset operations
     • Consider memory usage for large transformations
     • Use appropriate indexes for database operations
     • Tree of Thoughts Optimization: For complex transformations, prototype 3 approaches:

       # Approach 1: Direct vectorization
       df['metric'] = df['value'] * df['weight']
       
       # Approach 2: Apply with caching
       @lru_cache(maxsize=1000)
       def compute_metric(value, weight): 
           return value * weight
       df['metric'] = df.apply(lambda x: compute_metric(x['value'], x['weight']), axis=1)
       
       # Approach 3: Numpy operations
       df['metric'] = np.multiply(df['value'].values, df['weight'].values)
       
       # Benchmark and select best (typically 15-30% performance gain)