databricks

databricks Best Practices

This guide outlines the essential best practices for developing on Databricks. Adhere to these rules to ensure your code is maintainable, performant, and secure.

1. Code Organization and Structure

Leverage Git folders and Databricks Asset Bundles for all projects. Treat notebooks as version-controlled code, not isolated scripts. Extract reusable logic into Python modules.

• Version Control: Always use Git folders for notebooks and source code. ❌ BAD: Storing notebooks directly in Workspace without Git integration. ✅ GOOD:

  # In a Git-synced notebook (e.g., /Repos/user/my-repo/notebooks/my_pipeline.py)
  # This notebook is version-controlled and can import local modules.
  from ..src.utils import process_data
  
  df = spark.read.table("raw_data")
  processed_df = process_data(df)
  processed_df.write.mode("overwrite").saveAsTable("processed_data")
  

• Module Extraction: For any logic beyond simple notebook orchestration, extract it into Python modules (.py files) within your Git repository. ❌ BAD:

  # In a notebook cell
  def complex_transformation(df):
      # 50+ lines of transformation logic
      return df
  

  ✅ GOOD:

  # /Repos/user/my-repo/src/transformations.py
  def complex_transformation(df):
      # Modular, testable logic
      return df
  
  # In a notebook
  from src.transformations import complex_transformation
  df = complex_transformation(spark.read.table("staging"))
  

• Project Structure with Bundles: Use Databricks Asset Bundles to define and deploy your entire project (jobs, pipelines, models, notebooks, infrastructure) as a single, versioned unit. ✅ GOOD:

  # databricks-bundle.yml
  bundle:
    name: my-data-pipeline
  resources:
    jobs:
      my_etl_job:
        name: My ETL Job
        tasks:
          - task_key: process_data
            notebook_task:
              notebook_path: ./notebooks/main_pipeline.py
            new_cluster:
              spark_version: "14.3.x-scala2.12"
              node_type_id: "Standard_DS3_v2"
              num_workers: 3