kedro-deploy-databricks

Deploying a Kedro project to Databricks

Scope

This skill covers the Kedro side of a Databricks deployment. Split every answer along these lines:

kedro-databricks is a community-maintained plugin (https://github.com/JenspederM/kedro-databricks), not part of Kedro: check its version and open issues before relying on a behaviour. For anything in the Databricks row, say where the setting lives in the project and link the Databricks documentation listed under Out of scope. Never invent values such as node_type_id, spark_version or policy IDs.

CRITICAL: Never take kedro-databricks flags, bundle fields or Databricks behaviour from training data. Both change often. The installed code and --help output win over the docs, memory and this file, and versioned docs win over memory. Where kedro-databricks' docs disagree with its code, follow the code. Never generate these outdated patterns: dbx, databricks configure --token, /dbfs/FileStore/..., dbfs:/mnt/... or /dbfs/mnt/... paths, the databricks-iris starter, or a conf/<env>/databricks.yml with a top-level default: key instead of resources:.

Step 1: check what is installed

Run this with the project's interpreter:

python - <<'EOF'
import sys
from importlib.metadata import PackageNotFoundError, version
print(sys.executable, "venv" if sys.prefix != sys.base_prefix else "")
for name in ("kedro", "kedro-databricks", "kedro-datasets"):
    try:
        print(name, version(name))
    except PackageNotFoundError:
        print(name, "not installed")
try:
    from kedro.inspection import validate_deployment_grouping  # noqa: F401
    print("grouping check: available")
except ImportError:
    print("grouping check: not available")
EOF