Speech-To-Text-With-Databricks: Skill for Claude Code

Databricks Development Guide

This skill provides guidance for Databricks SDK, Databricks Connect, CLI, and REST API.

SDK Documentation: https://databricks-sdk-py.readthedocs.io/en/latest/ GitHub Repository: https://github.com/databricks/databricks-sdk-py

---

Environment Setup

• Use existing virtual environment at .venv or use uv to create one
• For Spark operations: uv pip install databricks-connect
• For SDK operations: uv pip install databricks-sdk
• Databricks CLI version should be 0.278.0 or higher

Configuration

• Default profile name: DEFAULT
• Config file: ~/.databrickscfg
• Environment variables: DATABRICKS_HOST, DATABRICKS_TOKEN

---

Databricks Connect (Spark Operations)

Use databricks-connect for running Spark code locally against a Databricks cluster.

from databricks.connect import DatabricksSession

# Auto-detects 'DEFAULT' profile from ~/.databrickscfg
spark = DatabricksSession.builder.getOrCreate()

# With explicit profile
spark = DatabricksSession.builder.profile("MY_PROFILE").getOrCreate()

# Use spark as normal
df = spark.sql("SELECT * FROM catalog.schema.table")
df.show()

IMPORTANT: Do NOT set .master("local[*]") - this will cause issues with Databricks Connect.

---

Direct REST API Access

For operations not yet in SDK or overly complex via SDK, use direct REST API:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

# Direct API call using authenticated client
response = w.api_client.do(
    method="GET",
    path="/api/2.0/clusters/list"
)

# POST with body
response = w.api_client.do(
    method="POST",
    path="/api/2.0/jobs/run-now",
    body={"job_id": 123}
)

When to use: Prefer SDK methods when available. Use api_client.do for:

• New API endpoints not yet in SDK
• Complex operations where SDK abstraction is problematic
• Debugging/testing raw API responses

---

Databricks CLI

# Check version (should be >= 0.278.0)
databricks --version

# Use specific profile
databricks --profile MY_PROFILE clusters list

# Common commands
databricks clusters list
databricks jobs list
databricks workspace ls /Users/me