optimize-databricks-costs

Optimize Databricks Costs

Run a read-only cost audit first, convert evidence into an impact-aware proposal, and mutate Databricks only after the user approves the exact affected workloads and impacts.

Operating rules

1. Default to read-only commands and SQL.
2. Do not edit, stop, delete, pause, resize, retag, revoke, or create anything during an audit or proposal.
3. Treat generated reports as private. Never commit hosts, IDs, emails, query text, secrets, or customer object names.
4. Price usage with system.billing.list_prices.pricing.effective_list.default. Call it effective list cost, not invoice cost.
5. Include retractions and restatements when summing billing usage; they correct earlier records.
6. Exclude the current day and an ingestion-lag window from baseline comparisons. Inspect partial recent days separately.
7. Attribute cost to resources and identity_metadata, then prove the underlying workload before describing a user or service principal.
8. Prefer reversible changes. Record the before state and rollback command before every approved mutation.

Phase 1: collect evidence

Confirm the Databricks CLI profile and current identity. List accessible warehouses, Apps, Jobs, pipelines, and clusters without changing them.

Resolve the bundled collector relative to this SKILL.md, then run it from any working directory:

python3 <skill-dir>/scripts/audit.py \
  --warehouse-id <warehouse-used-to-run-system-table-queries> \
  --recent-days 14 \
  --output-dir <private-report-directory>

Use --profile <name> for a non-default Databricks CLI profile. Use --dry-run to inspect the SQL without connecting.

If the script cannot access a system table, retain the successful evidence, report the permission gap, and continue with the accessible inventory. Do not broaden permissions during the audit.

Read audit-methodology.md when interpreting the output. Read system-tables.md when adapting a query to the available Databricks schema.