Cost Engineering on Databricks: Where the Money Goes and the Levers That Move It

Databricks bills are rarely high because of one big mistake. They are high because of many small defaults: an interactive cluster left running over the weekend, a nightly job on an oversized all-purpose cluster, a table with ten thousand tiny files scanned every hour. The good news is that the same few levers fix most of it.

The order matters. Measure first, then change things, then make the savings stick.

1. Find out where the money goes

Unity Catalog's billing system tables record usage by workspace, SKU, cluster, job, warehouse and tags. Start with a simple question: which jobs and warehouses cost the most last month?

SELECT
  usage_metadata.job_id,
  sku_name,
  SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_date >= date_sub(current_date(), 30)
GROUP BY ALL
ORDER BY dbus DESC
LIMIT 20;

Join with system.billing.list_prices to turn DBUs into currency, and with the jobs system tables to get names and owners. Build a small dashboard over it and share it with the teams that own the workloads. Visibility alone changes behaviour.

Tag everything with team, project and environment through cluster policies, so spend can be attributed without detective work.

2. Use the right kind of compute

• All-purpose compute is for people. Scheduled jobs should run on job compute or serverless jobs, which are priced lower and shut down when the job ends.
• Serverless removes idle time and start-up waits. It suits spiky, short and interactive workloads well; compare it against job compute for long, steady jobs.
• SQL warehouses for BI: use serverless where available, set auto-stop to a few minutes, and size by measured concurrency rather than by guesswork.
• Auto-termination on every interactive cluster. Thirty minutes is generous.

3. Right-size before you scale out

Look at cluster metrics for your most expensive jobs. Common findings:

• Memory-optimised nodes on jobs that are CPU-bound, or the reverse.
• Autoscaling ranges far wider than the job ever uses.
• Many small executors where fewer, larger ones would shuffle less.

Change one job at a time and compare cost and duration over a few runs.

4. Turn on Photon where it pays back

Photon speeds up SQL and DataFrame workloads, especially scans, joins and aggregations over Delta, and costs more per DBU. It pays back when the speed-up outweighs the premium, which is common for heavy SQL and less common for Python-heavy or I/O-bound jobs. Test it on your top jobs and keep it where the total cost per run falls.

5. Fix the data layout

Badly laid out tables make every query pay:

• Liquid clustering on the columns you filter by most, instead of hand-managed partitioning and Z-ordering.
• Predictive optimisation in Unity Catalog, so OPTIMIZE and VACUUM run when they help, without a scheduled job for each table.
• No tiny files: fix the writer (batch sizes, auto-compaction) rather than compacting forever.
• Materialise what dashboards ask for repeatedly, in gold tables or materialised views, rather than recomputing it from silver on every refresh.

6. Make the savings stick

Savings decay unless something enforces them:

• Cluster policies limit node types, maximum workers and auto-termination, and enforce tags.
• Budgets and alerts per workspace or tag, sent to the owning team.
• Chargeback or showback from the billing tables, so each team sees its own spend each month.
• A monthly review of the top 20 workloads. It takes an hour and catches regressions early.

A sensible first week

1. Build the billing dashboard and tag policy.
2. Move scheduled jobs off all-purpose compute.
3. Set auto-termination and auto-stop everywhere.
4. Pick the five most expensive jobs and right-size them.
5. Enable predictive optimisation and liquid clustering on the most-queried tables.

Measure again at the end of the month, and keep going down the list. If you would like an outside view, a fixed-scope health check produces a written report and a prioritised backlog in about a week.