When to Move from All-Purpose Compute to Job Compute

If we had to name the single most common finding when we look at a Databricks workspace, it would be this: scheduled jobs running on all-purpose clusters. It happens innocently. Someone builds a notebook on their interactive cluster, schedules it from the same screen, and it keeps running there for a year. It works, and it costs more than it should.

Why the compute type matters

Databricks has different kinds of compute for different jobs:

• All-purpose compute is for interactive development: notebooks, exploration, debugging. It stays up between commands and is shared by people.
• Job compute is created for a job run and terminated when the run ends. It is billed at a lower rate than all-purpose compute.
• Serverless jobs compute removes cluster sizing and start-up waits altogether. Databricks manages the capacity, and you pay for what the run uses.

Running a schedule on all-purpose compute pays the interactive rate, and often pays for idle time too, because the cluster waits for its auto-termination timeout between runs.

Find the jobs that should move

Use the system tables rather than guessing. Jobs whose tasks run on an existing all-purpose cluster show up in the jobs and billing tables:

SELECT
  j.name,
  u.usage_metadata.cluster_id,
  SUM(u.usage_quantity) AS dbus_last_30_days
FROM system.billing.usage u
JOIN system.lakeflow.jobs j
  ON u.usage_metadata.job_id = j.job_id
WHERE u.usage_date >= date_sub(current_date(), 30)
  AND u.sku_name LIKE '%ALL_PURPOSE%'
GROUP BY ALL
ORDER BY dbus_last_30_days DESC;

Table and column names in system tables evolve, so check the current schema in your workspace before you rely on a query like this. The point is the approach: rank by spend, start at the top.

Move them in batches

For each job:

1. Choose the target. Serverless jobs compute is the default choice where it fits. Keep classic job compute for workloads that need custom libraries, init scripts, specific instance types or networking that serverless does not support. That list changes, so confirm the current limits before you promise anything.
2. Declare it in code. In a Databricks Asset Bundle, set the job's task to use a job_cluster or serverless compute instead of an existing_cluster_id. The change is reviewed and deployed like any other.
3. Size classic job clusters from evidence. Start from the cluster metrics of past runs, not from the size of the interactive cluster the notebook was written on.
4. Compare cost per run for a week before and after. Report the change in DBUs and duration, on your workload, rather than quoting a general percentage.

Keep it that way

Savings decay unless something enforces them:

• Cluster policies stop new all-purpose clusters being created without auto-termination and tags, and can stop jobs from targeting all-purpose compute at all.
• Permissions: only a small group can create all-purpose clusters; everyone else uses shared or personal compute for development.
• A monthly check of the query above. Anything new at the top of the list gets moved.

What not to move

Not everything belongs on job compute:

• Interactive analysis and development stay on all-purpose or serverless notebook compute.
• Very short, very frequent jobs may be better combined into one job with several tasks, so you pay one start-up instead of many.
• Streaming jobs that run continuously need a deliberate choice of compute and a look at whether continuous running is really needed, or whether trigger(availableNow=True) on a schedule would do.