Databricks Asset Bundles: CI/CD Done Right

The fastest way to lose trust in a data platform is a production job nobody can reproduce. Someone changed a cluster setting in the UI six months ago, the person left, and now the job fails in a way nobody can explain. Databricks Asset Bundles (DABs) exist to make that impossible: everything a job needs is declared in files, versioned in git and deployed by a pipeline.

This is the pattern we use on every Databricks project.

One bundle, three targets

A bundle is a folder with a databricks.yml at its root. It declares the resources (jobs, DLT pipelines, model serving endpoints, permissions) and the targets they deploy to.

bundle:
  name: sales-platform
 
variables:
  catalog:
    description: Unity Catalog catalog for this environment
 
include:
  - resources/*.yml
 
targets:
  dev:
    mode: development
    default: true
    variables:
      catalog: dev_sales
  uat:
    variables:
      catalog: uat_sales
    run_as:
      service_principal_name: sp-sales-uat
  prod:
    mode: production
    variables:
      catalog: prod_sales
    run_as:
      service_principal_name: sp-sales-prod

Three details matter here:

• mode: development prefixes resources with your username and pauses schedules, so each engineer gets an isolated copy in dev without stepping on anyone else.
• The catalog is a variable. The same pipeline code writes to dev_sales or prod_sales depending on the target. No environment names are hard-coded in notebooks.
• run_as a service principal in uat and prod. Jobs run as a machine identity, not as whoever deployed them last.

Keep resources small and separate

Put each job or pipeline in its own file under resources/. Pull requests stay readable, and a reviewer can see exactly which job changed.

# resources/daily_sales.yml
resources:
  jobs:
    daily_sales:
      name: daily-sales
      schedule:
        quartz_cron_expression: "0 0 2 * * ?"
        timezone_id: Asia/Kolkata
      tasks:
        - task_key: transform
          pipeline_task:
            pipeline_id: ${resources.pipelines.sales_dlt.id}

The CI/CD flow

On every pull request, CI runs unit tests and databricks bundle validate. On merge to main, it deploys to uat automatically. Production waits for a manual approval in a protected GitHub environment.

# .github/workflows/deploy.yml (abridged)
jobs:
  validate:
    if: github.event_name == 'pull_request'
    steps:
      - uses: actions/checkout@v4
      - uses: databricks/setup-cli@main
      - run: pip install -r requirements-dev.txt && pytest
      - run: databricks bundle validate -t uat
 
  deploy-uat:
    if: github.ref == 'refs/heads/main'
    environment: uat
    steps:
      - uses: actions/checkout@v4
      - uses: databricks/setup-cli@main
      - run: databricks bundle deploy -t uat
 
  deploy-prod:
    needs: deploy-uat
    environment: prod   # protected: requires approval
    steps:
      - uses: actions/checkout@v4
      - uses: databricks/setup-cli@main
      - run: databricks bundle deploy -t prod

Authenticate the pipeline with OAuth for a service principal, using workload identity federation where your cloud supports it, so there is no long-lived token sitting in repository secrets.

Guardrails that keep it honest

The pipeline is only half the story. The other half is making the manual path unattractive:

• Engineers get read access in production, not edit. If they need to change something, they open a pull request.
• Production job permissions are declared in the bundle, so they are reviewed like code.
• Cluster policies limit node types and enforce auto-termination and tags, so a deployment cannot quietly double the bill.
• Drift is visible: rerunning a deployment restores the declared state, so a hand-made change does not survive the next release.

Testing that is worth the time

You do not need a huge test suite to get value:

• Unit tests for transformation functions with small in-memory DataFrames.
• Expectations in DLT for data quality at runtime.
• A smoke run in uat after deployment, checking that each job starts and writes rows.

That combination catches most real failures before they reach production.

Where to start

If your workspace already has dozens of hand-made jobs, generate bundle definitions from the existing ones with databricks bundle generate, commit them, and switch one job at a time to pipeline deployment. Within a few weeks, the UI becomes a place to look, not a place to change things.