Operations and CI/CD

Changes made by hand in the workspace UI don't scale past a single contributor. When a second person ships changes, or staging must match production, you need repeatable deployments, code review, and a promotion path from development to production. On Azure Databricks, you get these by defining your projects as code, keeping them in Git, and deploying them with continuous integration and continuous delivery (CI/CD).

Separate infrastructure from projects

Manage Azure Databricks DevOps in two layers, each with its own tool and lifecycle:

Layer What it includes Tool Examples
Platform infrastructure Resources outside a workspace, such as accounts, networks, metastores, workspaces, and cloud Identity and Access Management (IAM) Terraform Terraform examples
Azure Databricks projects Resources inside a workspace, such as jobs, pipelines, schemas, dashboards, apps, and registered models Declarative Automation Bundles Bundle configuration examples

Keeping the layers separate means that a change to a pipeline doesn't touch your network configuration, and a platform team can manage workspaces without editing the projects that run in them. To create workspaces with Terraform, see Create development, staging, and production workspaces.

Define projects with Declarative Automation Bundles

Declarative Automation Bundles, formerly known as Databricks Asset Bundles, are an infrastructure-as-code approach to managing your Azure Databricks projects. A bundle is an end-to-end definition of a project. It includes the source files, such as notebooks and Python files, the YAML definitions of the Azure Databricks resources that the project uses, and the tests that validate it. You deploy the bundle as a single unit to a target, such as development, staging, or production. See What are Declarative Automation Bundles?.

The following video demonstrates how to work with bundles:

The Databricks CLI includes the bundle commands, so a CI/CD runner, such as a GitHub Actions runner, needs only the Databricks CLI and workspace credentials to validate and deploy a bundle. For the resources that a bundle can define, see Declarative Automation Bundles resources.

The following repositories contain example bundles:

Repository Contents
bundle-examples/knowledge_base Bundles for individual resources and patterns, such as apps, dashboards, Genie Agents, jobs, pipelines, and Lakebase databases
databricks-dab-examples/flights An end-to-end project on a flights dataset, in simple, advanced, and bundle template versions
databricks-dab-examples/knowledge-base Examples that include an Azure DevOps CI/CD pipeline, a React and Lakebase app, metric views, and a uv-managed bundle

Develop bundles in Visual Studio Code

If your team develops in an integrated development environment (IDE), you can work on bundles locally with the Azure Databricks extension for Visual Studio Code. You can define, deploy, and run bundles for jobs and pipelines in a workspace without leaving the editor. The following video shows how to develop a bundle in Visual Studio Code and deploy it with GitHub Actions:

See Databricks IDE extension.

Deploy bundles with CI/CD

A typical CI/CD workflow validates the bundle and deploys it to a target workspace. Add your own test steps between validation and deployment. The following GitHub Actions workflow deploys a bundle to a staging target when changes are pushed to the main branch, such as when a pull request merges. It authenticates as a service principal with workload identity federation. GitHub doesn't store a client secret or access token, only the service principal's ID. The workflow reads the bundle target, catalog, and schema from the variables of a GitHub environment named staging. Before you use it, create a federation policy for the service principal. See Enable workload identity federation for GitHub Actions.

name: Deploy to staging
on:
  push:
    branches: [main]

permissions:
  # Required for workload identity federation.
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: staging
    env:
      DATABRICKS_AUTH_TYPE: github-oidc
      DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}
      DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }} # The service principal UUID.
      DATABRICKS_BUNDLE_TARGET: ${{ vars.ENVIRONMENT }}
      BUNDLE_VAR_catalog: ${{ vars.CATALOG }}
      BUNDLE_VAR_schema: ${{ vars.SCHEMA }}
    steps:
      - uses: actions/checkout@v4

      - name: Install Databricks CLI
        uses: databricks/setup-cli@main

      - name: Validate bundle
        run: databricks bundle validate --output json

      - name: Deploy bundle
        run: databricks bundle deploy

Environment variables that start with BUNDLE_VAR_ set the values of bundle variables, so the same bundle deploys to each environment with different catalogs and schemas. See Substitutions and variables in Declarative Automation Bundles. For a complete example, see Run a CI/CD workflow with a bundle that runs a pipeline update. The same validate-then-deploy pattern applies to other CI/CD tools, such as Azure DevOps. See Continuous integration and delivery on Azure Databricks using Azure DevOps.

Verify the deployment

After the workflow runs, check the following:

  1. In GitHub Actions, open the workflow run, and confirm that the Validate bundle and Deploy bundle steps succeeded.
  2. In the target workspace, confirm that the resources that the bundle defines are present.

Migrate existing workspace assets to bundles

If a job, pipeline, dashboard, Genie Agent, or app already exists in a workspace, you can generate its bundle configuration instead of writing it by hand. The following video covers creating a bundle, migrating workspace assets, and creating a base CI/CD pipeline:

Generate the configuration

Start from an existing bundle project. If you don't have one, see What are Declarative Automation Bundles?. Run databricks bundle generate with the resource type and the ID of the existing resource. For a job or a pipeline, for example:

databricks bundle generate job --existing-job-id <job-id>
databricks bundle generate pipeline --existing-pipeline-id <pipeline-id>

The command writes the configuration into your bundle. It also supports dashboard, genie-space, and app. See databricks bundle generate.

Bind the configuration to the existing resource

Add --bind to the generate command, or run databricks bundle deployment bind <key> <resource-id> --target <target> afterward. Binding links the bundle resource to the workspace resource, so that the next deploy updates it.

Validate and deploy to a non-production target

Run databricks bundle validate, and then deploy to a development or staging target first. See Migrate existing resources to a bundle.

Warning

If you deploy generated configuration without binding it, Declarative Automation Bundles creates a new resource instead of updating the existing one. Bind every resource before you deploy it.

Warning

The DABs migrator that the video demonstrates isn't an officially supported Azure Databricks tool. Validate generated bundles in a non-production workspace before you rely on them in CI/CD.

Organize repositories

Keep your Azure Databricks code and bundle configuration in a single Git repository. A single repository makes it easier for people and AI tools to share code and practices. If projects have separate deployment lifecycles, give each project its own bundle, and keep the bundles in the same repository. The exception is regulated industries that need separate repositories for confidentiality. See Single repository.

If confidentiality requires separate repositories, use one repository per project, with one bundle, one deployment boundary, and one owning team. Each repository's CI runs only for its own project, and a failed deploy has a clear owner.

Use trunk-based development: keep the main branch deployable, work in short-lived feature branches, and merge them back quickly.

For example, a marketing Customer 360 project and a finance revenue forecasting project each have their own bundle in the same repository. Each team deploys its project on its own schedule.

When to use each tool

Use this table to decide which tool owns a change. Terraform manages the platform, and Declarative Automation Bundles manage the projects inside a workspace, including the workspace configuration that a project needs.

Situation Use
Provision workspaces, networks, or cloud IAM Terraform
Platform settings must match across accounts or regions Terraform
Deploy Azure Databricks projects, such as jobs, pipelines, notebooks, and schemas Declarative Automation Bundles
Changes need review before they reach production Declarative Automation Bundles
A one-off notebook or prototype with a single owner, and nothing downstream depends on it Neither

Next steps

This is the last section of the Starter Journey. To revisit an earlier section, see Starter Journey: Set up Azure Databricks for production.

Resource Description
What are Declarative Automation Bundles? Learn how to create, configure, test, and deploy bundles
CI/CD on Azure Databricks Learn about CI/CD patterns, authentication, and deployment strategies on Azure Databricks