Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Changes made by hand in the workspace UI don't scale past a single contributor. When a second person ships changes, or staging must match production, you need repeatable deployments, code review, and a promotion path from development to production. On Azure Databricks, you get these by defining your projects as code, keeping them in Git, and deploying them with continuous integration and continuous delivery (CI/CD).
Separate infrastructure from projects
Manage Azure Databricks DevOps in two layers, each with its own tool and lifecycle:
| Layer | What it includes | Tool | Examples |
|---|---|---|---|
| Platform infrastructure | Resources outside a workspace, such as accounts, networks, metastores, workspaces, and cloud Identity and Access Management (IAM) | Terraform | Terraform examples |
| Azure Databricks projects | Resources inside a workspace, such as jobs, pipelines, schemas, dashboards, apps, and registered models | Declarative Automation Bundles | Bundle configuration examples |
Keeping the layers separate means that a change to a pipeline doesn't touch your network configuration, and a platform team can manage workspaces without editing the projects that run in them. To create workspaces with Terraform, see Create development, staging, and production workspaces.
Define projects with Declarative Automation Bundles
Declarative Automation Bundles, formerly known as Databricks Asset Bundles, are an infrastructure-as-code approach to managing your Azure Databricks projects. A bundle is an end-to-end definition of a project. It includes the source files, such as notebooks and Python files, the YAML definitions of the Azure Databricks resources that the project uses, and the tests that validate it. You deploy the bundle as a single unit to a target, such as development, staging, or production. See What are Declarative Automation Bundles?.
The following video demonstrates how to work with bundles:
The Databricks CLI includes the bundle commands, so a CI/CD runner, such as a GitHub Actions runner, needs only the Databricks CLI and workspace credentials to validate and deploy a bundle. For the resources that a bundle can define, see Declarative Automation Bundles resources.
The following repositories contain example bundles:
| Repository | Contents |
|---|---|
| bundle-examples/knowledge_base | Bundles for individual resources and patterns, such as apps, dashboards, Genie Agents, jobs, pipelines, and Lakebase databases |
| databricks-dab-examples/flights | An end-to-end project on a flights dataset, in simple, advanced, and bundle template versions |
| databricks-dab-examples/knowledge-base | Examples that include an Azure DevOps CI/CD pipeline, a React and Lakebase app, metric views, and a uv-managed bundle |
Develop bundles in Visual Studio Code
If your team develops in an integrated development environment (IDE), you can work on bundles locally with the Azure Databricks extension for Visual Studio Code. You can define, deploy, and run bundles for jobs and pipelines in a workspace without leaving the editor. The following video shows how to develop a bundle in Visual Studio Code and deploy it with GitHub Actions:
Deploy bundles with CI/CD
A typical CI/CD workflow validates the bundle and deploys it to a target workspace. Add your own test steps between validation and deployment. The following GitHub Actions workflow deploys a bundle to a staging target when changes are pushed to the main branch, such as when a pull request merges. It authenticates as a service principal with workload identity federation. GitHub doesn't store a client secret or access token, only the service principal's ID. The workflow reads the bundle target, catalog, and schema from the variables of a GitHub environment named staging. Before you use it, create a federation policy for the service principal. See Enable workload identity federation for GitHub Actions.
name: Deploy to staging
on:
push:
branches: [main]
permissions:
# Required for workload identity federation.
id-token: write
contents: read
jobs:
deploy:
runs-on: ubuntu-latest
environment: staging
env:
DATABRICKS_AUTH_TYPE: github-oidc
DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}
DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }} # The service principal UUID.
DATABRICKS_BUNDLE_TARGET: ${{ vars.ENVIRONMENT }}
BUNDLE_VAR_catalog: ${{ vars.CATALOG }}
BUNDLE_VAR_schema: ${{ vars.SCHEMA }}
steps:
- uses: actions/checkout@v4
- name: Install Databricks CLI
uses: databricks/setup-cli@main
- name: Validate bundle
run: databricks bundle validate --output json
- name: Deploy bundle
run: databricks bundle deploy
Environment variables that start with BUNDLE_VAR_ set the values of bundle variables, so the same bundle deploys to each environment with different catalogs and schemas. See Substitutions and variables in Declarative Automation Bundles. For a complete example, see Run a CI/CD workflow with a bundle that runs a pipeline update. The same validate-then-deploy pattern applies to other CI/CD tools, such as Azure DevOps. See Continuous integration and delivery on Azure Databricks using Azure DevOps.
Verify the deployment
After the workflow runs, check the following:
- In GitHub Actions, open the workflow run, and confirm that the Validate bundle and Deploy bundle steps succeeded.
- In the target workspace, confirm that the resources that the bundle defines are present.
Migrate existing workspace assets to bundles
If a job, pipeline, dashboard, Genie Agent, or app already exists in a workspace, you can generate its bundle configuration instead of writing it by hand. The following video covers creating a bundle, migrating workspace assets, and creating a base CI/CD pipeline:
Generate the configuration
Start from an existing bundle project. If you don't have one, see What are Declarative Automation Bundles?. Run databricks bundle generate with the resource type and the ID of the existing resource. For a job or a pipeline, for example:
databricks bundle generate job --existing-job-id <job-id>
databricks bundle generate pipeline --existing-pipeline-id <pipeline-id>
The command writes the configuration into your bundle. It also supports dashboard, genie-space, and app. See databricks bundle generate.
Bind the configuration to the existing resource
Add --bind to the generate command, or run databricks bundle deployment bind <key> <resource-id> --target <target> afterward. Binding links the bundle resource to the workspace resource, so that the next deploy updates it.
Validate and deploy to a non-production target
Run databricks bundle validate, and then deploy to a development or staging target first. See Migrate existing resources to a bundle.
Warning
If you deploy generated configuration without binding it, Declarative Automation Bundles creates a new resource instead of updating the existing one. Bind every resource before you deploy it.
Warning
The DABs migrator that the video demonstrates isn't an officially supported Azure Databricks tool. Validate generated bundles in a non-production workspace before you rely on them in CI/CD.
Organize repositories
Keep your Azure Databricks code and bundle configuration in a single Git repository. A single repository makes it easier for people and AI tools to share code and practices. If projects have separate deployment lifecycles, give each project its own bundle, and keep the bundles in the same repository. The exception is regulated industries that need separate repositories for confidentiality. See Single repository.
If confidentiality requires separate repositories, use one repository per project, with one bundle, one deployment boundary, and one owning team. Each repository's CI runs only for its own project, and a failed deploy has a clear owner.
Use trunk-based development: keep the main branch deployable, work in short-lived feature branches, and merge them back quickly.
For example, a marketing Customer 360 project and a finance revenue forecasting project each have their own bundle in the same repository. Each team deploys its project on its own schedule.
When to use each tool
Use this table to decide which tool owns a change. Terraform manages the platform, and Declarative Automation Bundles manage the projects inside a workspace, including the workspace configuration that a project needs.
| Situation | Use |
|---|---|
| Provision workspaces, networks, or cloud IAM | Terraform |
| Platform settings must match across accounts or regions | Terraform |
| Deploy Azure Databricks projects, such as jobs, pipelines, notebooks, and schemas | Declarative Automation Bundles |
| Changes need review before they reach production | Declarative Automation Bundles |
| A one-off notebook or prototype with a single owner, and nothing downstream depends on it | Neither |
Next steps
This is the last section of the Starter Journey. To revisit an earlier section, see Starter Journey: Set up Azure Databricks for production.
| Resource | Description |
|---|---|
| What are Declarative Automation Bundles? | Learn how to create, configure, test, and deploy bundles |
| CI/CD on Azure Databricks | Learn about CI/CD patterns, authentication, and deployment strategies on Azure Databricks |