Is it possible to deploy databricks cluster across environments using CICD?

Question

Hi Team,
I need your assistance in understanding Azure databricks cluster. My question is "Can we deploy the databricks cluster and libraries installed on it to other environments using CI/CD pipeline? I want to deploy it from DEV to UAT and UAT to PROD. Is it possible? If so, how can we do that?"

Answer 1

Hello @Goutham Kannekanti ,

Welcome to the Microsoft Q&A platform.

Unfortunately it is not possible to deploy databricks using CI/CD.

Actually CI/CD is not tied to any cluster. The CI/CD pipeline only moves your code (Notebook) from one environment to another. Attaching and running the notebook can be accomplished as part of the release pipeline but you will need to us a Batch script in your task and then install and use the Databricks CLI. Then you will need to create and run a job.

For more details, refer to the below links:

https://databricks.com/blog/2017/10/30/continuous-integration-continuous-delivery-databricks.html

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/ci-cd-azure-devops

Hope this helps. Do let us know if you any further queries.

------------

• Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
• Want a reminder to come back and check responses? Here is how to subscribe to a notification.