Share via

Is it possible to deploy databricks cluster across environments using CICD?

Goutham Kannekanti 1 Reputation point
2021-01-21T10:39:58.867+00:00

Hi Team,
I need your assistance in understanding Azure databricks cluster. My question is "Can we deploy the databricks cluster and libraries installed on it to other environments using CI/CD pipeline? I want to deploy it from DEV to UAT and UAT to PROD. Is it possible? If so, how can we do that?"

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,080 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 85,586 Reputation points Microsoft Employee
    2021-01-21T11:35:33.847+00:00

    Hello @Goutham Kannekanti ,

    Welcome to the Microsoft Q&A platform.

    Unfortunately it is not possible to deploy databricks using CI/CD.

    Actually CI/CD is not tied to any cluster. The CI/CD pipeline only moves your code (Notebook) from one environment to another. Attaching and running the notebook can be accomplished as part of the release pipeline but you will need to us a Batch script in your task and then install and use the Databricks CLI. Then you will need to create and run a job.

    For more details, refer to the below links:

    https://databricks.com/blog/2017/10/30/continuous-integration-continuous-delivery-databricks.html

    https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/ci-cd-azure-devops

    Hope this helps. Do let us know if you any further queries.

    ------------

    • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification.