Automate spin up of a cluster and an azure databricks workspace

Question

Hi, I am working on spinning up a cluster and a databricks workspace in a Resource Group. The idea is to create a template and minimize manual intervention, using the below code and not sure on how to setup the yaml file. If there any resources for doing same could you please share.

import json
import logging
import yaml
import base64
import requests
import time
from azure.common.credentials import ServicePrincipalCredentials
from azure.mgmt.resource import ResourceManagementClient
from azure.mgmt.resource.resources.models import DeploymentMode

YAML_VARS = "DemoDBWorkspace\\databricks_workspace_vars.yaml"
JSON_TEMPLATE_PATH = "DemoDBWorkspace\\databricks_premium_workspaceLab.json"
RESOURCE_GROUP_PARAMS = {"location": "eastus"}
RESOURCE_GROUP_NAME = "DemoRG"
JSON_REQUEST_PATH = "DemoDBWorkspace\\deploy_databricks_cluster_restapi.json"

Thank you

Answer 1

Hello @Raj D ,

Thanks for the question and using Microsoft Q&A platform.

To deploy Workspace using the ARM template.

Note: ARM templates are utilized in order to deploy Azure Databricks workspace.

Automation options - There are a few options available to use the Azure Databricks APIs:

• Databricks Terraform Resource Provider could be combined with Azure provider to create an end-to-end architecture, utilizing Terraform’s dependency and state management features.
• Python (or any other programming language) could be used to invoke the APIs (sample solution) providing a way to integrate with third-party or homegrown DevOps tooling.
• A readymade API client like Postman could be used to invoke the API directly.

For more details, refer Automate Azure Databricks Platform Provisioning and Configuration.

Hope this helps. Do let us know if you any further queries.

------------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.