Create a databricks job cluster and deployment

NIKHIL KUMAR 101 Reputation points
2024-01-15T15:03:24.7633333+00:00

We are creating a databricks workflow with a job cluster, is there a way :

  1. To create anddeploy the databricks worflow with job cluster using azure devops CI/Cd pipeline ?
  2. Can we directly create job cluster from the portal compute section ? ( Not going via worklow and create)
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,330 questions
{count} votes

1 answer

Sort by: Most helpful
  1. PRADEEPCHEEKATLA 90,541 Reputation points
    2024-01-16T00:16:30.68+00:00


    NIKHIL KUMAR
    - Thanks for the question and using MS Q&A platform.

    For your first question, yes, you can create and deploy a Databricks workflow with a job cluster using Azure DevOps CI/CD pipeline. You can use the Databricks CLI to create and manage Databricks resources, including clusters, jobs, and notebooks. You can also use the Databricks REST API to create and manage these resources. You can integrate these tools into your Azure DevOps pipeline to automate the creation and deployment of your Databricks workflow.

    For more details, refer to Create and run jobs using the CLI, API, or notebooks.

    For your second question, there is no option to create a job cluster directly from the "Compute" tab in the Databricks workspace portal but, you can choose the compute cluster created under compute tab. User's image

    To create a job cluster, you can follow these steps:

    1. Navigate to your Databricks workspace and click on the "Workflows" tab.
    2. Click on the Jobs and "Create Job" button.
    3. From here you can create a new job cluster or choose from existing compute clusters as shown.

    User's image

    Hope this helps. Do let us know if you any further queries.


    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

    1 person found this answer helpful.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.