Is it possible to define Auto-termination activity in Azure Databricks Jobs cluster

ASHWINI GAIKWAD 20 Reputation points
2023-12-04T17:27:04.91+00:00

Hello Team,

While creating azure databricks job cluster I figured out that auto-termination is disabled.

Requirement is to keep job cluster active for 5 mins even when the job fails. So is there any possible way to achieve it?

Note: Interactive cluster is not used in this case. Above requirement is to only achieve with job cluster if it is possible.

Regards,

Ashwini G

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,059 questions
{count} votes

Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 84,941 Reputation points Microsoft Employee
    2023-12-05T04:34:45.14+00:00

    @ASHWINI GAIKWAD - Thanks for the question and using MS Q&A platform.

    When you run a job on a New Job Cluster (which is usually recommended), the cluster terminates and is unavailable for restarting when the job is complete. On the other hand, if you schedule a job to run on an Existing All-Purpose Cluster that has been terminated, that cluster will autostart.

    For more details, refer to Azure Databricks - Terminate a cluster and Auto termination is disabled when starting a job cluster.

    Hope this helps. Do let us know if you any further queries.


    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

    0 comments No comments

0 additional answers

Sort by: Most helpful