Quota exceeded error when using ADF to trigger Databricks using job cluster

Chand, Anupam SBOBNG-ITA/RX 451 Reputation points
2023-04-04T09:29:32.98+00:00

In our data platform, we are using ADF to trigger databricks jobs. The legacy platform was using interactive cluster but as per Microsoft recommendations, we are shifting this to job clusters for cost benefit reasons. Previously when we trigger jobs via ADF, they all ran on the same interactive cluster. This meant shared resources and sometimes the jobs would take longer and occasionally fail. Now however, the job clusters are still part of the quota assigned at subscription level. Since each cluster is dedicated to each pipeline, when multiple pipelines run together, we may suddenly require a burst of cores and our pipelines are again failing due to quota exceeded exceptions. What is the best practice here when using job cluster and still prevent quota exceeded exceptions?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,964 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,724 questions
{count} votes