Quota exceeded error when using ADF to trigger Databricks using job cluster

Chand, Anupam SBOBNG-ITA/RX 461

In our data platform, we are using ADF to trigger databricks jobs. The legacy platform was using interactive cluster but as per Microsoft recommendations, we are shifting this to job clusters for cost benefit reasons. Previously when we trigger jobs via ADF, they all ran on the same interactive cluster. This meant shared resources and sometimes the jobs would take longer and occasionally fail. Now however, the job clusters are still part of the quota assigned at subscription level. Since each cluster is dedicated to each pipeline, when multiple pipelines run together, we may suddenly require a burst of cores and our pipelines are again failing due to quota exceeded exceptions. What is the best practice here when using job cluster and still prevent quota exceeded exceptions?

Matheus Fanali Giraldes 0 Reputation points

2023-04-04T20:00:36.5533333+00:00

Hi, Obviously, it depends on your purpose, but I recently found that Databricks recommends sometimes using a larger cluster for multiple robust jobs over a lot of tiny clusters, resulting in more processing time at a higher cost. I found in the Azure Databricks documentation very useful tips to help you in your case. https://learn.microsoft.com/en-us/azure/databricks/clusters/cluster-config-best-practices I hope it helps!
PRADEEPCHEEKATLA-MSFT 85,586 Reputation points Microsoft Employee

2023-04-10T08:05:56.33+00:00

Chand, Anupam SBOBNG-ITA/RX - Just checking in to see if the below answer provided by @
Matheus Fanali Giraldes helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Chand, Anupam SBOBNG-ITA/RX 461 Reputation points

2023-04-10T08:13:17.2166667+00:00

@PRADEEPCHEEKATLA-MSFT No it does not. The question is seeking guidance on the setting of quota limits while using job cluster as compared to interactive clusters. The answer provided is about using clusters of different sizes. I do not see the relation. As per the link which has been provided below it clearly says : "Once you’ve completed implementing your processing and are ready to operationalize your code, switch to running it on a job cluster. Job clusters terminate when your job ends, reducing resource usage and cost."
Matheus Fanali Giraldes 0 Reputation points

2023-04-10T14:26:34.9233333+00:00

I see. Have you tried to configure the Resource Manager? https://learn.microsoft.com/en-us/azure/azure-resource-manager/troubleshooting/error-resource-quota?tabs=azure-cli#solution this link give a step-by-step how to do it. Furthermore, @PRADEEPCHEEKATLA-MSFT answered another similar question about quotas` troubleshooting: https://learn.microsoft.com/en-us/answers/questions/611894/cluster-terminated-quota-exceeded-expectation-whil. Let me know if that works.

Share via

Quota exceeded error when using ADF to trigger Databricks using job cluster