Azure Databricks Too Many Requests errors

Matthieu Marshall 6

We are getting many errors with loading notebooks and also now running jobs on clusters due to Databricks saying it has too many requests.

For example, getting the below error message:

run failed with error message Cluster '0724-103023-f2llqh3p' was terminated. Reason: INIT_SCRIPT_FAILURE (CLIENT_ERROR). Parameters: instance_id:7755cb572caf4194bded4e921404b0a8, databricks_error_message:REQUEST_LIMIT_EXCEEDED: Your workspace(8937165168112498) is sending too many requests..

Also when just having a notebook open:

User's image

But I don't think we are issuing too many requests, our workloads have not gone up particularly recently.

How do I monitor this and see what is happening? What kind of requests is it talking about?

Bhargava-MSFT 31,116 Reputation points Microsoft Employee

2024-07-24T19:06:45.7333333+00:00

Hello Matthieu Marshall,

To monitor and understand the 'too many requests' errors in Databricks, you can leverage the comprehensive monitoring and logging capabilities provided by Azure Databricks.

Please check the documents below for detailed guidance.

https://medium.com/microsoftazure/comprehensive-look-at-azure-databricks-monitoring-logging-356f89128315

https://learn.microsoft.com/en-us/azure/architecture/databricks-monitoring/application-logs

I hope this helps.
Matthieu Marshall 6 Reputation points

2024-07-25T08:30:31.3366667+00:00

Thank you. Are you able to help me with more information about what kind of requests these are that cause this issue?

Is it too many jobs running at once or per hour?

Too many requests to the databricks API in general (potentially implicit through using the UI too)?

Too many spark jobs at once or per hour?
Bhargava-MSFT 31,116 Reputation points Microsoft Employee

2024-07-25T19:12:45.6333333+00:00

Hello @matthieu marshall

Too Many Jobs Running at Once or Per Hour:

Databricks has a limit on the number of concurrent jobs that can run. If you exceed this limit, you might see errors like the one you mentioned. A workspace is limited to 1000 concurrent task runs. If this limit is exceeded, you will receive a "Too Many Requests response" message

Too Many Requests to the Databricks API:

This includes both explicit API calls and implicit requests made through the UI. If your workspace is making too many requests in a short period, you might hit the request limit. This can happen even if your workloads haven't increased, as the limit is based on the number of requests.

Too Many Spark Jobs at Once or Per Hour:

Similar to the job limit, there will be a limit on the number of Spark jobs that can run concurrently or within a specific time frame. This can also contribute to the request limit being exceeded.

I hope this helps.
Matthieu Marshall 6 Reputation points

2024-07-29T14:17:21.5466667+00:00

Hello. Thank you for your latest answer although I've still not been able to solve the problem. I had to go away and find the limits that do exist in documentation (for others, they can be found here https://learn.microsoft.com/en-us/azure/databricks/resources/limits ). We don't think we are close to hitting those limits.

Anyway, also having reflected on it, we think the error message suggests that we are sending too many requests to from our workspace to other places causing network traffic issues. If possible, could you please confirm what the root cause of this specific error message is?

"REQUEST_LIMIT_EXCEEDED: Your workspace(8937165168112498) is sending too many requests.."
Bhargava-MSFT 31,116 Reputation points Microsoft Employee

2024-08-05T20:00:39.2566667+00:00

Hello Matthieu Marshall,

Sorry for the delayed response.

The error you are experiencing might be due to the workspace hitting the backend pod limit.

It is recommended to set a job delay or interval between the pipelines to avoid overload.

If this doesn't help, we need to further troubleshoot the issue by checking the logs.

I suggest opening a support case so that a support engineer can collect the logs and work with the product group.

If you don't have a support plan, please let me know so that I can provide a one-time free support request to work on this case.

I look forward to hearing from you.
Matthieu Marshall 6 Reputation points

2024-08-06T08:06:11.7433333+00:00

Thank you @Bhargava-MSFT !

I did raise a support request for this in the end last week because we were quite stuck. I have forwarded your suggestion about backend pod limit onto the support engineer too. Are you able to get involved with the support request if I share the details of it with you?

We are triggering the overwhelming majority of our jobs from Azure Data Factory. If you know how to get Azure Data Factory to limit it's usage of the Databricks API, that would be interesting, but otherwise I would find it bizarre that we need to limit our usage to run at most 100 jobs an hour.
Bhargava-MSFT 31,116 Reputation points Microsoft Employee

2024-08-06T17:24:49.24+00:00

Thank you for the additional details. I have reviewed your support ticket in the ticketing system. Currently, the support engineer is waiting on a response from the Product Group.

I suggest waiting until the PG provides more details.
Matthieu Marshall 6 Reputation points

2024-08-07T11:52:47.38+00:00

Thanks! It has been shared with us the full log error is

2024/08/06 03:47:59.221 WARN com.databricks.rpc.armeria.DatabricksServer$[tenant=8937165168112498 traceId=f72002a3d05f54b9d4938b94541c2f9c spanId=84fa2313f8174864 requestId=cbfd8712-e741-9ee2-88d4-4f328372a7d8 serverEventId=CgsI77nGtQYQ7r/bKRiUAToQdscTc0qkQHabYIH/+T4AoA== thread=ProjectsServiceFilesThreadPool-83]: [sreqId=6aea71bb, chanId=1f893b79, caddr=4.210.211.8, raddr=10.2.11.95:34622, laddr=10.2.186.162:3414][h1://cons-projects-0/api/2.0/workspace-files/list-files#GET] Unexpected exception: com.databricks.api.base.DatabricksServiceException: REQUEST_LIMIT_EXCEEDED: Your workspace(8937165168112498) is sending too many requests. com.databricks.api.base.DatabricksServiceException: REQUEST_LIMIT_EXCEEDED: Your workspace(8937165168112498) is sending too many requests.

So, this is from an ls of a folder that contains too many files. We are awaiting more details, but this is the likely cause and we are aware of a folder with 27000 notebooks in which we can clean up

Share via

Azure Databricks Too Many Requests errors

Your answer