Key 429 : Workflow

Question

Hello Team,

I am trying to run the 70 notebooks in single workflow in Azure data Bricks. and i am getting Key : 429 :

Error :
HttpResponseError Traceback (most recent call last)
/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/azure/appconfiguration/_azure_appconfiguration_client.py in get_configuration_setting(self, key, label, etag, match_condition, **kwargs)
256 try:
--> 257 key_value = self._impl.get_key_value(
258 key=key,

/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/azure/appconfiguration/_generated/operations/_azure_app_configuration_operations.py in get_key_value(self, key, label, accept_datetime, if_match, if_none_match, select, **kwargs)
450 error = self._deserialize.failsafe_deserialize(_models.Error, response)
--> 451 raise HttpResponseError(response=response, model=error)
452

HttpResponseError: Operation returned an invalid status 'Too Many Requests'
Content: {"type":"https://azconfig.io/errors/too-many-requests","title":"Resource utilization has surpassed the assigned quota","policy":"Total Requests Short Term","status":429}

During handling of the above exception, another exception occurred:

KeyError Traceback (most recent call last)
<command-1746602509716521> in <cell line: 8>()
6
7
----> 8 DLH_CLIENT_ID = app_config_client.get_configuration_setting(key='DLHAppConfig:DLHAppClientId',label='DLHApp').value
9 DLH_TENANT_ID = app_config_client.get_configuration_setting(key='DLHAppConfig:DLHAppTenantId',label='DLHApp').value
10 DLH_CLIENT_SECRET = app_config_client.get_configuration_setting(key='DLHAppConfig:DLHAppClientSecret',label='DLHApp').value

/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/azure/core/tracing/decorator.py in wrapper_use_tracer(*args, **kwargs)
76 span_impl_type = settings.tracing_implementation()
77 if span_impl_type is None:
---> 78 return func(*args, **kwargs)
79
80 # Merge span is parameter is set, but only if no explicit parent are passed

/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/azure/appconfiguration/_azure_appconfiguration_client.py in get_configuration_setting(self, key, label, etag, match_condition, **kwargs)
267 return None
268 except HttpResponseError as error:
--> 269 e = error_map[error.status_code]
270 raise e(message=error.message, response=error.response)
271 except binascii.Error:

Please advise.

Regards
Rohit

Answer 1

Hello @Rohit Kulkarni ,

Thanks for the question and using MS Q&A platform.

You may experience this issue, when the requests that exceed the rate limit.

To ensure high quality of service under heavy load, Databricks enforces rate limits for all REST API calls. Limits are set per endpoint and per workspace to ensure fair usage and high availability.

Requests that exceed the rate limit return a 429 response status code.

For information on rate limits for API requests, see API rate limits.

For more details, refer to Databricks REST API reference - Rate limits.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is jhow you can be part of Q&A Volunteer Moderators