Synapse Spark "Request exceeds throttling limits"

Julian Breunung 22

I'm using a Synapse Spark Notebook to sequentially integrate multiple parquet files into a delta table.
I'm running into the following issue:

AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Metadata service API com.microsoft.catalog.metastore.sasClient.SasClient$API@3219edf6 failed with status -1 (null) Response Body ({"result":"DependencyError","errorId":"TooManyRequests","errorMessage":"BBCServiceException is [Request exceeds throttling limits: PerClusterNodesReads1Min=15, throttlingKey=1133da19-6380-42e5-82b3-b09287fb1fc0; Retry after 5 seconds.]. TraceId : 015adc1c-c4b6-4d24-b2db-16cedb950c59. Error Component : BBC"}))

Where is the throttiling limit defined? Can it be adjusted? Is there any way to mitigate the issue?

HimanshuSinha-msft 19,386 Reputation points Microsoft Employee

2022-08-19T22:41:22.133+00:00
Hello @Anonymous ,
Thanks for the question and using MS Q&A platform.

As we understand the ask here is how to get around with the throttling issue , please do let us know if its not accurate.
The error message is very clear askinh you to indice a 5 sec delay , whcih may / may not be possible . I don't think you can set the value .
I think you can scale up the cluster and that should help .

Please do let me if you have any queries.
Thanks
Himanshu

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Want a reminder to come back and check responses? Here is how to subscribe to a notification

If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
Julian Breunung 22 Reputation points

2022-08-22T06:02:28.867+00:00

Hi @HimanshuSinha-msft ,

Thanks for the quick replay. First of all I'm trying to understand why the throttling occurs and where.
The error message talks about the SasClient API. Is it a limitation on the Storage Account or is it a limitation on Synapse side?
Since I'm working only on one storage account in the given context I'm not quite sure why so many requests are made.

Regards,
Julian
Andrine Elsetrønning 1 Reputation point

2022-08-22T07:31:26.79+00:00

Hello!

My team also got this error last week. Is this a new limit?
Our pipeline-setup had been working perfectly until thursday evening (around 8.00 pm, the 18th UTC).
Where is the definition of this limitation? Is there some workaround that does not involve scaling up?
Peters Roel EXT 1 Reputation point

2022-08-23T12:51:12.823+00:00

We are also encountering this problem.
There's no way to tryCatch the error and insert a delay, since it's a Spark error.
Upscaling the cluster didn't help either.

Share via

Synapse Spark "Request exceeds throttling limits"