ADLS Gen 2 ErrorCode=UserErrorDataStoreServiceThrottling

Question

ADLS Gen 2 ErrorCode=UserErrorDataStoreServiceThrottling

Lokesh 211

We have noticed that our QA ADLS storage account gives performance issues intermittently and one such issues is related to throttling and time out.

We had raised a ticket earlier as well, however, didn't get any satisfactory explanation.

Could anyone from Microsoft work with us to resolve this?

ErrorCode=UserErrorDataStoreServiceThrottling,'Type=Microsoft.DataTransfer.Common.Shared.DataStoreThrottlingException,Message=Failed with potential throttling error when accessing AzureBlobFS at side. You are suggested to check and increase the allowed request rate for the data store, or reduce the concurrent workload. More details please refer https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-performance-troubleshooting.. Details: Account: 'xxxx'. FileSystem: 'xxx'. ErrorCode: 'OperationTimedOut'. Message: 'Operation could not be completed within the specified time.'. RequestId: 'af91c927-301f-0058-6732-e89460000000'. TimeStamp: 'Tue, 25 Oct 2022 05:28:29 GMT'.. ,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Azure.Storage.Data.Models.ErrorSchemaException,Message=Operation returned an invalid status code 'InternalServerError',Source=Microsoft.DataTransfer.ClientLibrary,'

0 comments

1 answer

Your answer

Answer 1

MartinJaffer-MSFT 26,161

Hello and welcome to Microsoft Q&A @Lokesh . My condolences your support ticket didn't give a satiffactory explanation. If you share the ticket number with me, I can take a look.

Taking apart this error, I see several things:

Failed with potential throttling error when accessing AzureBlobFS at side

This would suggest there are so many read/write requests at once, a limit was hit on the storage service. The default maximum request rate per storage account is 20000 requests per second for standard Azure Storage accoutns. For requests on a single blob, 500 requests per second. You can go to the storage account and check the metrics to see if you get anywhere near these.

Usually, it would say "at Source side" or "at Sink side". Just "at side" doesn't seem right.

Operation could not be completed within the specified time

This one is a little more ambiguous. Does it mean the pipeline activity timed out, or some underlying service timed out? If it is the pipeline activity, that happens when none of the retries is successful. How long did this pipeline run? How long did this activity run? This would give better hints.

Operation returned an invalid status code 'InternalServerError

Internal server error is when something breaks server-side and the request cannot be meaningfully completed. The cause behind internal server error is not something I can look into. If you tell me your support ticket number, I can ask them to look into the logs, with that requestID.

Azure.Storage.Data.Models.ErrorSchemaException

This sounds like there is something wrong with the dataset schema.

So if I had to create a story from these errors, I would guess...
You have a forEach loop containing many copy activities running in parallel. There is something corrupted in the schema so the copy fails. The copy activity retries until it times out. However there are so many requests, that some get throttled.

Lokesh 211 Reputation points

2022-10-27T04:32:13.707+00:00

Thank you Martin for getting back.
The ticket number is 2210260010001587.

There are nested for each loop iteration. The first one is set to a batch number of 5 and the second one to the default batch number which I believe would be 50.

I think at any given point in time it should not exceed 5 x 50 parallel threads.

Looked at the insights under Monitoring blade, and the transactions did not go beyond 8k.

Another observation is there is a time when the availability went to 0 percent, however, the number of transactions at that time was 23.
Would like to reitrate, we see this behaviour only in QA RG. In dev we have never seen this issue.
(we have also seen that sometimes when you try to download blobs it takes forever. )

Additionally, the schema has no role to play here. since its a straightforward API to csv conversion. Able to create 98 percent of such files, only a few error out because of throttling issues. So this error message could be misleading.
MartinJaffer-MSFT 26,161 Reputation points

2022-10-31T21:38:00.05+00:00

A for-each inside a for-each? Data Factory doesn't let me do that.
However, lets suppose you do have 5x50 instances running. Each Copy activity can also have multiple parallel connections, depending upon settings and connector. Lets suppose each copy activity tries for 4 parallel read/writers. 4 x 5 x 50 = 20 x 50 = 1000. 1000 > 500.

Definition of availability metric:

The percentage of availability for the storage service or the specified API operation. Availability is calculated by taking the TotalBillableRequests value and dividing it by the number of applicable requests, including those that produced unexpected errors. All unexpected errors result in reduced availability for the storage service or the specified API operation.

Definition of Transaction metric:

The number of requests made to a storage service or the specified API operation. This number includes successful and failed requests, as well as requests which produced errors. Use ResponseType dimension for the number of different type of response.

I haven't found a definition of TotalBillableRequests on the Azure-monitor metrics page. However, 23 does sound too small. I wonder how it handles long-running requests.

Error messages frequently are misleading, there are so many layers, the cause gets obfuscated.

Thank you for sharing the ticket number.
Looking at the case notes, looks like the support engineer was trying to get your availability details to schedule call with you and a storage engineer. They are/were waiting on your reply. @Lokesh
Lokesh 211 Reputation points

2022-11-08T06:19:19.437+00:00

Thanks, Martin. We are using foreach loop and then calling a child pipeline to invoke another foreach loop activity.

Had a call with the support engineer and unfortunately, they are not able to find the root cause.
The data volume is very less and it looks like MS team is suggesting this could be due to some load balancing issues happening at the server side.
MartinJaffer-MSFT 26,161 Reputation points

2022-11-22T18:22:05.3+00:00

Thanks for letting me know. Load balancing actually makes sense of much of the errors.

Share via

ADLS Gen 2 ErrorCode=UserErrorDataStoreServiceThrottling

1 answer

Your answer