Why are my Data Factory pipelines failing ocassionally due to same error: "request_throttled, Message: Your app has been throttled by AAD due to too many requests"?

61521973 31

Hi!

I use Data Factory for multiple projects (different tenants) that have been running daily for months, some of them for over one year.
Along my pipelines, some linked-services are configured by getting secrets from Key vaults resided in same resource groups as data factory.

In the past 2 weeks, in some of the nightly loads, an error has ocurred in any of the following activities: Lookup, Copy activity, Web activity due to the same error:

"
Error code
2403

Details
Get access token from MSI failed for Datafactory ddc-df-prod, region we. Please verify resource url is valid and retry. Details: Acquire MI token from AAD failed. ErrorCode: request_throttled, Message: Your app has been throttled by AAD due to too many requests. To avoid this, cache your tokens see https://aka.ms/msal-net-throttling.
"

As the error message suggests, I should "cache my tokens". I am not really sure that I am able to achieve this in data factory.
My pipelines are failing "sometimes" and I have not changed anything in the last weeks.

Thanks in advanced,
Gilfredo

Daniel Sanders 1 Reputation point

2022-09-28T09:51:48.023+00:00

We are seeing the same issue at 2 different cliënt.

Looking forward to an answer or workaround
ShaikMaheer-MSFT 38,401 Reputation points Microsoft Employee

2022-09-29T06:50:02.49+00:00

Hi @61521973 ,

Thank you for posting query in Microsoft Q&A Platform.

Looks strange issue. Are pipelines are running good after failure when you re-run them?
61521973 31 Reputation points

2022-09-29T07:06:13.493+00:00

Yes, They run good afterwards. I think must of the failures happen between 2.30-4.00 am CET.
During our working day they go fine.

We have implemented "Retry" + "Retry interval" at 90 seconds. But the error has happened again.
ShaikMaheer-MSFT 38,401 Reputation points Microsoft Employee

2022-09-29T07:08:21.943+00:00

Thank you. I am trying to check with internal team if they can share any insights about this. I will post updates here once I hear back.
Boris Martinkevich 1 Reputation point

2022-09-29T08:34:48.677+00:00

Piggybacking on this one.

We are also seeing this more frequently the last couple of weeks on different tenants.
Sometimes, the very first child pipeline from the trigger is getting the error. Always around 04 CEST (UTC +2).

A rerun afterwards fixes it.
Nestor 46 Reputation points

2022-09-29T09:21:10.207+00:00

Ditto... having the same issue and happening at random times between 3:00am and 4:00am CET.
I cannot afford to rerun during the day, so I ended up removing the dependency on Key Vault for now.
Waiting on a support request to understand why this is happening and how to mitigate it.

I notice almost everyone is indicating CET time zone, wondering if it's related only to this region (our tenant is in West Europe).
ShaikMaheer-MSFT 38,401 Reputation points Microsoft Employee

2022-09-29T12:45:07.863+00:00

Hi @61521973 ,

I just got confirmation from internal team too, that there was AAD outage in few regions that could have lead to these issues. Apologies for the inconvenience. In case of urgency, I would encourage you to raise support ticket. Thank you.
Daniel Sanders 1 Reputation point

2022-10-04T07:15:01.75+00:00

Thanks for this confirmation. This issue wasn't reporter on status.azure.com, nor can I find it today is the historica issues.

Is there another platform/monitor/website where we might have been able to spot this?

Share via

Why are my Data Factory pipelines failing ocassionally due to same error: "request_throttled, Message: Your app has been throttled by AAD due to too many requests"?