Data Factory - Internal Server Error and AzureResourceProviderThrottling errors

Question

Data Factory - Internal Server Error and AzureResourceProviderThrottling errors

Quinn, Katie 1

We had multiple Data Factory pipelines fail over the last few days (4/11-4/13) with several intermittent errors.

One example of the Internal Server Error we saw:

The internal server errors were on pipeline steps that were transforming the data and not moving data from a source to a sink. Is there a way to diagnose Internal Server Errors within Data Factory?

Another error we were seeing was specific to an Azure Resource Provider Throttling error. The following error message appeared:

Unexpected failure while waiting for the cluster (0412-081827-waxen351) to be ready.Cause Unexpected state for cluster (0412-081827-waxen351): AZURE_RESOURCE_PROVIDER_THROTTLING(CLOUD_FAILURE): azure_error_code:AzureResourceProviderThrottling,azure_error_message:Encountered Azure Resource Provider throttling. Please try again later. Details: ,databricks_error_message:Error code: AzureResourceProviderThrottling, error message: Encountered Azure Resource Provider throttling. Please try again later.

Is there a reason why we would be seeing an Azure Resource Provider Throttling error message? Was there an outage or planned maintenance with databricks?

The errors were intermittent, however, we saw enough instances of these errors across our pipelines to be concerned.

1 answer

Your answer

Answer 1

KranthiPakala-MSFT 46,642 Microsoft Employee Moderator

Hi @Quinn, Katie ,

Welcome to Microsoft Q&A forum and sorry for your experience.

We usually notice the internal server error when there is an issue with the ADF dependent service Databricks.
As per my conversation with internal team, there was an outage reported by Databricks service on 4/13 probably that could be the reason you are seeing these errors. But the issue is resolved now.

In case if you still continue to receive these errors please do share the latest pipeline and activity runID's for the failed ones so that we can escalate to product team to have a deeper analysis.

You can also check the status of databricks from the status page here: https://status.azuredatabricks.net/
This page also contains info about the planned maintenance.

Hope this info helps. Please do share the pipeline and activity runID's if you ever notice these errors.

Thank you

----------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

Quinn, Katie 1 Reputation point

2021-04-15T20:39:04.353+00:00

Hi @KranthiPakala-MSFT

I just saw another instance of an Internal Server Error. I did not expect there to be another Databricks outage based on their status page. Would you be able to provide some insight into this issue?
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2021-04-15T20:49:29.663+00:00

Hi @Quinn, Katie ,

Sorry for your experience and thanks for getting back. Could you please share the failed pipeline and activity runID so that I can escalate to internal team to check on the backend logs to figure out the root cause.

We look forward to your response.

Thank you
Quinn, Katie 1 Reputation point

2021-04-15T20:55:11.467+00:00

Hi @KranthiPakala-MSFT ,

I believe the pipeline runID is 58619738-03b6-4279-b8d2-df95448dcad7 and the activity runID is d167158a-7071-4b05-9f81-d92d8c2e3176.
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2021-04-15T21:17:47.763+00:00

Hi @Quinn, Katie ,

Thanks for the details. I will escalate this to internal team and will get back to you as soon as I have an update on the same.

Thank you for your patience.

Share via

Data Factory - Internal Server Error and AzureResourceProviderThrottling errors

1 answer

Your answer