How to Fix the InternalServerError in Executing a ADF Pipeline?

Nitin Patil 20

I have a foreach loop in my Azure Data Factory pipeline with three activities. The first activity uses a Self-Hosted Integration Runtime (SHIR) to copy data from an On-premises SQL Server to Azure Data Lake Storage Gen2 (ADLS2). The second activity is a pipeline (PL_Delta_Infinity_ADLS_To_ASQL) that invokes a data flow (DF_Upsert_To_ASQL_Infinity). This data flow reads data from ADLS2 and performs an upsert operation to write it into an Azure SQL Database using the Alter row transformation.

Up until 10 AM today, everything was functioning correctly. However, after 10 AM, the second activity began encountering an error. The error code reported is "InternalServerError," with a failure type of "system error," and the details indicate "InternalServerError executing request." This issue specifically occurs during the process of reading data from ADLS2 and writing it to the Azure SQL Database.

I have verified that the Azure SQL Database is accessible and operational by connecting through SQL Server Management Studio (SSMS). Additionally, I confirmed the functionality of the linked services to ADLS2 and the Azure SQL Database by successfully previewing the data through them.

I'm seeking assistance to understand the root cause of this issue and to troubleshoot the "InternalServerError" error affecting the data flow operation in my Azure Data Factory pipeline. Any guidance on resolving this would be greatly appreciated.

Bhargava-MSFT 31,116 Reputation points Microsoft Employee

2024-07-22T19:05:01.5266667+00:00
Hello Nitin Patil,

Internal server error could be a transient issue. Could you please retry and see if it fixes the issue?

Additionally, please check the below:

Integration Runtime Resources:

Ensure that the Integration Runtime has enough cores and memory to handle the data flow task. Resource constraints can sometimes lead to internal server errors.

Self-Hosted Integration Runtime.

Verify that the SHIR is running properly.

Ensure that the SHIR has the necessary permissions to access the On-premises SQL Server and ADLS2.

Service Health:

Check the Azure Status page for any ongoing service issues in your region that might be affecting your pipeline.
Nitin Patil 20 Reputation points

2024-07-23T12:36:31.5166667+00:00

@Bhargava-MSFT

The data integration pipeline involves two main activities: transferring data from an on-premises SQL Server to Azure Data Lake Storage Gen2 (ADLS2), and then from ADLS2 to an Azure SQL Database. The latter activity, which uses the AutoResolveIntegrationRuntime, has encountered failures with an "Internal server error." This runtime scales automatically, limiting manual core and size management.

Interestingly, the first activity, transferring data from on-premises SQL to ADLS2, utilizes a Self-Hosted Integration Runtime (SHIR) and has never failed. Despite this, I reviewed its configuration: Standard D2s v3 (2 vCPUs, 8 GiB memory), which has shown no health events or Azure outages. Notably, the highest CPU utilization over the last 7 days was 9.6%, with peak disk read/write bytes of 3.6 GiB and 5.01 GiB respectively, and disk read/write per second at 1.28 and 2.

This data indicates that metrics are within acceptable limits. The pipeline involves transferring data from approximately 8 on-premises tables to ADLS2 and then to Azure SQL Database.

However, the pipeline encountered multiple failures yesterday for activity "PL_Delta_Infinity_ADLS_To_ASQL" with the recurring "Internal server error." I checked Azure Service health, which reported only one warning related to "1 emerging issue under investigation: CrowdStrike Falcon agent guidance."

Could this issue be related to the recent CrowdStrike matter or another Azure-related issue? What exactly is the "Internal server error," and you mentioned it's transient; what causes it?

I'm seeking this information to enhance my understanding. Your assistance in clarifying these points would be greatly appreciated.
Nitin Patil 20 Reputation points

2024-07-23T12:38:32.07+00:00

@Bhargava-MSFT

The data integration pipeline involves two main activities: transferring data from an on-premises SQL Server to Azure Data Lake Storage Gen2 (ADLS2), and then from ADLS2 to an Azure SQL Database. The latter activity, which uses the AutoResolveIntegrationRuntime, has encountered failures with an "Internal server error." This runtime scales automatically, limiting manual core and size management.

Interestingly, the first activity, transferring data from on-premises SQL to ADLS2, utilizes a Self-Hosted Integration Runtime (SHIR) and has never failed. Despite this, I reviewed its configuration: Standard D2s v3 (2 vCPUs, 8 GiB memory), which has shown no health events or Azure outages. Notably, the highest CPU utilization over the last 7 days was 9.6%, with peak disk read/write bytes of 3.6 GiB and 5.01 GiB respectively, and disk read/write per second at 1.28 and 2.

This data indicates that metrics are within acceptable limits. The pipeline involves transferring data from approximately 8 on-premises tables to ADLS2 and then to Azure SQL Database.

However, the pipeline encountered multiple failures yesterday with the recurring "Internal server error." I checked Azure Service health, which reported only one warning related to "1 emerging issue under investigation: CrowdStrike Falcon agent guidance."

Could this issue be related to the recent CrowdStrike matter or another Azure-related issue? What exactly is the "Internal server error," and you mentioned it's transient; what causes it?

I'm seeking this information to enhance my understanding. Your assistance in clarifying these points would be greatly appreciated.

Accepted answer

Bhargava-MSFT 31,116 Reputation points Microsoft Employee

2024-07-24T22:50:54.6366667+00:00

Hello Nitin Patil,

Thanks for providing the additional details. After further checking with my internal team, the failures could be due to a transient networking issue.

This error should be very rare, and retries should typically mitigate this.

If the retries do not resolve the issue, please raise a support ticket so that the support team can engage with the product group to investigate further.

If you don't have a support plan, please let me know so I can provide a one-time free support request to address this.

I look forward to hearing from you.
Please sign in to rate this answer.

1 person found this answer helpful.
Nitin Patil 20 Reputation points

2024-07-25T17:55:32.04+00:00

@Bhargava-MSFT I tried the next day, and all was working fine. Thanks for your support.

Bhargava-MSFT 31,116 Reputation points Microsoft Employee

2024-07-25T19:00:54.1666667+00:00

Hi Nitin Patil,

Glad to know that your issue was resolved.

if you don't have any further questions, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to Fix the InternalServerError in Executing a ADF Pipeline?

0 additional answers

Your answer