I reviewed various answers regarding "Internal Server Error - Executing requests." I looked into SHIR and the VM that hosts it, Azure SQL Database, and ADF resource health. I did not notice anything unusual.
Also, my use case scenario differs from the one described in the forum.
My use case scenario -
We have a SHIR and a VM on Azure.
We have an on-prem Oracle and SQL Server.
We use the Azure SQL Database and Azure Data Factory.
The flow of a pipeline is -
We use a For Each Loop
Three activities in a For Each Loop
Copy Data - Copies the data from on-prem SQL database tables to Azure Data Lake Gen2 and saves it as a parquet file.
A Data Flow - Reads the parquet file and loads the data to the Azure SQL database
Execute SP task - It marks the completion progress. A very lightweight task.
We bring in 3 tables. Master, it's child and it's child. The third table has @50 M rows when two others have @10 -20 M each.
The first two tables were complete, and the third task ( for a table with @50 M rows) failed for the first time, giving the error "Internal Server Error Executing Request."
The dataflow task failed during the production deployment, which is a concern.
I checked the VM for SHIR, resource health of the ADF, Azure SQL Database, and ADLS2. I did not find any obvious.
Where should I look to get more information? What should I do to overcome this error?