Data Factory pippeline data flow activity failed by saying unexpected error (5000)

Question

Hello Everyone,We are using Data Factory to move our SQL data to Snowflake Data Warehouse. Everything works fine but suddenly on 20th-August, 2024. Our pipeline Data Flow Activities started failing randomly.

sometimes it was working fine for some tables and other tables fail. And vice versa. It was totally random. We tried to figure out what was the issue but it said the following error:The error came for exact 1 day for 10 hours continously and then it got fixed automatically.

Error encountered during attempt 1 to start job actor. Exception: [System.TimeoutException: This can happen if message is dropped when service is busy or its long running operation and taking more time than configured Operation Timeout.

at Microsoft.ServiceFabric.Services.Communication.Client.ServicePartitionClient1.InvokeWithRetryAsync[TResult](Func2 func, CancellationToken cancellationToken, Type[] doNotRetryExceptionTypes)

at Microsoft.ServiceFabric.Services.Remoting.V2.Client.ServiceRemotingPartitionClient.InvokeAsync(IServiceRemotingRequestMessage remotingRequestMessage, String methodName, CancellationToken cancellationToken)

at Microsoft.ServiceFabric.Services.Remoting.Builder.ProxyBase.InvokeAsyncV2(Int32 interfaceId, Int32 methodId, String methodName, IServiceRemotingRequestMessageBody requestMsgBodyValue, CancellationToken cancellationToken)

at Microsoft.Marlin.JobService.QueueService.SfJobQueue.StartJobActorAsync(JobState state, String traceId, CancellationToken token) in C:__w\1\s\JobService\Microsoft.Marlin.JobService.QueueService\Queue\SfJobQueue.cs:line 4508]

Answer 1

Hi @Nialish Arshad - Verbund Pflegehilfe

Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!

Apologies for the inconvenience.

As I understand that you are facing an issue with your Data Flow Activities in Azure Data Factory. Based on the error message you provided, it looks like there was a timeout exception encountered during the attempt to start the job actor. This can happen if the message is dropped when the service is busy or if it's a long-running operation that's taking more time than the configured operation timeout.

There might be a chance of an intermittent issue causing the problem. Could you please follow the steps below:

Temporary Glitch: Clearing cookies and caches can sometimes resolve temporary glitches within the application that might be causing the issue.

Corrupted Data: In rare cases, corrupted data stored in the browser's cache related to ADF Pipeline might be causing the issue. Clearing the cache removes this potentially problematic data.

We suggest trying out these additional troubleshooting steps to see if they can resolve your issue.

• Check if there are any network or connectivity issues between your SQL database and Snowflake Data Warehouse. You can also check if there are any issues with the Azure Data Factory service itself by checking the Azure status page.
• Check if there are any issues with the Azure integration runtime (IR) that you are using. You can try restarting the IR or creating a new one to see if that resolves the issue.

Additionally, you may check the Azure Status page at https://status.azure.com/en-us/status for any current service incidents or outages affecting the regions you are utilizing.

I hope this information helps, please do let us know if you have any further queries.