Data Flow failing: Unknown Error while using data flow debug in Azure data factory

Question

Hello My azure data factory pipeline has started failing. it was working fine earlier, but it started failing form today with following Error:

Spark job failed: {
\"shortMessage\":\"DF-SYS-0 DataFlowManagerClient.getPayload fail to getPayload request for run:, exception:java.io.IOException: Retry Request, fail to reach https://dmcusaksigaks.svc.datafactory.azure.com/sparkjob/6a330cb9-ee7c-434e-8569-1e8fc83a35db/getJobPayload with with code: 500, exception:\",\"detailedMessage\":\"Failure 2022-09-29 00:46:49.601 failed DebugManager.processJob, run=6a330cb9-ee7c-434e-8569-1e8fc83a35db, errorMessage=DF-SYS-0 DataFlowManagerClient.getPayload fail to getPayload request for run:6a330cb9-ee7c-434e-8569-1e8fc83a35db, exception:java.io.IOException: Retry Request, fail to reach https://dmcusaksigaks.svc.datafactory.azure.com/sparkjob/6a330cb9-ee7c-434e-8569-1e8fc83a35db/getJobPayload with with code: 500, exception:\"}}\n"
}

I have not made any changes in any git repo for the factories also there is no change in azure subscription for this pipeline or the linked services

Answer 1

Hi @Kirti Chaturvedi ,

Thankyou for using Microsoft Q&A platform and thanks for posting your question here.

As you stated, the error message is not enough to derive the root cause of the issue. This could be a transient issue , could you please retry running the dataflow after sometime. Also, Could you please test connection for all the underlying linked services used in the dataflow.