Synpase Mapping Dataflows error - "errorCode": "3259", "message": "Livy job state dead caused by unknown error

Question

Hello,

We started noticing this error in our production earlier this week and noticed the same later in our test environment as well. It happens for random pipelines that uses mapping dataflows. The pipeline succeeds after the rerun attempts but would like to know why this started happening recently.

• Error that shows on the pipeline. "errorCode": "3259", "message": "Livy job state dead caused by unknown error
• Error from the mapping dataflow output log Livy Id=[0] Job failed during run time with state=[dead].", "ErrorCode": "LIVY_JOB_STATE_DEAD" } ], "LivyInfo": { "JobCreationRequest": null }, "State": "dead", "Log": [ "\tat org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:935)", "\tat org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:944)", "\tat org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)", "Caused by: org.apache.hadoop.fs.azure.AzureException: No credentials found for account adfdataflowncusgen2.blob.core.windows.net in the configuration, and its container synapsedataflowjars is not accessible using anonymous credentials. Please check if the container exists first. If it is not publicly available, you have to provide account credentials.", "\tat org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.connectUsingAnonymousCredentials(AzureNativeFileSystemStore.java:867)", "\tat org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1085)", "\t... 28 more", "\nstderr: ", "\nYARN Diagnostics: ", "No YARN application is found with tag livy-batch-0-tvsxugf1 in 300 seconds. This may be because 1) spark-submit fail to submit application to YARN; or 2) YARN cluster doesn't have enough resources to start the application in time. Please check Livy log and YARN log to know the details."