Mapping data flow error: PathNotFound The specified path does not exist

Question

Mapping data flow error: PathNotFound The specified path does not exist

William Connell 0

Just recently the scheduled pipelines running in our Azure Synapse Analytics workspace started to fail with the error message.

"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: at Sink 'sinkXXXX': Operation failed: "The specified path does not exist."

It fails on the mapping data flow sink. But if I rerun the pipeline then it might fail on the same sink or a different sink.

The path in question is not the folder in the data lake that the sink is actually writing to, but an _temporary folder inside that folder. For example

404, GET, https://<storageaccount>.dfs.core.windows.net/<container>?upn=false&resource=filesystem&maxResults=5000&directory=Bronze/Something/Something/Something/_temporary/0/_temporary/attempt_202307060632398990770822798711859_0958_m_000000_958&timeout=90&recursive=false

From my understanding in Spark underneath the mapping data flow when writing a dataframe to parquet it creates the _temporary folder, writes the files to the _temporary folder, and then moves them to the main folder once the job is complete, and then deletes the _temporary folder.

But the sink is failing because it cannot find the _temporary folder that it has just created.

Everything was working okay and the pipelines were running okay, but on the 26th of June 2023 multiple pipelines started failing with this error.

It is like the storage account cannot keep up with the Azure Synapse Analytics.

Any help or suggestions would be appreciated.

Stephen Connell 21 Reputation points

2023-07-06T14:27:06.2966667+00:00

Looking at the metrics we do see some latency issues:

This occurs at the same time as these errors started to occur.
As an example on the 4th July we see almost 2 seconds E2E latency:
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-07-06T22:35:14.35+00:00

@William Connell Welcome to the MS Q&A platform.

Yes, your understanding is correct. In Spark, underneath the mapping data flow, temporary files are created in a directory named _temporary/0/

Please confirm if this is a temporary issue. Or if this is still happening?

If this still happens, I suggest implementing retry logic to resolve the issue until the errors are completely resolved.
William Connell 0 Reputation points

2023-07-07T10:39:36.0666667+00:00

Hi @Bhargava-MSFT it was an issue that was happening consistently with all pipelines and mapping data flows from the 26th of June 2023 till the 6th of July 2023.

I implemented Custom Sink Ordering in the mapping data flows so that the sinks are not running in parallel within the mapping data flows, and the sinks are now running sequentially.

https://learn.microsoft.com/en-us/azure/data-factory/data-flow-sink#custom-sink-ordering

That seems to have reduced the number of simultaneous requests to the data lake from the Spark Cluster, and the storage account is able to cope.

So the pipelines and mapping data flows are running now.

But I can still see higher that expected latency on the storage account.
William Connell 0 Reputation points

2023-07-07T11:06:17.2533333+00:00

I can see a lot of Blob GetPathStatus errors on the storage account over the last 7 days.
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-07-07T18:51:36.1633333+00:00

@William Connell

Thanks for the additional details. The issue is more of a backend synapse spark cluster (could be due to increased traffic on the spark cluster)

I suggest opening a support case for a deeper investigation. If you don't have a support plan, please let me know. I can provide you one-time free support request.

I am looking forward to hearing from you.