Synapse Pipeline from Databricks Delta Lake to Dedicated Pool

Question

We are in the process of implementing a new solution involving a Synapse pipeline to transfer data from an Azure Databricks Delta Lake to a Synapse dedicated pool. I’m utilizing a copy activity with a dynamically constructed query against Databricks as the source and an existing table in the dedicated pool as the sink. The copy method is set to Polybase with staging enabled in the settings. Upon running the pipeline, I encounter the following error: “Operation failed: ‘The specified filesystem does not exist.’ 404, HEAD.” However, when I copy and paste the exact same query in the expression field during Preview data, it executes successfully and produces the expected output. We utilize the unity catalog in Databricks, and there’s concern that it might not be fully implemented in Synapse. However, the puzzling aspect is why the preview works while the debug does not. Any assistance on this matter would be greatly appreciated.

Answer 1

Hi @Brianna C ,

Welcome to Microsoft Q&A forum and thanks for reaching out here.

Based on the error message you provided, it seems that the specified filesystem does not exist. This error can occur if the path to the file or directory is incorrect or if the file or directory does not exist.

To troubleshoot this issue, you can try the below steps:

1. The error occurs when trying to copy data from an Azure Databricks Delta Lake table to an SQL table directly.
2. To fix this, you need to enable staging in the Copy activity and link it to a specific ADLS Gen 2 account or blob storage account.
3. Ensure that you provide the correct path, as the error is caused when the path you provide is not available.

To resolve this:

Go to Copy activities Settings >> Enable staging >> Staging settings >> provide Staging account linked service >> provide correct path.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.