Is it possible to access Databricks DBFS from Azure Data Factory?

Question

I would like to use the Copy Data Activity in Data Factory to move data from/to Databricks DBFS. I have my databricks workspace successfully linked to Data Factory as a linked service.

If I select Azure Delta Storage as a dataset source or sink, I am able to access the tables in the cluster (not the DBFS) and preview the data, but when validating there is an error because the tables are not delta tables.

Is there a way to directly connect to DBFS?

Answer 1

Hello @NewUser and welcome to Microsoft Q&A.

I think the easiest way to make data available to DBFS, is to first have DBFS mount a blob storage or data lake gen2 account. Then have Data Factory write the files to that storage account.

This has a few upsides over trying to write directly to the DBFS root. By confining the writes done by Data Factory to the mounted storage, the rest of DBFS is protected from accidental overwrites and helps keep your data secure by making it impossible for Data Factory to extract data outside this silo.