Using Microsoft Spark Utilities inside Azure Data Factory

Question

Hi,

I'd like to know how using the Microsoft Spark Utilities inside Azure Data Factory.

In particular, I want to use the file system utility name "mssparkutils.fs.mv".

Any helps to me, please? Thanks

Answer 1

Hi Pasquale Massimo Scorca,

Thank you for posting query in Microsoft Q&A Platform.

Microsoft Spark Utilities are available as part of Spark runtime in Synapse Analytics. This runtime is not directly available in ADF. You need to consider running Synapse notebook from pipeline using Synapse notebook activity. There by, ADF runs that notebook on Synapse runtime and hence you will be able to use Microsoft Spark Utilities from that notebook.

You need to consider using Synapse Analytics notebook activity in ADF pipeline. Please check below link to know more.

https://techcommunity.microsoft.com/t5/azure-data-factory-blog/orchestrate-and-operationalize-synapse-notebooks-and-spark-job/ba-p/3724379

Hope this helps. Please let me know how it goes and if any further queries.

---

Please consider hitting Accept Answer button. Accepted answers help community as well.