Using Microsoft Spark Utilities inside Azure Data Factory

Pasquale Massimo Scorca 1 Reputation point


I'd like to know how using the Microsoft Spark Utilities inside Azure Data Factory.

In particular, I want to use the file system utility name "".

Any helps to me, please? Thanks

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,177 questions
{count} votes

1 answer

Sort by: Most helpful
  1. ShaikMaheer-MSFT 31,806 Reputation points Microsoft Employee

    Hi Pasquale Massimo Scorca,

    Thank you for posting query in Microsoft Q&A Platform.

    Microsoft Spark Utilities are available as part of Spark runtime in Synapse Analytics. This runtime is not directly available in ADF. You need to consider running Synapse notebook from pipeline using Synapse notebook activity. There by, ADF runs that notebook on Synapse runtime and hence you will be able to use Microsoft Spark Utilities from that notebook.

    You need to consider using Synapse Analytics notebook activity in ADF pipeline. Please check below link to know more.

    Hope this helps. Please let me know how it goes and if any further queries.

    Please consider hitting Accept Answer button. Accepted answers help community as well.