Using Microsoft Spark Utilities inside Azure Data Factory

pmscorca 792 Reputation points
2023-05-22T10:33:02.3766667+00:00

Hi,

I'd like to know how using the Microsoft Spark Utilities inside Azure Data Factory.

In particular, I want to use the file system utility name "mssparkutils.fs.mv".

Any helps to me, please? Thanks

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,533 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee
    2023-05-23T09:58:42.5933333+00:00

    Hi Pasquale Massimo Scorca,

    Thank you for posting query in Microsoft Q&A Platform.

    Microsoft Spark Utilities are available as part of Spark runtime in Synapse Analytics. This runtime is not directly available in ADF. You need to consider running Synapse notebook from pipeline using Synapse notebook activity. There by, ADF runs that notebook on Synapse runtime and hence you will be able to use Microsoft Spark Utilities from that notebook.

    You need to consider using Synapse Analytics notebook activity in ADF pipeline. Please check below link to know more.

    https://techcommunity.microsoft.com/t5/azure-data-factory-blog/orchestrate-and-operationalize-synapse-notebooks-and-spark-job/ba-p/3724379

    Hope this helps. Please let me know how it goes and if any further queries.

    Please consider hitting Accept Answer button. Accepted answers help community as well.