Usage of Python & Spark in Azure Data Factory

Mahesh Kumar SSM 66 Reputation points
2021-03-21T12:49:39.383+00:00

I am new to Azure Data Factory. Please help to clarify the following...

  1. Is learning Python & Spark of any help in ADF ?
  2. Are there specific applications/tasks that can only be handled by Spark ?

Thanks.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,341 questions
0 comments No comments
{count} votes

Accepted answer
  1. Vaibhav Chaudhari 38,471 Reputation points
    2021-03-22T05:31:26.18+00:00
    1. Not much. You can write python code or any code to run in spark cluster in Azure databricks then just call this code via notebook from Data factory. ADF is mostly be used to copy the data from various sources, do transformation using Data flow (UI)
    2. Machine learning, streaming data or any data analytics work can be done in spark effectively.

    You can try to explore Azure databricks if your focus is more on spark and writing code in python, R, scala or spark sql.


    Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav


0 additional answers

Sort by: Most helpful