Find best activity for implemet ML scenario in Azure Data Factory

Mohsen Akhavan 936 Reputation points
2021-07-09T16:09:29.29+00:00

I'm new to Azure Data Factory. I want to implement the below scenario and I want to know which activity is suitable for this scenario.

  1. My data is on the Postgres database.
  2. Our data factory an activity copies data from the Postgres database.
  3. In this step run preprocessing function then is a python file. Also, the Python function needs the "scaler.sav" file.
  4. The output of step 3 use as input for step 4 and run the classification function then is a python file. Also, the Python function needs the "ExtraTreesClassifier.sav" file.
  5. Insert the output of step 4 into the database.

The diagram of this scenario. I need help to know which activity should use for this scenario.

113369-image.png

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,537 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,443 questions
Azure Database for PostgreSQL
{count} votes

1 answer

Sort by: Most helpful
  1. HimanshuSinha-msft 19,376 Reputation points Microsoft Employee
    2021-07-13T00:25:11.19+00:00

    Hello @Mohsen Akhavan ,
    Thanks for the ask and using the Microsoft Q&A platform .

    As I understand at this time you have couple of options .

    Use Azure databricks activity : Since you are using Python , you can use ADB activity which will call a notebook and you can run you python script . This should work .

    Use custom activity : You can use a custom activity and run the azure batch in the back ground . Please read about the same here .

    Please do let me know how it goes .
    Thanks
    Himanshu
    Please do consider clicking on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

    0 comments No comments