Find best activity for implemet ML scenario in Azure Data Factory

Question

I'm new to Azure Data Factory. I want to implement the below scenario and I want to know which activity is suitable for this scenario.

1. My data is on the Postgres database.
2. Our data factory an activity copies data from the Postgres database.
3. In this step run preprocessing function then is a python file. Also, the Python function needs the "scaler.sav" file.
4. The output of step 3 use as input for step 4 and run the classification function then is a python file. Also, the Python function needs the "ExtraTreesClassifier.sav" file.
5. Insert the output of step 4 into the database.

The diagram of this scenario. I need help to know which activity should use for this scenario.

Answer 1

Hello @Mohsen Akhavan ,
Thanks for the ask and using the Microsoft Q&A platform .

As I understand at this time you have couple of options .

Use Azure databricks activity : Since you are using Python , you can use ADB activity which will call a notebook and you can run you python script . This should work .

Use custom activity : You can use a custom activity and run the azure batch in the back ground . Please read about the same here .

Please do let me know how it goes .
Thanks
Himanshu
Please do consider clicking on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members