I need help about design data pipline or data flow in Azure Data Factory

Question

I'm new to Azure Data Factory and I want to create an Azure Data Factory for the below scenario.
I have a Postgres database and add data every minute.

1. I want a data flow loaded data from Postgres with query ( I don't need all of the rows I need some rows).
2. Then, run some manipulation (function) on the output of step1.
3. The output of step 2 should be sent to a database or a topic of service bus.

I need some help with which activity and solution I should use.

Answer 1

Hi @Mohsen Akhavan ,

Thank you for posting query in Microsoft Q&A Platform. Below is the details.

1. I want a data flow loaded data from Postgres with query ( I don't need all of the rows I need some rows).

Copy Activity helps to perform data movement from source to sink without any transformation or manipulation. You have to use Query option so that you can take out only selected rows. But you mentioned selected rows of your source data should first undergo transformations. Hence You should go with Data Flows to implement scenario.
Copy Activity - https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-overview

2. Then, run some manipulation (function) on the output of step1.

Data flow source transformation helps to run query on your source to get selected data

Source Transformation - https://learn.microsoft.com/en-us/azure/data-factory/data-flow-source

3. The output of step 2 should be sent to a database or a topic of service bus.

You can use Sink transformation in dataflow to load transformed data in to different storages. Please check below link for supported storages.
Sink Transformation - https://learn.microsoft.com/en-us/azure/data-factory/data-flow-sink

Service bus cannot be used as Sink in Dataflows, Hence workaround would first load data in supported storages using dataflows and from there load it to Service bus if require.

Hope this information is helpful. Thank you.

---

• Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer.
  https://learn.microsoft.com/en-us/answers/articles/25904/accepted-answers.html
• Want a reminder to come back and check responses? Here is how to subscribe to a notification. 
  https://learn.microsoft.com/en-us/answers/articles/67444/email-notifications.html