I need help about design data pipline or data flow in Azure Data Factory

Mohsen Akhavan 936 Reputation points
2021-06-21T09:12:04.727+00:00

I'm new to Azure Data Factory and I want to create an Azure Data Factory for the below scenario.
I have a Postgres database and add data every minute.

  1. I want a data flow loaded data from Postgres with query ( I don't need all of the rows I need some rows).
  2. Then, run some manipulation (function) on the output of step1.
  3. The output of step 2 should be sent to a database or a topic of service bus.

I need some help with which activity and solution I should use.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,933 questions
{count} votes

Accepted answer
  1. ShaikMaheer-MSFT 38,311 Reputation points Microsoft Employee
    2021-06-22T07:04:13.757+00:00

    Hi @Mohsen Akhavan ,

    Thank you for posting query in Microsoft Q&A Platform. Below is the details.

    1. I want a data flow loaded data from Postgres with query ( I don't need all of the rows I need some rows).

    Copy Activity helps to perform data movement from source to sink without any transformation or manipulation. You have to use Query option so that you can take out only selected rows. But you mentioned selected rows of your source data should first undergo transformations. Hence You should go with Data Flows to implement scenario.
    Copy Activity - https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-overview

    1. Then, run some manipulation (function) on the output of step1.

    Data flow source transformation helps to run query on your source to get selected data

    107989-surcetrans.png

    Source Transformation - https://learn.microsoft.com/en-us/azure/data-factory/data-flow-source

    1. The output of step 2 should be sent to a database or a topic of service bus.

    You can use Sink transformation in dataflow to load transformed data in to different storages. Please check below link for supported storages.
    Sink Transformation - https://learn.microsoft.com/en-us/azure/data-factory/data-flow-sink

    Service bus cannot be used as Sink in Dataflows, Hence workaround would first load data in supported storages using dataflows and from there load it to Service bus if require.

    Hope this information is helpful. Thank you.


    1 person found this answer helpful.
    0 comments No comments

0 additional answers

Sort by: Most helpful