Azure data factory

Question

Dear Team,
I need to copy a data from a single table to multiple tables with mapping of columns in azure data factory by connecting with a PostgreSQL server. only one source table and I need to send that source data to the multiple sink tables by mapping a column and some mandatory columns of sink tables need to generate a unique id for those columns how can I do that in azure data factory please help me to achieve my requirement.

Answer 1

In ADF, you need to create a Data Flow where you add a Source transformation and configure it to read data from your PostgreSQL table (using the appropriate linked service)

If you need to generate unique IDs or derive new columns, add a Derived Column transformation where you use an expression to create unique IDs (for example, using a combination of current date and time or a GUID function if available)

Then, add multiple Sink transformations to your Data Flow, one for each target table, and configure each one to write data for a different PostgreSQL table.

For each Sink transformation, map the columns from the source to the appropriate columns in the target table.

Then, specify the insert method in each sink transformation. You can use "Insert" to add new rows to the sink tables.

Configure your sink tables appropriately if they require specific settings or preconditions (for example, preexisting primary key values).

Once the Data Flow is configured, add it to a Pipeline and publish.

You can create a trigger to execute the Pipeline based on your schedule or requirements.

Answer 2

Hi Sai Varma Eswararaju ,

Welcome to Microsoft Q&A platform and thanks for posting your query here.

As per my understanding you want to copy data from a single source to multiple sink tables in postgreSQL using ADF pipeline. Please let me know if that is not the correct understanding.

In addition to Amira's response of using Dataflow, you could try using copy activity within foreach loop on top of a lookup activity output (having metadata of sink tablenames). you could use dynamic mapping in the mapping section of copy activity.

Here is the detailed video about the same: Dynamic Column mapping in Copy Activity in Azure Data Factory . Kindly take it as reference and see if it helps.

Thankyou. Kindly accept the answer by clicking on Accept answer button. Thankyou