DATA FACTORY - LOOP EXECUTION

Question

DATA FACTORY - LOOP EXECUTION

Igor Rezende 0

Hi everybody, how are you? I achivied a pipeline in Azure Data Factory doing a loop in a database and reading its tables automatically. However, i didnt find a way to do a single pipeline that do a loop in different databases and it reads their different tables. I have built a pipeline with a loop for each database. But i would like a single pipeline. Does anyone know can i do this? Or the loop is only possible for each database? Thanks!

QuantumCache 20,371 Reputation points Moderator

2024-02-27T00:12:49.4466667+00:00

Hello @Igor Rezende Welcome to QnA forum! I hope you are doing great!

However, i didnt find a way to do a single pipeline that do a loop in different databases and it reads their different tables

Did you try the Dataflow to query multiple SQL Tables as sources? Please share more info on your scenario. How many SQL Servers? How many Tables? What is the criteria to loop multiple tables, is there any lookup which will provide the list of tables? Screenshot of your current Pipeline activities, to see if we can help fix your pipeline?
Igor Rezende 0 Reputation points

2024-02-27T13:17:16.0433333+00:00

Hello @SatishBoddu-MSFT, I need to perform a "copy data" operation across different tables stored in various databases (Oracle, PostgreSQL, SQL Server, etc.), essentially different linked services. I aim to store the data in distinct directories within the data lake. I've successfully implemented a loop to read all tables within a database. However, I found myself having to create multiple pipelines for each database. My goal is to automate all of this within a single pipeline. Currently, I'm using "get metadata" to retrieve the names of the tables I require, "for each" to iterate through them, and "look up" to manage ingestion over time. Moreover, it was crucial to devise a method to automatically populate the parameters for each distinct linked service. Could you help me with this?
QuantumCache 20,371 Reputation points Moderator

2024-02-29T06:17:08.5833333+00:00

Igor Rezende Thanks for providing additional context.

Your answer

QuantumCache 20,371 Reputation points Moderator

2024-02-27T00:12:49.4466667+00:00

Hello @Igor Rezende Welcome to QnA forum! I hope you are doing great!

However, i didnt find a way to do a single pipeline that do a loop in different databases and it reads their different tables

Did you try the Dataflow to query multiple SQL Tables as sources? Please share more info on your scenario. How many SQL Servers? How many Tables? What is the criteria to loop multiple tables, is there any lookup which will provide the list of tables? Screenshot of your current Pipeline activities, to see if we can help fix your pipeline?
Igor Rezende 0 Reputation points

2024-02-27T13:17:16.0433333+00:00

Hello @SatishBoddu-MSFT, I need to perform a "copy data" operation across different tables stored in various databases (Oracle, PostgreSQL, SQL Server, etc.), essentially different linked services. I aim to store the data in distinct directories within the data lake. I've successfully implemented a loop to read all tables within a database. However, I found myself having to create multiple pipelines for each database. My goal is to automate all of this within a single pipeline. Currently, I'm using "get metadata" to retrieve the names of the tables I require, "for each" to iterate through them, and "look up" to manage ingestion over time. Moreover, it was crucial to devise a method to automatically populate the parameters for each distinct linked service. Could you help me with this?
QuantumCache 20,371 Reputation points Moderator

2024-02-29T06:17:08.5833333+00:00

Igor Rezende Thanks for providing additional context.

Share via

DATA FACTORY - LOOP EXECUTION

Your answer