Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,536 questions
DATA FACTORY - LOOP EXECUTION
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 16%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EIR%3C/text%3E%3C/svg%3E)
Igor Rezende
0
Reputation points
Hi everybody, how are you? I achivied a pipeline in Azure Data Factory doing a loop in a database and reading its tables automatically. However, i didnt find a way to do a single pipeline that do a loop in different databases and it reads their different tables. I have built a pipeline with a loop for each database. But i would like a single pipeline. Does anyone know can i do this? Or the loop is only possible for each database? Thanks!
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 38%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EQ%3C/text%3E%3C/svg%3E)
QuantumCache 20,261 Reputation points2024-02-27T00:12:49.4466667+00:00 Hello @Igor Rezende Welcome to QnA forum! I hope you are doing great!
However, i didnt find a way to do a single pipeline that do a loop in different databases and it reads their different tables
Did you try the Dataflow to query multiple SQL Tables as sources? Please share more info on your scenario. How many SQL Servers? How many Tables? What is the criteria to loop multiple tables, is there any lookup which will provide the list of tables? Screenshot of your current Pipeline activities, to see if we can help fix your pipeline?
-
Igor Rezende 0 Reputation points
2024-02-27T13:17:16.0433333+00:00 Hello @SatishBoddu-MSFT, I need to perform a "copy data" operation across different tables stored in various databases (Oracle, PostgreSQL, SQL Server, etc.), essentially different linked services. I aim to store the data in distinct directories within the data lake. I've successfully implemented a loop to read all tables within a database. However, I found myself having to create multiple pipelines for each database. My goal is to automate all of this within a single pipeline. Currently, I'm using "get metadata" to retrieve the names of the tables I require, "for each" to iterate through them, and "look up" to manage ingestion over time. Moreover, it was crucial to devise a method to automatically populate the parameters for each distinct linked service. Could you help me with this?
-
QuantumCache 20,261 Reputation points
2024-02-29T06:17:08.5833333+00:00 Igor Rezende Thanks for providing additional context.
Sign in to comment
Sign in to answer