Copy data from multiple Azure DB's to one Azure DB with Data Factory

Lee Taylor 31

I'm not sure that this is possible for this service or not.
I have many databases of the same schema that I want to copy data on a monthly data into one single database. I need to be able to use a Table that contains a list of these databases and iterate through the, one record at a time and pull the Id of the database (GUID) and the connection string of the database and then connect to that database and copy slices of data to put into the ONE database. Any ideas? Can this be done?

1 answer

John Aherne 516 Reputation points

2020-09-17T16:52:06.827+00:00

You can use a parameterized linked service to do that:
https://learn.microsoft.com/en-us/azure/data-factory/parameterize-linked-services
Please sign in to rate this answer.
Lee Taylor 31 Reputation points

2020-09-17T17:03:53.177+00:00

Well that is part of the problem. How do I iterate through a table and pull the necessary information to use in these parameters?

John Aherne 516 Reputation points

2020-09-17T18:04:42.013+00:00

Use a lookup activity to get your list of databases.
Use a forEach activity to iterate through that list.
Use a copy activity in the forEach.
In the source of the copy activity, use a paramaterized linked service to pull the data.

MartinJaffer-MSFT 26,091 Reputation points

2020-09-18T05:51:41.063+00:00

@Lee Taylor did JohnAherne-0132 second reply answer your question, or would you like more assistance/details?

MartinJaffer-MSFT 26,091 Reputation points

2020-09-22T20:18:20.293+00:00

@Lee Taylor If you found your own solution, could you please share it here to aid the community?
If you are still struggling, please let us know.
Thank you
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Copy data from multiple Azure DB's to one Azure DB with Data Factory

1 answer

Your answer