How to iterate "Copy data" activity that points to different REST API endpoints

King Java 480 Reputation points
2024-02-08T21:55:25.93+00:00

Hi Folks,

I am trying to find a better solution to not clone 50+ pipelines. Basically, I have about 50+ REST API endpoints: Example:

https://api.example.com/Provider-Info?providerUid=1

https://api.example.com/Provider-Info?providerUid=2

....

https://api.example.com/Provider-Info?providerUid=51

I have created a simple pipeline to test using "Copy data", and it is working: User's image

I have source as REST (linked service): User's image

I am trying to modify this pipeline so that it dynamically alternates providerUid (from static like I have now).

So, data gets pulled from REST API --> Azure SQL Server table.

I am trying to alternate providerUid each time and iterate so that data gets inserted into Azure SQL for 50+ times using one pipeline.

What is best architecture to make this work?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,070 questions
{count} votes

Accepted answer
  1. phemanth 3,455 Reputation points Microsoft Vendor
    2024-02-17T04:18:16.3966667+00:00

    @King Java
    Welcome to Microsoft Q&A platform and thanks for posting your question. I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

    • __Ask: __I am trying to find a better solution to not clone 50+ pipelines. Basically, I have about 50+ REST API endpoints: Example:

    https://api.example.com/Provider-Info?providerUid=1 https://api.example.com/Provider-Info?providerUid=2 .... https://api.example.com/Provider-Info?providerUid=51 I have created a simple pipeline to test using "Copy data", and it is working: User's image I have source as REST (linked service): User's image I am trying to modify this pipeline so that it dynamically alternates providerUid (from static like I have now). So, data gets pulled from REST API --> Azure SQL Server table. I am trying to alternate providerUid each time and iterate so that data gets inserted into Azure SQL for 50+ times using one pipeline. What is best architecture to make this work?

    Solution:

    Here are screenshots: User's image User's image Inside ForaEach --> CopyData: User's image User's image User's image Results: User's image WafaStudios' video gave me a key (@item()): https://www.youtube.com/watch?v=KuWYuHlUwD0

    If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information. If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

    ---Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.


1 additional answer

Sort by: Most helpful
  1. King Java 480 Reputation points
    2024-02-17T00:50:18.3266667+00:00

    I was able to figure out using ForEach and Parameter after many trials :) .

    Here are screenshots: User's image

    User's image

    Inside ForaEach --> CopyData: User's image

    User's image

    User's image

    Results: User's image

    WafaStudios' video gave me a key (@item()):

    https://www.youtube.com/watch?v=KuWYuHlUwD0