batch output of a lookup and then pass to a web action through for each loop

Hamza Outa 61 Reputation points
2024-03-27T13:30:16.76+00:00

I do a lookup on a table to receive some ID's. Then I take these ID's and put them in the body of a http POST request. Right now I have it like this:

User's image

The problem is that each row would be it's own request, and I don't want to send +100k requests at the same time.

The API allowes for multiple id's in one body. so now I wanted to batch it so that each request has 5k id's. Another thing is that I can't store any data (data only exists during runtime).

So is it possible to take the output of a lookup and batch it into X-amount and iterate over it passing each batch to a web action? (without storing any data e.g. blob)

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,489 questions
{count} votes

Accepted answer
  1. Smaran Thoomu 9,045 Reputation points Microsoft Vendor
    2024-04-01T11:45:25.26+00:00

    Hi @Hamza Outa

    I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

    Issue: I do a lookup on a table to receive some ID's. Then I take these ID's and put them in the body of a http POST request. Right now, I have it like this:

    The problem is that each row would be it's own request, and I don't want to send +100k requests at the same time.

    The API allows for multiple id's in one body. so now I wanted to batch it so that each request has 5k id's. Another thing is that I can't store any data (data only exists during runtime).

    So is it possible to take the output of a lookup and batch it into X-amount and iterate over it passing each batch to a web action? (without storing any data e.g. blob).

    Solution: I tried using item().position but that seems to return nothing. so instead of trying to batch in data factory, i changed the lookup to a query that concats the rows together over a window.

    SQLCopy

    WITH
    

    and then in the web task I do this:

    @concat('{"data":{"filter":{"field":"","value":[',item().ids, ']},"flows":["aaaa","bbbbbb"]
    
    
    

    If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

    I hope this helps!

    If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.


    Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

    0 comments No comments

0 additional answers

Sort by: Most helpful