batch output of a lookup and then pass to a web action through for each loop

Question

I do a lookup on a table to receive some ID's. Then I take these ID's and put them in the body of a http POST request. Right now I have it like this:

The problem is that each row would be it's own request, and I don't want to send +100k requests at the same time.

The API allowes for multiple id's in one body. so now I wanted to batch it so that each request has 5k id's. Another thing is that I can't store any data (data only exists during runtime).

So is it possible to take the output of a lookup and batch it into X-amount and iterate over it passing each batch to a web action? (without storing any data e.g. blob)

Answer 1

Hi @Hamza Outa

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Issue: I do a lookup on a table to receive some ID's. Then I take these ID's and put them in the body of a http POST request. Right now, I have it like this:

The problem is that each row would be it's own request, and I don't want to send +100k requests at the same time.

The API allows for multiple id's in one body. so now I wanted to batch it so that each request has 5k id's. Another thing is that I can't store any data (data only exists during runtime).

So is it possible to take the output of a lookup and batch it into X-amount and iterate over it passing each batch to a web action? (without storing any data e.g. blob).

Solution: I tried using item().position but that seems to return nothing. so instead of trying to batch in data factory, i changed the lookup to a query that concats the rows together over a window.

SQLCopy

WITH

and then in the web task I do this:

@concat('{"data":{"filter":{"field":"","value":[',item().ids, ']},"flows":["aaaa","bbbbbb"]

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

I hope this helps!

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.