date in batches

Question

Hello,
At present I have created a data factory pipeline, pl_load which takes a parameter as pDaysBack.
The pipeline takes this parameter and builds a string as follows to build a string to call an api:

@markus.bohland@hotmail.de (
dataset().dateFieldName, ' > ', addDays(utcNow(), int(dataset().pDaysBack)),
' and ', dataset().dateFieldName, ' < ', utcNow()
)

i.e. if you pass in, for example, -365 for the pDaysBack, then it builds the string something like:
assuming todays date is 26/5/2022

companyDate > 26/5/2021 and companyDate < 26/5/2022

All the above works fine at present.

Question:
Now, I would like to tell the pipeline to do the above but in batches of say 30 days --> pBatches=30 instead of doing it all in one go.
perhaps there should be a loop with some parameters in which I can put my existing pipeline (pl_load).

perhaps:
parameter --> pdaysBack = -365
parameter --> pBatches = 30
variable vStartDate --> addDays(utcNow(), int(dataset().pdaysBack)) --> which will give : 26/5/2021
so in the pipeline for my above expression in a loop:

@markus.bohland@hotmail.de (
dataset().dateFieldName, ' > ', addDays(utcNow(), int(dataset().pdaysBack)),
' and ', dataset().dateFieldName, ' < ', addDays(utcNow(), int(dataset().pdaysBack)) + pBatche1, then batch2... until it reaches now
)

How can I do this please?
Thank you

Answer 1

Hi @arkiboys , how have you been? Let me see if I understand this one right, I think it looks fun.

You already have a string-building expression, and want to use that logic inside a loop so it repeats with smaller and smaller input value until to reaches now. Kind of like

Language C:  
  
integer inputValue = -365;  
  
for( inputValue ; inputValue = 0 ; inputValue = inputValue + 30 )  
    buildMyString( inputValue );  
  
Language Python:  
  
inputValue = -365  
  
while inputValue < 0 :  
    buildMyString( inputValue )  
    inputValue = inputValue + 30  

For these situations, I have a couple tricks.

1. Divide pdaysBack by pBatches to find out how many batches we need (pNumBatches).
2. use range( 1 , pNumBatches ) to generate a series ( 1, 2, 3, 4, 5 ... pNumBatches).
3. Feed this series into a ForEach loop. Inside the loop an AppendVariable activity will multiply each item by pBatches, and append it to a new list variable. This may re-order things unless you do sequential.
4. Back in the main pipeline, I feed this modified series to another ForEach loop which gives each item to your stringBuilder. The stringBuilder can use another AppendVariable
5. Once that loop is finished, we have an array of strings, just need to join() them together in Set Variable and done!

Answer 2

@arkiboys

I combined a couple steps. This one figures out how many days back, and sends that to execute pipeline.
207292-batchlogic.txt