Hi,
I have created the below pipeline to get user activity data calling REST api -
These are the variables i defined
Now these are the output of the activities in pipeline .
The first web activity is generating an Access Bearer token as
Now the second web activity "Get User Activity Data" is utilising that bearer token to call rest api and fetching user activity data and continuation token. You can see here that it fetched one record and continuation token -
Now i am storing output of the above web activity in two variables . The set variable activity "Set User Data"" is using an array type variable to store the record part of the output as
and the set variable activity "Set Continuation Token" is using a string variable to store the continuation token of the output as -
Now i am passing that continuation token to until activity and inside until activity i am using another web activity to again call the rest api using the continuation token to get the next records of user activity. The until activity will run until we get the null continuation token and it will keep iterating through the loop
Condition for until activity -
Inside until activity
Now inside until activity you can see the web activity is being used to call the rest api again to get the user activity data and again i am storing the record part of output to the array variable using set variable activity and the continuation token to the string variable.
Now see the output of until activity . We can see that at some point of time the continuation token is getting null and this is the time when the until activity will break execution -
and this is the final records that are being stored to an array variable inside until activity
This is output of set variable activity "Set output data"
In the above output you can see only the records are being stored which is the desired output .
Now the issues that i am facing -
1)- to store the record part of the output of the web activity i am using set variable activity which is using array variable inside . The first set variable activity is being used outside the until loop which stores the record coming as an output of first web activity -
and the second set variable activity inside until loop which is storing the records of web activity inside until loop -
Now since i am using set variable activity to store the record in array variable so inside until loop in each iteration the array variable is holding the new records from the latest iteration. For example the until loop iterates for 50 times then inside until the array variable holds the records of the 50th iteration and all the previous records are not stored.
so two things are needed to change this.
-
- rather using set variable activity inside until loop i need to use some other activity which can append the records of each iteration to the same array variable defined out side the until loop and that's how at the last run of until loop we would have all the records stored in a single array variable which has already been defined outside the until loop. This one -
I was thinking of using the append activity but since append activity needs to do some indexing and for that we need to know total number of pages in the output of web activity and in my case web activity output in not having any such property as shown -
so i am not able to understand how to achieve this . Also, if we achieve this and get succeed to append all the records from until activity to the very first array variable defined outside the until activity I.e we will get a single array variable with all the records . After this, The challenge is how to copy those records to csv file in datalake storage.
I have done 50% of the work but i am stuck at these two points now. Can anyone suggest the workflow and the steps .
Thanks in advance .
@ShaikMaheer-MSFT , @AnnuKumari-MSFT , @Subashri Vasudevan