Share via

Want to append output of two web activities

Amar Agnihotri 926 Reputation points
2022-06-27T06:08:55.263+00:00

Hi,
I have created the below pipeline to get user activity data calling REST api -
215222-image.png

These are the variables i defined
215202-image.png

Now these are the output of the activities in pipeline .

The first web activity is generating an Access Bearer token as
215188-image.png

Now the second web activity "Get User Activity Data" is utilising that bearer token to call rest api and fetching user activity data and continuation token. You can see here that it fetched one record and continuation token -
215127-image.png

Now i am storing output of the above web activity in two variables . The set variable activity "Set User Data"" is using an array type variable to store the record part of the output as
215128-image.png

and the set variable activity "Set Continuation Token" is using a string variable to store the continuation token of the output as -

215205-image.png

Now i am passing that continuation token to until activity and inside until activity i am using another web activity to again call the rest api using the continuation token to get the next records of user activity. The until activity will run until we get the null continuation token and it will keep iterating through the loop
215170-image.png

Condition for until activity -
215150-image.png

Inside until activity
215139-image.png

Now inside until activity you can see the web activity is being used to call the rest api again to get the user activity data and again i am storing the record part of output to the array variable using set variable activity and the continuation token to the string variable.

Now see the output of until activity . We can see that at some point of time the continuation token is getting null and this is the time when the until activity will break execution -

215190-image.png
and this is the final records that are being stored to an array variable inside until activity
215223-image.png

This is output of set variable activity "Set output data"
215129-image.png
In the above output you can see only the records are being stored which is the desired output .

Now the issues that i am facing -

1)- to store the record part of the output of the web activity i am using set variable activity which is using array variable inside . The first set variable activity is being used outside the until loop which stores the record coming as an output of first web activity -
215224-image.png

and the second set variable activity inside until loop which is storing the records of web activity inside until loop -
215232-image.png

Now since i am using set variable activity to store the record in array variable so inside until loop in each iteration the array variable is holding the new records from the latest iteration. For example the until loop iterates for 50 times then inside until the array variable holds the records of the 50th iteration and all the previous records are not stored.
so two things are needed to change this.

    • rather using set variable activity inside until loop i need to use some other activity which can append the records of each iteration to the same array variable defined out side the until loop and that's how at the last run of until loop we would have all the records stored in a single array variable which has already been defined outside the until loop. This one -
      215215-image.png

I was thinking of using the append activity but since append activity needs to do some indexing and for that we need to know total number of pages in the output of web activity and in my case web activity output in not having any such property as shown -

215281-image.png

so i am not able to understand how to achieve this . Also, if we achieve this and get succeed to append all the records from until activity to the very first array variable defined outside the until activity I.e we will get a single array variable with all the records . After this, The challenge is how to copy those records to csv file in datalake storage.

I have done 50% of the work but i am stuck at these two points now. Can anyone suggest the workflow and the steps .

Thanks in advance .

@ShaikMaheer-MSFT , @AnnuKumari-MSFT , @Subashri Vasudevan

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.


Answer accepted by question author

Subashri Vasudevan 11,306 Reputation points Volunteer Moderator
2022-06-27T16:21:31.237+00:00

Hi @Amar Agnihotri ,

I think the error is because, you are trying to add an array (activityeventEntitites) to array variable.

May be we can try to join the array values and append as string.

join(variables('ARRAY_VARIABLE'), ',')

Try this and let us know for any issue.

Thanks.

Was this answer helpful?

1 person found this answer helpful.

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.