How to copy data from web URL output ?

Question

How to copy data from web URL output ?

Amar Agnihotri 926

Hi,
I am using REST api to get user activity data and on calling that api in web activity i am getting data with continuation token . This is the format of the out put -

Along with this i am getting continuation token --

In the current case it returned two records in the list and after two records it returned continuation token . I want to copy those two records as .csv files in datalake . I need to copy that highlighted records only

Can anyone suggest how can we copy that part of data coming as an output of web activity?

MartinJaffer-MSFT 26,236 Reputation points

2022-06-22T18:04:08.523+00:00
Hello and welcome to Microsoft Q&A @Amar Agnihotri .

Your ask seems to be either "How do I write the output of Web activity to a file" or "How do I only write some of the data?".

Generally, if you want to copy data, you should use the Copy Data activity. You might want to look into the REST connector and HTTP connector. The REST connector has pagination abilities to leverage the continuationToken. Actually, you would want to leverage the "continuationUri" in the "absoluteUrl" pagination rule, is easier.

The reason I am not suggesting Web activity, is because Web activity works much differently, it makes available to the pipeline the direct response. As such, you would need 2-3 Web activities to move data, one to read, one to write. Also, it does not to format change (i.e. json -> csv) and does not make guarantees to the fidelity of non-text data (compressed or binary, etc.).

For the writing Web activity, you would need to do a lot of work, basically writing the csv yourself in the Web body, just referencing the output of the previous Web activity. You could do some clever work with looping to automate doing multiple records, but still very tedious.

What did interest me, was you mentioned wanting to get 2 records or "that part of data". If the ask is really "How do I only write some of the records returned, not all the records?" Then Web activity actually becomes a more reasonable option, after Data Flow.

Clarify which and I'll share more details.

List item
Amar Agnihotri 926 Reputation points

2022-06-23T05:01:29.677+00:00

Hi @Wafa Ibrahim Ali Asakrah ,

I would like to tell that i am already using web activity and REST connector to call the api

This is my source dataset

The question is how can i copy that part of data from the web output and then store it in csv files in datalake.
MartinJaffer-MSFT 26,236 Reputation points

2022-06-23T18:50:28.693+00:00

My apologies, if I was not clear enough. I meant to use the Copy activity instead of Web activity. Your picture shows 2 Web activities, no copy activity. See below screenshot.

Note it can use the bearer token. Also note the Pagination rules. Yours may look slightly different. I did my best guess, on your implementation.

Does this help, @Amar Agnihotri ?
Amar Agnihotri 926 Reputation points

2022-06-24T05:28:14.033+00:00

Hi @MartinJaffer-MSFT ,

First of all thanks for the reply and putting your efforts to make my path easy. It seems that your guidance can help me getting out of the issues. Now i tell you step by step what have i done so far and where am i stuck now . I am trying to store user activity data using GetActivityEvents REST api . Here is the documentation -

https://learn.microsoft.com/en-us/rest/api/power-bi/admin/get-activity-events

Now what have i done is here -

Since to call this api we need to generate Bearer access token and then we have to use that bearer access token to call this api so here i used two web activities-

The first is being used to generate bearer access token and here is the configuration and code -

and in the output you can see that the bearer access token is being generated -

Now i am passing this output to next web activity which is being used to call the rest api -

In the URL the code is -

where the start and end date are being decided from the parameter period that i declared globally as -
Amar Agnihotri 926 Reputation points

2022-06-24T05:38:11.917+00:00

Continued ----

If we put -10 means the api will fetch the data for the last 10th day and if we put -1 then the data for yesterday will be fetched

Now in the Authorisation section you can see that i am using access token which has been generated as an output of first web activity-

Now lets see the out output of second web activity which is used to call rest api -

You can see in the output that it is fetching some records and a continuation token

Now i am stuck after this . I want to use until activity to iterate through different records untill we get null continuation token . When continuation token will be null untill activity will stop and then we will append all the records which have been fetched and will try to save all those records in a single csv file in a datalake . This is how we will get the records for single day in a csv file. So mainly there are two points where i am stuck at

1) - How to put conditions in until activity to loop through a process and stop when continuation token is null.

2) - How to append all those records which we are getting in each run into single record and write those into csv file and save that file in a datalake.
Amar Agnihotri 926 Reputation points

2022-06-24T05:53:38.75+00:00

continued ---

I am trying this way to achieve that -
I am saving the out put of second web activity in two variables -

The first one is storing the record -

and the second one is storing the continuation token -

Now i am passing the new continuation token to until activity as shown -

I am putting this null condition in expression in until activity

and Inside until activity i am using three activities as -

Here the Web activity is being used to again call the rest api using the new continuation token which has been stored in a variable

This is the URL -

and again i am using two variables two store the record coming and other one to store the new continuation token .

I was thinking of that when continuation token == null will match then this until activity will stop and we will have all the records in the variable and then we will perform append and copy activity.
Amar Agnihotri 926 Reputation points

2022-06-24T05:56:22.3+00:00

continued ---

I have done above steps so far but my until loop is not working and also i am not able to get that how to use copy activity and where to use that . Should i use that inside until or outside until?

Hi @MartinJaffer-MSFT I hope you understand my problem now.. Now if you have seen all the above description of my task then i would like you to suggest a better workflow to achieve that .

Thanks in advance
Amar Agnihotri 926 Reputation points

2022-06-24T06:21:03.63+00:00

@MartinJaffer-MSFT ,
Hi ,
I have tried that way also but getting an error-

This is a result of second activity

after this i am using copy activity as -

but getting this error in copy activity

Your answer

MartinJaffer-MSFT 26,236 Reputation points

2022-06-22T18:04:08.523+00:00

Hello and welcome to Microsoft Q&A @Amar Agnihotri .

Your ask seems to be either "How do I write the output of Web activity to a file" or "How do I only write some of the data?".

Generally, if you want to copy data, you should use the Copy Data activity. You might want to look into the REST connector and HTTP connector. The REST connector has pagination abilities to leverage the continuationToken. Actually, you would want to leverage the "continuationUri" in the "absoluteUrl" pagination rule, is easier.

The reason I am not suggesting Web activity, is because Web activity works much differently, it makes available to the pipeline the direct response. As such, you would need 2-3 Web activities to move data, one to read, one to write. Also, it does not to format change (i.e. json -> csv) and does not make guarantees to the fidelity of non-text data (compressed or binary, etc.).

For the writing Web activity, you would need to do a lot of work, basically writing the csv yourself in the Web body, just referencing the output of the previous Web activity. You could do some clever work with looping to automate doing multiple records, but still very tedious.

What did interest me, was you mentioned wanting to get 2 records or "that part of data". If the ask is really "How do I only write some of the records returned, not all the records?" Then Web activity actually becomes a more reasonable option, after Data Flow.

Clarify which and I'll share more details.

List item
Amar Agnihotri 926 Reputation points

2022-06-23T05:01:29.677+00:00

Hi @Wafa Ibrahim Ali Asakrah ,

I would like to tell that i am already using web activity and REST connector to call the api

This is my source dataset

The question is how can i copy that part of data from the web output and then store it in csv files in datalake.
MartinJaffer-MSFT 26,236 Reputation points

2022-06-23T18:50:28.693+00:00

My apologies, if I was not clear enough. I meant to use the Copy activity instead of Web activity. Your picture shows 2 Web activities, no copy activity. See below screenshot.

Note it can use the bearer token. Also note the Pagination rules. Yours may look slightly different. I did my best guess, on your implementation.

Does this help, @Amar Agnihotri ?
Amar Agnihotri 926 Reputation points

2022-06-24T05:28:14.033+00:00

Hi @MartinJaffer-MSFT ,

First of all thanks for the reply and putting your efforts to make my path easy. It seems that your guidance can help me getting out of the issues. Now i tell you step by step what have i done so far and where am i stuck now . I am trying to store user activity data using GetActivityEvents REST api . Here is the documentation -

https://learn.microsoft.com/en-us/rest/api/power-bi/admin/get-activity-events

Now what have i done is here -

Since to call this api we need to generate Bearer access token and then we have to use that bearer access token to call this api so here i used two web activities-

The first is being used to generate bearer access token and here is the configuration and code -

and in the output you can see that the bearer access token is being generated -

Now i am passing this output to next web activity which is being used to call the rest api -

In the URL the code is -

where the start and end date are being decided from the parameter period that i declared globally as -
Amar Agnihotri 926 Reputation points

2022-06-24T05:38:11.917+00:00

Continued ----

If we put -10 means the api will fetch the data for the last 10th day and if we put -1 then the data for yesterday will be fetched

Now in the Authorisation section you can see that i am using access token which has been generated as an output of first web activity-

Now lets see the out output of second web activity which is used to call rest api -

You can see in the output that it is fetching some records and a continuation token

Now i am stuck after this . I want to use until activity to iterate through different records untill we get null continuation token . When continuation token will be null untill activity will stop and then we will append all the records which have been fetched and will try to save all those records in a single csv file in a datalake . This is how we will get the records for single day in a csv file. So mainly there are two points where i am stuck at

1) - How to put conditions in until activity to loop through a process and stop when continuation token is null.

2) - How to append all those records which we are getting in each run into single record and write those into csv file and save that file in a datalake.
Amar Agnihotri 926 Reputation points

2022-06-24T05:53:38.75+00:00

continued ---

I am trying this way to achieve that -
I am saving the out put of second web activity in two variables -

The first one is storing the record -

and the second one is storing the continuation token -

Now i am passing the new continuation token to until activity as shown -

I am putting this null condition in expression in until activity

and Inside until activity i am using three activities as -

Here the Web activity is being used to again call the rest api using the new continuation token which has been stored in a variable

This is the URL -

and again i am using two variables two store the record coming and other one to store the new continuation token .

I was thinking of that when continuation token == null will match then this until activity will stop and we will have all the records in the variable and then we will perform append and copy activity.
Amar Agnihotri 926 Reputation points

2022-06-24T05:56:22.3+00:00

continued ---

I have done above steps so far but my until loop is not working and also i am not able to get that how to use copy activity and where to use that . Should i use that inside until or outside until?

Hi @MartinJaffer-MSFT I hope you understand my problem now.. Now if you have seen all the above description of my task then i would like you to suggest a better workflow to achieve that .

Thanks in advance
Amar Agnihotri 926 Reputation points

2022-06-24T06:21:03.63+00:00

@MartinJaffer-MSFT ,
Hi ,
I have tried that way also but getting an error-

This is a result of second activity

after this i am using copy activity as -

but getting this error in copy activity

Share via

How to copy data from web URL output ?

Your answer