How to iterate web response based on last_timestamp

Question

How to iterate web response based on last_timestamp

Anuganti Suresh 200

Hello,

web activity using start_timestamp 1/1/1970 0:0:0 (converted to milliseconds "1692815400000") and generating response. In web response will get "last_timestamp": "1693329602424" as attached image. If the result has more than 1000 activities, the last_timestamp in return is used for getting the next list results. How to make design flow of activities?

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Anuganti Suresh 200 Reputation points

2023-12-13T09:34:24.1733333+00:00

Question is for access token generated from web(Get Access Token Audit Log) and using this token web (Response Audit Log) extract response from REST API. This response have limitation of 1000 users per token. If in case more than 1000 users are available then by using last_timestamp it has to pull next records.
phemanth 15,775 Reputation points Microsoft External Staff Moderator

2023-12-13T14:44:17.41+00:00

@Anuganti Suresh

Here’s a high-level design flow that you can consider:

Initialize your start_timestamp as 1692815400000 (1/1/1970 0:0:0 in milliseconds).

Send a request to the server with the start_timestamp.

Receive the response from the server. The response should contain the activities and the last_timestamp (“1693329602424” in your case).

Check the number of activities in the response. If the number of activities is more than 1000:

Update the start_timestamp with the last_timestamp from the response.

Go back to step 2. This creates a loop that continues fetching activities in batches until there are less than 1000 activities in the response.

Once you receive less than 1000 activities in the response, you can end the process.

This flow ensures that you’re fetching activities in manageable batches, and using the last_timestamp to keep track of where to start fetching in the next batch. Remember to handle any potential errors or exceptions that might occur during the process.

Please note that this is a high-level flow and the actual implementation might vary based on the specific requirements of your project and the API you’re working with
phemanth 15,775 Reputation points Microsoft External Staff Moderator

2023-12-14T07:41:33.5+00:00

@Anuganti Suresh

just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Anuganti Suresh 200 Reputation points

2023-12-14T07:45:33+00:00

@phemanth , tried pipeline as below. If you have better/optimized pipeline suggest me.
phemanth 15,775 Reputation points Microsoft External Staff Moderator

2023-12-15T06:30:30.17+00:00

@Anuganti Suresh

Efficient Web Activities: Ensure that the “Get Access Token Activity” and “Response Activity Audit” are as efficient as possible to reduce waiting time.

Parallel Processing: If your tasks are independent, consider parallel processing to handle multiple tasks simultaneously.

Condition Simplification: Review the conditions in “If Condition1” to ensure they are not too complex or can be simplified.

Loop Optimization: Check if there is a need for loop optimization in both the true and false branches of your condition.

ForEach Optimization: In the “ForEach1” block, ensure that the steps like “ResponseIterate,” “Copydata1,” and the IF condition block are optimized for best performance.

Your answer

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Anuganti Suresh 200 Reputation points

2023-12-13T09:34:24.1733333+00:00

Question is for access token generated from web(Get Access Token Audit Log) and using this token web (Response Audit Log) extract response from REST API. This response have limitation of 1000 users per token. If in case more than 1000 users are available then by using last_timestamp it has to pull next records.
phemanth 15,775 Reputation points Microsoft External Staff Moderator

2023-12-13T14:44:17.41+00:00

@Anuganti Suresh

Here’s a high-level design flow that you can consider:

Initialize your start_timestamp as 1692815400000 (1/1/1970 0:0:0 in milliseconds).

Send a request to the server with the start_timestamp.

Receive the response from the server. The response should contain the activities and the last_timestamp (“1693329602424” in your case).

Check the number of activities in the response. If the number of activities is more than 1000:

Update the start_timestamp with the last_timestamp from the response.

Go back to step 2. This creates a loop that continues fetching activities in batches until there are less than 1000 activities in the response.

Once you receive less than 1000 activities in the response, you can end the process.

This flow ensures that you’re fetching activities in manageable batches, and using the last_timestamp to keep track of where to start fetching in the next batch. Remember to handle any potential errors or exceptions that might occur during the process.

Please note that this is a high-level flow and the actual implementation might vary based on the specific requirements of your project and the API you’re working with
phemanth 15,775 Reputation points Microsoft External Staff Moderator

2023-12-14T07:41:33.5+00:00

@Anuganti Suresh

just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Anuganti Suresh 200 Reputation points

2023-12-14T07:45:33+00:00

@phemanth , tried pipeline as below. If you have better/optimized pipeline suggest me.
phemanth 15,775 Reputation points Microsoft External Staff Moderator

2023-12-15T06:30:30.17+00:00

@Anuganti Suresh

Efficient Web Activities: Ensure that the “Get Access Token Activity” and “Response Activity Audit” are as efficient as possible to reduce waiting time.

Parallel Processing: If your tasks are independent, consider parallel processing to handle multiple tasks simultaneously.

Condition Simplification: Review the conditions in “If Condition1” to ensure they are not too complex or can be simplified.

Loop Optimization: Check if there is a need for loop optimization in both the true and false branches of your condition.

ForEach Optimization: In the “ForEach1” block, ensure that the steps like “ResponseIterate,” “Copydata1,” and the IF condition block are optimized for best performance.

Share via

How to iterate web response based on last_timestamp

Your answer