How to separate the some of the execution tree logs to show on transaction search on application Insights?

Question

I've have multiple application has let say 20 apps in a sequence(both rest and non rest), out 14 apps are in the loop per each record process. If I Process 1000 records it will loop 14 apps. if any of the record failed to process in any application, I would like to check the sequence where it was failed, but on trasaction search, it is loading whole execution tree of 1000 records processed, due to this application Insights is freezing and also difficult to track the issue.

To solve this issue, we follow to change the operationId and ParentId metrics, tried using Activity Class and other approach using event telemetry. After the changes, If I check on Trasaction Search, and the same issue is repeating loading whole execution tree by combination of operationIds which are involved on the operation.

I would like to separate each record process sequence of 14apps like from 1000 whole processed logs to 1000 separate operations. This way can be track any the issues easily I believe on applicationInsights.

How to solve the whole sequence of transactions into one by one on transaction Search in application insights?

Answer 1

Prasanna Kumar T, thank you for posting this question on Microsoft Q&A.

Based on my understanding, your execution flow involves calling approximately 20 apps/APIs, of which 14 apps/APIs are required to process each event/message which triggered the execution - i.e., if 1000 event/message batch initiated the execution, these are processed as batch for the first 5-6 apps and then each message is individually processed by the 14 apps. I also think you are facing this issue when reviewing the "End-to-end transaction details" in "Transaction search" and the scenario involves "distributed tracing". Please correct me if my understanding is incorrect.

Based on the scenario described, here are my thoughts and suggestions to approach this issue:

1. Operation Id - For this kind of distributed tracing, the Operation_id is the unique global identifier which remains unique throughout the execution flow which enables each sequence of execution to be tied together. For details, see Data model for telemetry correlation. This should not be changed, because in current scenario even if you are able to change the Operation_id just to be able to isolate the loop execution of all events/messages in batch, it will remove this entire operation from the other set of activities and API/Endpoint calls. While this might solve the issue of trying to troubleshoot the looping scenario, it will be difficult to troubleshoot the end-to-end scenario. Therefore, in my opinion this should not be touched.
2. As a solution, I would instead advice querying the "Log Analytics Workspace" directly to be able to filter down the logs to the specific sequence that is required. The telemetry that you see in AppInsights is stored in Log Analytics workspace from whcih the views query and present the visuals. If you click the "LA" button as shown in the image below from a sample "End-to-end transaction detail" page, it would take you to the LA workspace's query editor:

The sample query used to construct this view is (as available after the clicking the button) -

// All telemetry for Operation ID: 279ac2599dXXXXXXXXXXXXXXXXXXXXXX
union *
// Apply filters
| where timestamp > datetime("2023-09-20T02:17:12.741Z") and timestamp < datetime("2023-09-22T02:17:12.741Z")
| where operation_Id == "279ac2599dXXXXXXXXXXXXXXXX"

Basically, the query unions all the entries in tables (union*) and filters them by timestamp and operation_id as the operation sequence of a particular execution flow is required. This emits all the telemetry collected in AppInsights. For the specific case mentioned in the question, you could also add additional filters for the endpoints for which you need to see the logs.

Hope this helps.

If the answer did not help, please add more context/follow-up question for it, and we will help you out. Else, if the answer helped, please click Accept answer so that it can help others in the community looking for help on similar topics.