How to export application insights data into smaller json file within same hour

Question

How to export application insights data into smaller json file within same hour

Ali Gohar 20

Hi,

As continuous export is not available for workspace-based application insights resources and we have to use diagnostic settings for exporting the data to our storage. we are working on migrating application insights resources to workspace-based resources. We were exporting our event data (custom events) to the data lake storage account by continuous export and now have to shift it to diagnostic settings. With diagnostic settings, it is adding all the data for an hour in one JSON file (PT1H.json) which is an append blob, but for my work, I want to have multiple smaller files (As we have multiple smaller blobs from continuous export), so that I can process those files separately.

I am wondering if it is possible to create multiple smaller JSON files within the same hour (Every 5 or 10 minutes)? is there any configuration for that on application insights or data lake storage side?

Regards,

Ali

Accepted answer

0 additional answers

Your answer

Answer 1

AnuragSingh-MSFT 21,506

@Ali Gohar , thank you for the question.

There are no settings that can be applied to alter the frequency and content of files that get created using Diagnostic Settings exporting data to Azure Storage Account. In case you need some transformation on the exported logs, you could try using Azure function/Data Factory etc. to transform the data once stored in storage account.

Ali Gohar 20 Reputation points

2023-08-16T09:51:02.3466667+00:00

Thanks for clarification
Ali Gohar 20 Reputation points

2023-08-16T10:39:27.52+00:00

@AnuragSingh-MSFT For workspace-based application insights Microsoft suggests using diagnostic settings for exporting data, but granularity for JSON files is low. Can we use Data Export (tab in log analytics workspace) from the log analytics workspace to export data to the storage account because it is creating JSON files after every 5 minutes? Is there any difference in exporting the data from the diagnostics settings or the Data export tab from the log analytics workspace (only application insights data is going to the log analytics workspace)?
AnuragSingh-MSFT 21,506 Reputation points

2023-08-21T04:44:04.1833333+00:00

Ali Gohar, thank you for the reply. The export option in "Log Analytics" can be used in this case and as you rightly mentioned, it creates blobs that are stored in 5-minute folders. For details, see Log Analytics Workspace Data export: Storage Account.

Another key difference (apart from granularity of logs being stored) in how data gets exported using the Diagnostic Setting and Export option is the latency. As described in the overview section, the ingestion Pipeline forwards data to both locations simultaneously to LA workspace as well as the Storage Account/Event Hubs. However, with diagnostic settings, there is a lag of few minutes between data getting stored in LA workspace and then it appearing in the storage account.

Even if there are other logs getting to the LA workspace apart from it being the backend workspace for AppInsights, the export option provides opportunity to select the specific table from which the data needs to be exported to storage. Therefore, you may specifically select the corresponding tables of the AppInsights from LA workspace and use this option as well. For details on corresponding table names in AppInsights and in LA workspace, see AppInsights Workspace Table Structure.

Hope this helps.

Share via

How to export application insights data into smaller json file within same hour

0 additional answers

Your answer