Alternative to Application Insights for logging messages larger than 32 kB

Question

What alternatives are there in Azure to Application Insights for logging data that may be larger than 32 kB?

Our application sends requests to a third-party API to process a list of order items. We want to log request and response data for the API calls to determine which order items sent to the API are processed successfully and which fail. We are currently logging the request and response data in JSON form to the Application Insights traces table, message column. However, that column has a 32 kB size limit and sometimes our request data can reach 60 kB in size, resulting in it being truncated. We can't break up the request to make it smaller because all items in the same order need to be processed together.

Is there somewhere else in Azure we could write our logs that doesn't have a 32 kB limit? I'm a little confused about Azure Monitor Logs and Log Analytics, compared to Application Insights, so I don't know if they would be a more flexible alternative to Application Insights. And I don't know if there are other possible alternatives in Azure.

We're not just recording a single field in each log message. We also need to be able to record other, smaller, fields, like OperationId and CorrelationId, in addition to the request and response data.

Although the data in each log message may be up to 60 kB we only make around 1000 requests to the API per day, and would like to keep the data for no more than 90 days.

Answer 1

Hi @Simon Elms ,

Application Insights telemetry stores its data to services similar to Azure Data Explorer (basically, this is PaaS service of SaaS for Log Analytics+Application Insights telemetry).
You could have a look at this service due to there is no hard limits on the amount and size of data that can be ingested into a database. Ingested data is persisted to limitless durable storage (Azure blob storage).

https://learn.microsoft.com/en-us/azure/data-explorer/data-explorer-overview