Will I lose telemetry by disabling diagnostic settings and enabling workspace based application insights?

Anonymous
2024-02-16T12:10:59.0833333+00:00

We recently converted all our application insights resources from classic to workspace-based ahead of the end-of-support deadline. After we did this, we noticed duplicate logs appearing in app insights. From searching around, we found the cause was that the diagnostic settings on these resources were also streaming telemetry to the log analytics workspace. Removing the diagnostic settings from these resources stopped the duplicates appearing, but we have two questions following this:

  • The azure docs on diagnostic settings says 'each Azure resource requires its own diagnostic setting'. Can you confirm that it's not absolutely essential for each resource to have a diagnostic setting enabled? I assume that this point means 'each resource requires its own diagnostic setting if you want that resource to export telemetry'
  • If we remove diagnostic settings and rely just on the workspace-based AI resource, will we be losing any telemetry? In other words, does making the application insights resources workspace-based mean all types of telemetry are streamed to log analytics, making diagnostic settings redundant?

For further context, we previously had diagnostic settings enabled at three different 'levels' of resource:

  • on the application resource (function app or app service)
  • on the application insights resource
  • on the log analytics workspace resource

Each diagnostic setting was configured to stream all its logs to the same log analytics workspace. We found we needed to remove the diagnostic setting from all three levels to stop the duplicated logs. We want to have confidence that doing this won't mean we lose any logs or metrics.

Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
3,662 questions
0 comments No comments
{count} votes

Accepted answer
  1. Ben Gimblett 4,560 Reputation points Microsoft Employee
    2024-02-16T16:37:04.5366667+00:00

    Hi Jonathan
    This is actually a really good question and often a cause for some confusion.

    Firstly lets take a step back , because when we're talking about App Insights and Az Monitor Diagnostic Settings we're really talking about different things.

    • App Insights is agent or SDK based APM (app performance monitoring) and these days, as you point out, it uses Log Analytics as the resource for storing the data collected (a kind of read optimized tabular database) . Prior to that "classic" it actually used a similar technology to Log Analytics but that resource was hidden away and not accessible directly.

    *Az Monitor Diagnostic setting
    This applies to almost all Az resources and principally it allows you to pick up Logs and send them to a destination of your choice, mostly commonly Log Analytics. You have other choices too - like streaming out to Event Hubs or sending to a storage account.
    This facility will also allow you to pick up Az monitor metrics - I tend to advise customers not to do this (opt into sending metrics to log analytics) unless they really have a good reason.
    Why? Because general metrics are retained for you in Az for free and accessible
    Retention for metrics is discussed here https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-platform-metrics#retention-of-metrics
    The key point is you already get them for free. If you send them to Log Analytics it costs you (ingress charge) and so you need to have a good reason for doing that. It can be useful in some scenarios but I dont come across the need often

    To help with the Q&A here lets provide some examples where you definitely want Diagnostic Settings enabled
    For example
    *Application Gateway, Azure Front Door - both these allow for Access Logs and WAF Logs (if you're using the WAF policy). And clearly these are pretty useful in context and almost all customers want and need them. If you dont configure/enable the collection - they dont exist.
    Another good example in that category is Azure Firewall.
    *App Service provides a number of possible logs that could be collected - certainly the web server (access log) is one to consider

    **
    *
    In your case you mention also have configured diagnostic settings on both App Insights (as a resource) and Log Analytics (as a resource)
    You almost certainly dont need to do this. Its there to provide you a way to egress certain tables if you want to (for example to a downstream SIEM)
    If you're just picking up APM data from App Insights to store in Log Analytics, this could be from a function or App Service then you dont need diagnostic settings on either the LA or AI resource

    *

    So back to App service / Functions

    Is there overlap between the logs you can export from App service via diagnostic settings and App Insights? Possibly, but it depends what you're doing

    IF we look at what's available for App service https://learn.microsoft.com/en-us/azure/app-service/troubleshoot-diagnostic-logs#overview then we see one place you may get overlap
    Which is application logs
    Now, if you're just using App Insights for this - then the logs go to the "traces " (or AppTraces) table and all's good. But it is possible through a log abstraction to have App Insights as a log sink and still be writing the logs out. In this case , and depending on a lot of detail linked from that doc, the diagnostic setting MIGHT pick this up as well. But it's pretty easy to check. You really want to do one or the other in majority of cases

    In summary then:
    *Only opt into logs from Diagnostic settings which make sense (for example a web access log, or firewall log, or anything else which is valuable) on a per resource basis)
    *most customers dont use diagnostic logs on App Insights/Log Analytics resource. Although there are edge case reason you may do this (like exporting)
    *dont opt into metrics via Az diagnostics unless you really need them in log queries or for a solution you opt into on top of Az logs.
    *Do send app logs through app insights, but you probably dont need to persist them anywhere else using a standard logging framework
    *Do watch the ingress costs using the cost management query available to you in Log Analytics and check costs once a month or so to ensure they're within budget

    I think I covered everything, let me know if not or any additional queries on this subject

    2 people found this answer helpful.

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.