Hi Jonathan
This is actually a really good question and often a cause for some confusion.
Firstly lets take a step back , because when we're talking about App Insights and Az Monitor Diagnostic Settings we're really talking about different things.
- App Insights is agent or SDK based APM (app performance monitoring) and these days, as you point out, it uses Log Analytics as the resource for storing the data collected (a kind of read optimized tabular database) . Prior to that "classic" it actually used a similar technology to Log Analytics but that resource was hidden away and not accessible directly.
*Az Monitor Diagnostic setting
This applies to almost all Az resources and principally it allows you to pick up Logs and send them to a destination of your choice, mostly commonly Log Analytics. You have other choices too - like streaming out to Event Hubs or sending to a storage account.
This facility will also allow you to pick up Az monitor metrics - I tend to advise customers not to do this (opt into sending metrics to log analytics) unless they really have a good reason.
Why? Because general metrics are retained for you in Az for free and accessible
Retention for metrics is discussed here https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-platform-metrics#retention-of-metrics
The key point is you already get them for free. If you send them to Log Analytics it costs you (ingress charge) and so you need to have a good reason for doing that. It can be useful in some scenarios but I dont come across the need often
To help with the Q&A here lets provide some examples where you definitely want Diagnostic Settings enabled
For example
*Application Gateway, Azure Front Door - both these allow for Access Logs and WAF Logs (if you're using the WAF policy). And clearly these are pretty useful in context and almost all customers want and need them. If you dont configure/enable the collection - they dont exist.
Another good example in that category is Azure Firewall.
*App Service provides a number of possible logs that could be collected - certainly the web server (access log) is one to consider
**
*
In your case you mention also have configured diagnostic settings on both App Insights (as a resource) and Log Analytics (as a resource)
You almost certainly dont need to do this. Its there to provide you a way to egress certain tables if you want to (for example to a downstream SIEM)
If you're just picking up APM data from App Insights to store in Log Analytics, this could be from a function or App Service then you dont need diagnostic settings on either the LA or AI resource
*
So back to App service / Functions
Is there overlap between the logs you can export from App service via diagnostic settings and App Insights? Possibly, but it depends what you're doing
IF we look at what's available for App service https://learn.microsoft.com/en-us/azure/app-service/troubleshoot-diagnostic-logs#overview then we see one place you may get overlap
Which is application logs
Now, if you're just using App Insights for this - then the logs go to the "traces " (or AppTraces) table and all's good. But it is possible through a log abstraction to have App Insights as a log sink and still be writing the logs out. In this case , and depending on a lot of detail linked from that doc, the diagnostic setting MIGHT pick this up as well. But it's pretty easy to check. You really want to do one or the other in majority of cases
In summary then:
*Only opt into logs from Diagnostic settings which make sense (for example a web access log, or firewall log, or anything else which is valuable) on a per resource basis)
*most customers dont use diagnostic logs on App Insights/Log Analytics resource. Although there are edge case reason you may do this (like exporting)
*dont opt into metrics via Az diagnostics unless you really need them in log queries or for a solution you opt into on top of Az logs.
*Do send app logs through app insights, but you probably dont need to persist them anywhere else using a standard logging framework
*Do watch the ingress costs using the cost management query available to you in Log Analytics and check costs once a month or so to ensure they're within budget
I think I covered everything, let me know if not or any additional queries on this subject