Missing Azure Function Logs: What more can influence log sampling if its disabled on App Insights and Function Apps host.json

Question

Problem Statement:

We have a strange issue of missing Function App logs; where out of 100 executions, we are missing logs for 10 to 15 executions each time, even though we can see that the respective function was executed for all 100 times as it did what it was supposed to do for all the 100 distinct files.

We have tried running that function app multiple times, but we consistently miss 10 to 15 execution logs out of 100 executions.

Relevant Details for our Environment:

• The relevant Function App for which we are facing missing logs issue has two functions. One is blob triggered and the other is service bus triggered. Logs are missing for both functions, such that for blob triggered function we are missing logs for 10-15 executions out of 100 executions and for service bus triggered function we are missing logs for less than 5 executions out of 100 executions.
• Function Apps are hosted on Windows Elastic Premium Plan (EP1) with Zone Redundancy and Auto Scaling from 3 to x nodes.
• Functions are coded in C# using .NET 6, and deployed using Azure DevOps Pipelines.
• Function Apps have FUNCTIONS_EXTENSION_VERSION = ~4, FUNCTIONS_WORKER_RUNTIME = dotnet, WEBSITE_CONTENTOVERVNET = 1  and vnetRouteAllEnabled. The outbound rules for the NSG on the associated subnet isn't blocking the outbound traffic.
• Data Sampling for Azure Application Insights is set to 100% retention for ingestion sampling. Azure Application Insights is Log Analytics Workspace based with daily cap set in few GBs and we have checked that the missing logs are not due to reaching the daily cap on Log Analytics Workspace.
• Sampling for Application Insights is disabled in the host.json for the Function App so that all the logs are sent to Application Insights without sampling. The log level is configured for our specific classes as the logger is injected using Dependency Injection, and we are happy with the verbosity of logs we get for executions where we do get the logs.
  
• The Function App configuration has got the setting for both APPINSIGHTS_INSTRUMENTATIONKEY and APPLICATIONINSIGHTS_CONNECTION_STRING, pointing to the same Application Insights.

Concluding Remarks:

What's strange is that out of 100 executions, we are getting logs for significant majority of executions, that means that there is nothing which is blocking the submission of logs to App Insights from our function apps, however the missing logs for remaining 10 to 15 executions is strange when sampling is disabled in host.json and no data ingestion sampling is configured on Application Insights. So, what more can influence such a behaviour?

We have tried repeating this exercise multiple times as the first function is blob triggered and we upload 100 files to the relevant blob container in around 3 seconds, which we can see are processed in matter of few seconds as they get moved elsewhere as per the business logic; however, the function execution logs are missing for anywhere between 10 to 15 executions out of those 100 executions each time.

These missing logs will cause us problems during the operational troubleshooting of file deliveries from third parties etc. So, we really need to get this missing logs sorted. We have option to write to custom log tables, but would like to avoid those and rely on the standard logging to Application Insights from Azure Functions capability.

So, would welcome any suggestions/answers for configuration changes to get the logs in Application Insights for 100% of the Function executions.