Azure Function losing logs

Question

Azure Function losing logs

Prerna Bhand 20

Hi,

I have a service bus - topic triggered Azure Function running in EP1 mode with an tested scale out setting of 25 instances. It is unfortunately on 3.x version at the moment, because we have some backward compatibility issues with MS Dataverse. A couple of problems I am facing with bulk migration (around 5-6 million records)

Every incoming message with its whole body is logged by the function at Information level. I pick it up from Application Insights to just check how many messages were processed by the function. So far it worked, but recently it appears as if the function loses some of the logging and these logs simply do not appear when I query in Application Insights. When I see the function execution count, that appears correct with the number of messages received in the queue. Why is the function losing logs? Here is the default host.json configuration... { "version": "2.0", "logging": { "applicationInsights": { "samplingSettings": { "isEnabled": true, "excludedTypes": "Request" } } }}
For production level setting, I just want to see errors and exceptions using the log level - Error. Is it going to affect the other function monitoring related features? I rely heavily on the function monitoring (Because the logs are not complete) to check the execution count, success count etc . Unfortunately the documentation is too confusing here.
Could the 3.x function version be a problem here? I cannot upgrade it to 4.x without proper testing and this can only be done in a month's time. But if its a road blocker, I would need to address it immediately.

Many thanks in advance for any inputs..

MayankBargali-MSFT 70,876 Reputation points

2024-01-22T10:33:45.3233333+00:00

@Prerna Bhand Thanks for reaching out.

When you mentioned that "function loses some of the logging and these logs simply do not appear when I query in Application Insights." so I believe you query for a timeframe in azure portal and the count was retrieved and you validated the same with the message that was send.

Can you confirm what was the count that was retirved as there is limt when you query application insights. If you are using portal than Maximum records returned by a log query is 30K as documented here so want to confirm if this was not the case for which you observed this behavior.

Did you observe this behavior only for some instance. I will suggest to validate for the short timeframe. Is this issue intermittent?

To answer your question:

For the first query it's possible that the logs are being lost due to sampling settings in your host.json configuration. The sampling settings you provided indicate that only certain types of logs (excluding "Request") will be included in Application Insights. If your logs are being sampled out, they won't appear in Application Insights. You may want to adjust your sampling settings to include the logs you're interested in case if all logs are not logged correctly. For Intermittent issue it should not be configuration issue.

Regarding your second question, setting the log level to "Error" will only log errors and exceptions, and will not affect other monitoring features such as execution count and success count. However, keep in mind that you may miss important information that could help you diagnose issues if you only log errors and exceptions.

Regarding your third question without more information, it's difficult to say for sure. If you're unable to upgrade to version 4.x at this time, you may want to try adjusting your logging settings to see if that resolves the issue.
Prerna Bhand 20 Reputation points

2024-01-22T11:16:12.89+00:00
Thanks @MayankBargali-MSFT . To answer your question about the 30K limit, yes this limit was not overcrossed and I was checking in a timeframe. The query looked like below. I expected the count to be considerably more according to the function execution count, but it was quite less.

traces | where timestamp between (datetime('2024-01-16T00:00:00')..datetime('2024-01-17T23:59:00')) |where message has "Message ID" | summarize count()
MayankBargali-MSFT 70,876 Reputation points

2024-01-23T14:14:54.47+00:00

@Prerna Bhand Do you see any exact occurrence so it can help to troubleshoot. I am not sure if this would help in finding the cause as we would need these needs to look into any one occurrence to validate whether the logs were pushed or not. This would need to look at backend logs and review the logs that you mentioned over one on one support. I will suggest you to engage us over one on one supported request to validate it further. In case if you don't have the support plan then please let me know.
Prerna Bhand 20 Reputation points

2024-02-04T15:41:29.8+00:00

Thanks @MayankBargali-MSFT , I do not have a support plan and it indeed sounds like a plan to engage one-on-one support. How can I go about it? Please let me know. Also for your info, I tested a recent migration with 10K messages, isSamplingEnabled = False, I could see all messages in the logs. But was this the cause of logs being lost earlier? Thanks
MayankBargali-MSFT 70,876 Reputation points

2024-02-06T04:04:29.6133333+00:00

@Prerna Bhand Please refer to my private message so we can connect offline. isSamplingEnabled = False should not cause this behavior and this would need to investigate further.

Your answer

MayankBargali-MSFT 70,876 Reputation points

2024-01-22T10:33:45.3233333+00:00

@Prerna Bhand Thanks for reaching out.

When you mentioned that "function loses some of the logging and these logs simply do not appear when I query in Application Insights." so I believe you query for a timeframe in azure portal and the count was retrieved and you validated the same with the message that was send.

Can you confirm what was the count that was retirved as there is limt when you query application insights. If you are using portal than Maximum records returned by a log query is 30K as documented here so want to confirm if this was not the case for which you observed this behavior.

Did you observe this behavior only for some instance. I will suggest to validate for the short timeframe. Is this issue intermittent?

To answer your question:

For the first query it's possible that the logs are being lost due to sampling settings in your host.json configuration. The sampling settings you provided indicate that only certain types of logs (excluding "Request") will be included in Application Insights. If your logs are being sampled out, they won't appear in Application Insights. You may want to adjust your sampling settings to include the logs you're interested in case if all logs are not logged correctly. For Intermittent issue it should not be configuration issue.

Regarding your second question, setting the log level to "Error" will only log errors and exceptions, and will not affect other monitoring features such as execution count and success count. However, keep in mind that you may miss important information that could help you diagnose issues if you only log errors and exceptions.

Regarding your third question without more information, it's difficult to say for sure. If you're unable to upgrade to version 4.x at this time, you may want to try adjusting your logging settings to see if that resolves the issue.
Prerna Bhand 20 Reputation points

2024-01-22T11:16:12.89+00:00

Thanks @MayankBargali-MSFT . To answer your question about the 30K limit, yes this limit was not overcrossed and I was checking in a timeframe. The query looked like below. I expected the count to be considerably more according to the function execution count, but it was quite less.

traces | where timestamp between (datetime('2024-01-16T00:00:00')..datetime('2024-01-17T23:59:00')) |where message has "Message ID" | summarize count()
MayankBargali-MSFT 70,876 Reputation points

2024-01-23T14:14:54.47+00:00

@Prerna Bhand Do you see any exact occurrence so it can help to troubleshoot. I am not sure if this would help in finding the cause as we would need these needs to look into any one occurrence to validate whether the logs were pushed or not. This would need to look at backend logs and review the logs that you mentioned over one on one support. I will suggest you to engage us over one on one supported request to validate it further. In case if you don't have the support plan then please let me know.
Prerna Bhand 20 Reputation points

2024-02-04T15:41:29.8+00:00

Thanks @MayankBargali-MSFT , I do not have a support plan and it indeed sounds like a plan to engage one-on-one support. How can I go about it? Please let me know. Also for your info, I tested a recent migration with 10K messages, isSamplingEnabled = False, I could see all messages in the logs. But was this the cause of logs being lost earlier? Thanks
MayankBargali-MSFT 70,876 Reputation points

2024-02-06T04:04:29.6133333+00:00

@Prerna Bhand Please refer to my private message so we can connect offline. isSamplingEnabled = False should not cause this behavior and this would need to investigate further.

Share via

Azure Function losing logs

Your answer