Azure Monitor IIS Logs upload stopped working for one machine

Question

We have IIS Log Uploads configured (via Log Analytics Workspace > Agents Configuration > IIS Logs) and has been working fine for months. I recently noticed that around the IIS Log upload for one of the three machines stopped working around 5th Feb 2021. It's continuing to work fine for the others. Other logs, e.g. Event and Perf, are still uploading fine.

Any ideas how I can troubleshoot this?

Events like this are in the event log for the machine that fails to upload:

A module of type "Microsoft.EnterpriseManagement.Mom.Modules.CloudFileUpload.CloudFileUploadWriteAction" reported an exception Microsoft.EnterpriseManagement.Mom.Modules.CloudFileUpload.FileUploadException: Unable to get blob container for iislog from https://[SOME_GUID].ods.opinsights.azure.com/ContainerService.svc. Will keep trying according to the specified policy. ---> System.Net.WebException: The request was aborted: The request was canceled.
at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult)
at Microsoft.EnterpriseManagement.Mom.Modules.CloudFileUpload.CloudFileUploadWriteAction.GetRequestStreamCallback(IAsyncResult asynchronousResult)
--- End of inner exception stack trace --- which was running as part of rule "Microsoft.IntelligencePacks.IISLogUpload.LogUpload" running for instance "" with id:"[SOME_GUID]" in management group "AOI-[SOME_GUID]".

and

A module of type "Microsoft.EnterpriseManagement.Mom.Modules.CloudFileUpload.CloudFileUploadWriteAction" reported an exception Microsoft.EnterpriseManagement.Mom.Modules.CloudFileUpload.FileUploadException: Unable to upload file W3SVC2-u_ex21020515.log.iislog to https://advisor[A_LONG_NUMBER] .blob.core.windows.net/iislogblob/v0/tenant=[SOME_GUID]/00000000-0000-0000-0000-000000000001/AOI-[SOME_GUID]/[SOME_GUID]/W3SVC2/u_ex21020515.log. Will keep trying according to the specified policy. ---> System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
at System.Net.TlsStream.EndRead(IAsyncResult asyncResult)
at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at Microsoft.EnterpriseManagement.Mom.Modules.CloudFileUpload.CloudBlockBlobLite.EndUploadFromStream(IAsyncResult asyncResult)
at Microsoft.EnterpriseManagement.Mom.Modules.CloudFileUpload.CloudFileUploadWriteAction.UploadFromStreamAsyncCallback(IAsyncResult result)
--- End of inner exception stack trace --- which was running as part of rule "Microsoft.IntelligencePacks.IISLogUpload.LogUpload" running for instance "" with id:"{[SOME_GUID]}" in management group "AOI-[SOME_GUID]".

thanks,

Rory

Answer 1

I've now resolved the issue by uninstalling the agent and reinstalling the latest version (sometimes things are so easy!). I had quite an old version 8.x. In order to install the latest I had to apply a registry change to force .NETFramwork to SchUseStrongCrypto=1 (so for example possibly it stopped working because the logs server required stronger crypto than the agent was using ... although I think the other servers were configured identically).

fyi - no restart required for the uninstall & reinstall process, which was good.

Now it's uploading ... but only files from now onwards :( So if anyone has a suggestion on how to upload old files I'd love to hear it!

Rory