Microsoft.Azure.NotificationHubs.Messaging.MessagingException: Unexpected exception encountered

Question

I am calling NotificationHubClient.UpdateRegistrationAsync, and I am getting the below error. It seems to happen whether I'm calling in a loop or just making a single call. The error started a week or two ago. We pay for 200,000 devices and are only using 3,323 of them. With an error like "Unexpected exception encountered," I don't know what to do. I also don't know what to do with the Tracking ID. I wonder if I can enter it somewhere to get more information.
I would like advice on what to try. Thank you.

Exception:
Type: Microsoft.Azure.NotificationHubs.Messaging.MessagingException
Message: Unexpected exception encountered TrackingId:0fc0ee10-1817-4f3e-b0eb-017b8e29e39a,TimeStamp:2022-09-27T11:00:37.6378315Z
Source: Microsoft.Azure.NotificationHubs
Stack Trace:
at Microsoft.Azure.NotificationHubs.ExceptionsUtility.HandleUnexpectedException(Exception ex, String trackingId)
at Microsoft.Azure.NotificationHubs.NotificationHubClient.SendRequestAsync(HttpRequestMessage request, String trackingId, HttpStatusCode[] successfulResponseStatuses, CancellationToken cancellationToken)
at Microsoft.Azure.NotificationHubs.NotificationHubClient.SendRequestAsync(HttpRequestMessage request, String trackingId, HttpStatusCode successfulResponseStatus, CancellationToken cancellationToken)
at Microsoft.Azure.NotificationHubs.NotificationHubClient.GetAllEntitiesImplAsyncTEntity

InnerException 1:
Type: System.Net.Http.HttpRequestException
Message: The SSL connection could not be established, see inner exception.
Source: System.Net.Http
Stack Trace:
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
at Microsoft.Azure.NotificationHubs.NotificationHubClient.SendRequestAsync(HttpRequestMessage request, String trackingId, HttpStatusCode[] successfulResponseStatuses, CancellationToken cancellationToken)

InnerException 2:
Type: System.IO.IOException
Message: Received an unexpected EOF or 0 bytes from the transport stream.
Source: System.Net.Security
Stack Trace:
at System.Net.Security.SslStream.<FillHandshakeBufferAsync>g__InternalFillHandshakeBufferAsync|182_0TIOAdapter
at System.Net.Security.SslStream.ReceiveBlobAsyncTIOAdapter
at System.Net.Security.SslStream.ForceAuthenticationAsyncTIOAdapter
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)

Answer 1

To benefit the community, just sharing summary of our comments/offline discussions:

Suggested to implement, retry policy in the code, as that would be the best bet considering the less amount of time and intermittent pattern of the issue.

NotificationHubSettings.RetryOptions Property

NotificationHubRetryOptions Class

We suspect from the exception that it is highly likely that the SSL handshake is successful, but the connection is shutting down in between which could be either from Service (Notification Hub) or it could be client as well. The machine from where the code runs could also be factored-in, as onprem there are multiple factors which could affect the SSL connection.

Since the issue was intermittent and doesn’t follow a pattern. If feasible, capture a fiddler or network traces when the issue occurs for deeper investigation.

Thanks for your patience and collaboration, wessleym