Azure Site Recovery Critical Notification: vCenter health is in Critical state

Question

Greetings,
We have a VMware to azure site recovery VMs and we are receiving this alerts today relating to vCenter:
Message: Discovery of the vCenter server X.X.X.X:443 failed with the error System.ServiceModel.CommunicationException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. ---> System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. at System.Net.HttpWebRequest.GetResponse() at System.ServiceModel.Channels.HttpChannelFactory1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) --- End of inner exception stack trace --- Server stack trace: at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason) at System.ServiceModel.Channels.HttpChannelFactory1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Vim25Api.VimPortType.ContinueRetrievePropertiesEx(ManagedObjectReference _this, String token) at VMware.VSphere.Management.Operations.VSphereUtilities.RetrieveProperties(VimPortType service, ServiceContent serviceContent, PropertyFilterSpec pfSpec) at VMware.VSphere.Management.Operations.VSphereUtilities.GetEntityViews(VimPortType service, ServiceContent serviceContent, String moRefType, String[] moRefProperies) at VMware.VSphere.Management.Operations.VirtualMachineOperations.GetVmViewsByProps() at VMware.VSphere.Management.VSphereManagementClient.GetInfrastructureViewLite(String[] options, OsType osType) at InMageDiscovery.Program.Main(String[] args) .

Possible Causes: Refer to the vSphere Power CLI documentation for possible causes.

Recommendation: Resolve the error: System.ServiceModel.CommunicationException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. ---> System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. at System.Net.HttpWebRequest.GetResponse() at System.ServiceModel.Channels.HttpChannelFactory1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) --- End of inner exception stack trace --- Server stack trace: at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason) at System.ServiceModel.Channels.HttpChannelFactory1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Vim25Api.VimPortType.ContinueRetrievePropertiesEx(ManagedObjectReference _this, String token) at VMware.VSphere.Management.Operations.VSphereUtilities.RetrieveProperties(VimPortType service, ServiceContent serviceContent, PropertyFilterSpec pfSpec) at VMware.VSphere.Management.Operations.VSphereUtilities.GetEntityViews(VimPortType service, ServiceContent serviceContent, String moRefType, String[] moRefProperies) at VMware.VSphere.Management.Operations.VirtualMachineOperations.GetVmViewsByProps() at VMware.VSphere.Management.VSphereManagementClient.GetInfrastructureViewLite(String[] options, OsType osType) at InMageDiscovery.Program.Main(String[] args) . Retry the operation once the error is resolved.

The replication is not affected, we receive successful heartbeats from process server and vCenter. We recently upgraded the process server agent to 9.48. Is this a VMware issue or a ASR issue?

Answer 1

Hello @Brenda Rojas Thank you for your patience and I apologize for the delayed response!

I see, you mentioned that Azure Site Recovery notifications are triggered for vCenter health in critical state. Usually these alerts are triggered when there is communication failure. For example: If port 443 is disabled or network connectivity is blocked.

You also mentioned that you recently upgraded the process server agent to 9.48. Ideally upgrades do not cause any failures. But at times, a network blip may cause these kind of issues.

Good thing is, the replication is not affected and you receive successful heartbeats from process server and vCenter.

I don't find this to be a problem. But if this is causing any replication failure then that need to be investigated further to understand root cause for this issue.

Please feel free to reply if you need further assistance in this matter!