Exchange 2019 Hybrid Configuration wizard timeout error

Bill Seymour 5

I'm trying to setup hybrid Exchange between my Exchange 2019 server and Microsoft. I'm using the Exchange server for a grand total of two email addresses (different domains), both to the same AD user (me). I'm migrating one domain to make sure everything works, then will add the second domain. My Exchange server is sitting behind NAT with appropriate DNS entries to my Cisco router and it works as expected by itself. I am using Let's Encrypt certificates but am on a residential internet service, so Spamhaus flags me as inappropriate, which means a lot of my emails don't get delivered. My hope is that once the hybrid setup is complete that issue will go away since the MX records will point to MS servers. I've gone through the configuration without issue until I run the hybrid configuration wizard, which fails in the verify stage with this error:

2024.05.25 03:28:53.295 ERROR 10349 [Client=UX, Page=HybridConnectorInstall, Thread=19] The connection to the server '<GUID>.resource.mailboxmigration.his.msappproxy.net' could not be completed., The call to '<GUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' timed out.

Nothing seem amiss, except the MSAPProxy times out. I assume there is something on my end it doesn't like, so just ignores me. :( Has anyone got ideas on where I go from here?

2 answers

Amit Singh 4,861 Reputation points

2024-05-27T08:31:21.7733333+00:00
The error message indicates a timeout when trying to connect to '<GUID>.resource.mailboxmigration.his.msappproxy.net'.

Here are a few troubleshooting steps you can try:

Ensure that there are no network issues preventing your Exchange server.

Check if no firewall rules or network configurations blocking the connection.

Double-check your DNS settings to ensure that the Exchange server can resolve the hostname '<GUID>.resource.mailboxmigration.his.msappproxy.net' correctly.

Verify that your Let's Encrypt certificate is properly configured and trusted by all parties involved.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment
Noah Ma-MSFT 1,765 Reputation points Microsoft Vendor

2024-05-27T09:03:10.3833333+00:00
Hi @Bill Seymour,

Thank you for posting to Microsoft Community.

Based on your description, I understand you got an error in “Validating Hybrid Agent” page.

I suggest you could try the following to troubleshoot.

Check whether TLS 1.2 is enabled.

Please disabled and enabled MRS option and then restarted IIS to take the action effective.

Enable basic authentication on Web Services Virtual Directory by the following command: Set-WebServicesVirtualDirectory -Identity "Server\EWS (default Web site)" –BasicAuthenticaition $true

Check proxy and firewall settings and ensure your firewall settings to allow connections from O365. You can refer to Microsoft 365 URLs and IP address ranges - Microsoft 365 Enterprise | Microsoft Learn for more information.

You could refer to the requirements part of The Microsoft Hybrid Agent Public Preview - Microsoft Community Hub to make sure it is configured correctly.

Also, please check if Exchange Server Extended Protection was turned on, try to turn Extended Protection OFF in IIS (EWS) to see if it works. You could refer to Exchange Server support for Windows Extended Protection | Microsoft Learn for more information.

Hope it helps and if there are anything else you need help, please feel free to contact me.
Please sign in to rate this answer.
Bill Seymour 5 Reputation points

2024-05-27T18:37:41.23+00:00

I checked and I'm set for TLS 1.2 with 1.0 and 1.1 disabled.

I've disabled, then enabled MRS on the EWS directory, then restarted IIS.

Authentication on EWS was set to Anonymous and Windows, I enabled Basic.

I've got the Windows Defender Firewall set to allow incoming and outgoing connection for the correct ports, and have the incoming ports allowed through NAT on the router to the Exchange server.

When I rerun the wizard, I again get about a five minute wait (334 seconds) and the timeout error while Validating Hybrid Agent for Exchange usage.

Bill Seymour 5 Reputation points

2024-05-27T18:41:16.1766667+00:00

Interesting, this time it didn't seem to generate a new logfile in C:\ProgramData\Microsoft Hybrid Service\Logging

Bill Seymour 5 Reputation points

2024-05-27T18:48:57.3+00:00

I take that back, it had a couple of 0 length files dated from the other day, that when opened were actually the correct ones and dated today. File Explorer obviously didn't refresh them.

exception: 'System.Net.WebSockets.WebSocketException (0x80004005): An internal WebSocket error occurred.

That's a different error than before, there is no reference to the resource.mailboxmigration.his.msappproxy.net URL in these files.

Noah Ma-MSFT 1,765 Reputation points Microsoft Vendor

2024-05-28T07:27:06.77+00:00

Hi,

This may also be related to Exchange Server Extended Protection, please double check if turn Extended Protection off in IIS (EWS).

Also, you could use steps below to check the connection between your Exchange on-premises and Exchange online.

Download sample script

Switch EMS to the script location and import the cmdlets by running the following command Import-Module .\HybridManagement.psm1

Check the connection with command below: Test-HybridConnectivity -TestO365Endpoints

For more information, you could refer to “Verify connectivity” of Microsoft Hybrid Agent | Microsoft Learn.

Have you tried to install a new agent. Maybe the Hybrid Wizard will overwrite the failed agent when you did that.

If there are anything missed or you need help, please let me know.

Bill Seymour 5 Reputation points

2024-05-28T17:40:18.9533333+00:00

It looks like the connectivity test goes through fine:

Testing connection to mscrl.microsoft.com on port 80

Testing connection to crl.microsoft.com on port 80

Testing connection to ocsp.msocsp.com on port 80

Testing connection to www.microsoft.com on port 80

Testing connection to login.windows.net on port 443

Testing connection to login.microsoftonline.com on port 443

Testing connection to outlook.office.com on port 443

Performing GET on https://outlook.office.com:443

Testing connection to outlook.office365.com on port 443

Performing GET on https://outlook.office365.com:443

Testing connection to nexus.microsoftonline-p.com on port 443

Performing GET on https://nexus.microsoftonline-p.com:443

Testing connection to login.microsoftonline.com on port 443

Performing GET on https://login.microsoftonline.com:443

I then went into IIS and turned extended protection off for the EWS directory, it was on as expected.

I then reran the wizard and after the 5 minutes or so, it timed out. Looks like the same error:

exception: 'System.Net.WebSockets.WebSocketException (0x80004005): An internal WebSocket error occurred. Please see the innerException, if present, for more details. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

Bill Seymour 5 Reputation points

2024-05-29T18:34:58.94+00:00

I tried to post a response, but it was apparently removed for some reason. :(

TLS 1.2 is enabled, 1.0 and 1.1 disabled. I've disabled MRS then re-enabled it and restarted IIS. No change. I have basic authentication enabled on the EWS virtual directory (also anonymous and Windows authentication). All appropriate ports are forwarded through my router using NAT and nothing is blocked going out. I've turned off Server Extended Protection. Still no change, the wizard times out. As far as I can see the DNS resolves correctly, though putting the '<GUID>.resource.mailboxmigration.his.msappproxy.net' URL into a browser obviously doesn't display anything. I won't try to post the exception shown in the logfile, since that may be what caused my previous replies to get pulled.

Bill Seymour 5 Reputation points

2024-05-29T18:37:04.35+00:00

Actually, let me try to post that snip by itself:

Microsoft.Online.EME.Hybrid.Agent.Service.EXE Error: 0 : Web socket exception. ConnectionId, '410ac685-e0ff-46ed-9c1d-b1d0e68b9010', exception: 'System.Net.WebSockets.WebSocketException (0x80004005): An internal WebSocket error occurred. Please see the innerException, if present, for more details. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)

at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

Bill Seymour 5 Reputation points

2024-05-29T18:37:59.1233333+00:00

Yeah, I just tried to post only the first couple of lines of the logfile, and it apparently went no nowhere.

Noah Ma-MSFT 1,765 Reputation points Microsoft Vendor

2024-05-30T09:29:40.9866667+00:00

Hi,

Based on the method you have tried, here are some ways you also could try.

As others said above, it could also be related to certificate, please verify that your certificate is properly configured and renewed.

Also, please try to use option for install new agent during HCW setup. The Hybrid Wizard may overwrite the failed agent when you did that.

Hope it helps and if there are any updates, please let me know.

Bill Seymour 5 Reputation points

2024-05-31T01:45:35.9+00:00

How would I test to verify that a Let's Encrypt certificate is valid? The cert is properly configured and has quite some time before it's up for renewal. My Exchange clients have no issues connecting.

I have tried running the Microsoft Remote Compatibility Analyzer, checking for SSL. I get a success with warning:

Analyzing the certificate chains for compatibility problems with versions of Windows.

The test passed with some warnings encountered. Please expand the additional details.

Additional Details

The Microsoft Connectivity Analyzer can only validate the certificate chain using the Root Certificate Update functionality from Windows Update. Your certificate may not be trusted on Windows if the "Update Root Certificates" feature isn't enabled.

Since all my clients connect easily using Outlook on Windows or Android I expect that isn't an issue, but perhaps the Microsoft side thinks it is?

Noah Ma-MSFT 1,765 Reputation points Microsoft Vendor

2024-06-03T10:00:06.57+00:00

Hi,

It also could be related with choosing the wrong MRSProxy URL in the HCW.

Make sure you enable BasicAuth and MRSProxy on EWS as mentioned above.

Then please kindly ensure the Internal/External URL in Exchange Server are set right.

Also, you could try to set the External EWS URL during the install to the appproxy FQDN, then switch it back after HCW to see if it worked.

Hope it could be helpful.

Bill Seymour 5 Reputation points

2024-06-03T17:55:49.6733333+00:00

So in EAC; Virtual Directories; EWS both the internal and external point to EWS/Exchange.asmx as I believe to be correct, using https, and MRSProxy is enabled. The internal and external domains are correct. Authentication is set to Integrated Windows and Basic. When I try to open the site using the FQDN I get a login prompt then this:

You have created a service.

To test this service, you will need to create a client and use it to call the service. You can do this using the svcutil.exe tool from the command line with the following syntax:

svcutil.exe https://mylocaldomain.local:444/EWS/Services.wsdl

Noah Ma-MSFT 1,765 Reputation points Microsoft Vendor

2024-06-06T09:39:36.2266667+00:00

Hi,

Please connect to PowerShell and run the command below:

Test-MigrationServerAvailability -ExchangeRemoteMove: $true -RemoteServer 'dd500728-4fec-405f-a706-a3b245576f1f.resource.mailboxmigration.his.msappproxy.net' -Credentials (Get-Credential -UserName domain\admin)

If the error is same given, check the Hybrid Agent Status (ACTIVE or INACTIVE) by re-running Modern HCW and check it in the GUI. You could refer the Checking the Status of Your Hybrid Agents part of Microsoft Hybrid Agent | Microsoft Learn.

If the Hybrid Agent is ACTIVE, check and confirm with Performance Monitor that you see the requests.

If the request counters (for #of requests) go up on the Agent machine when you do Test-MigrationServerAvailability to the Hybrid Agent, it's likely to be an issue with the on-premises infrastructure, especially proxy and firewall settings.

Please kindly check all the links below and make sure all the requirements have been met:

install requirements

system requirements

port and protocol requirements

For more information, please refer to Troubleshooting Hybrid Migration Endpoints in Classic and Modern Hybrid - Microsoft Community Hub.

Bill Seymour 5 Reputation points

2024-06-06T17:11:26.1966667+00:00

OK, reran the wizard since it's been a while... When I get to the agent page it is shown and active. Performance Monitor is showing no requests at all. Test-MigrationServerAvailability returns this:

RunspaceId : 0f35aab4-9746-4bf1-9100-2bc2127664b1

Result : Failed

Message : The connection to the server

'bcb337755-98b7-4f79-8440-355cf623101c.resource.mailboxmigration.his.msappproxy.net' could not be completed.

ConnectionSettings :

SupportsCutover : False

ErrorDetail : Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the

server 'bcb337755-98b7-4f79-8440-355cf623101c.resource.mailboxmigration.his.msappproxy.net' could not be completed. ---> Microsoft.Exchange.MailboxReplicationService.RemoteTransientException: The call to 'https://bcb337755-98b7-4f79-8440-355cf623101c.resource.mailboxmigration.his.msappproxy.ne t/EWS/mrsproxy.svc' timed out. Error details: The request channel timed out attempting to send after 00:00:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. --> The HTTP request to 'https://bcb337755-98b7-4f79-8440-355cf623101c.resource. mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' has exceeded the allotted timeout of 00:00:00. The time allotted to this operation may have been a portion of a longer timeout. ---> Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The request channel timed out attempting to send after 00:00:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The HTTP request to 'https: //bcb337755-98b7-4f79-8440-355cf623101c.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy. svc' has exceeded the allotted timeout of 00:00:00. The time allotted to this operation may have been a portion of a longer timeout. --- End of inner exception stack trace --- --- End of inner exception stack trace --- at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationServiceFault.<>c__DisplayClas s97_0.<ReconstructAndThrow>b__0() at Microsoft.Exchange.MailboxReplicationService.ExecutionContext.Execute(Action operation) at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationServiceFault.ReconstructAndTh row(String serverName, VersionInformation serverVersion) at Microsoft.Exchange.MailboxReplicationService.WcfClientWithFaultHandling`2.<>c__DisplayClass7 _0.<CallService>b__0() at Microsoft.Exchange.Net.WcfClientBase`1.CallService(Action serviceCall, String context) at Microsoft.Exchange.MailboxReplicationService.WcfClientWithFaultHandling`2.CallService(Action serviceCall, String context) at Microsoft.Exchange.Migration.MigrationExchangeProxyRpcClient.CanConnectToMrsProxy(Fqdn serverName, Guid mbxGuid, NetworkCredential credentials, LocalizedException& error) --- End of inner exception stack trace --- at Microsoft.Exchange.Migration.DataAccessLayer.ExchangeRemoteMoveEndpoint.VerifyConnectivity() at Microsoft.Exchange.Management.Migration.MigrationService.Endpoint.TestMigrationServerAvailab ility.InternalProcessEndpoint(Boolean fromAutoDiscover)

IsValid : True

Identity :

ObjectState : New

Bill Seymour 5 Reputation points

2024-06-06T17:25:23.32+00:00

When I attempt to run the wizard, and replace the default for the server itself with the external FQDN I get a "WinRM service is not enabled" error. When I attempt a "net start winrm" in an elevated shell it says it's already running, so I assume there's an issue there with remote access to the service. My guess is that is normal, since you don't want external access, but I may be wrong there.

Noah Ma-MSFT 1,765 Reputation points Microsoft Vendor

2024-06-11T10:08:10.7233333+00:00

Hi,

Thanks for the detailed response.

I understand you have tried some resolutions, and it seems did not work. However, since the issue has been ongoing for some time now, to avoid taking much of your precious time I advise you please open a service request in Microsoft 365 Admin, and our Online will investigate further, for they have resources to check in the back end if need be.

Your kind understanding is highly appreciated. Thank you for your cooperation.
Sign in to comment

Share via

Exchange 2019 Hybrid Configuration wizard timeout error

2 answers