Unable to Register Exchange Hybrid Agent

Question

Unable to Register Exchange Hybrid Agent

OscarNHD 25

Hi,

We were originally having some issues migrating mailboxes and the advice I found was to uninstall the hybrid agent and run the Hybrid Configuration Wizard again, so that is what I did, however when trying to run the HCW it fails to register the agent. I have checked all the prerequisites and we seem to meet all the requirements.

Looking at the logs the only error appears to be the bellow:

2024.12.27 01:51:17.046         10332 [Client=UX, fn=SendAsync, Thread=19] START PATCH https://graph.microsoft.com/beta/XXXXXXXX {"onPremisesPublishing":{"applicationServerTimeout":"Default","applicationType":"microsoftapp","externalAuthenticationType":"passthru","externalUrl":"https://XXXX.resource.mailboxmigration.his.msappproxy.net:443/","internalUrl":"https://XXXXXXX:443/","isOnPremPublishingEnabled":true,"isTranslateHostHeaderEnabled":false,"isTranslateLinksInBodyEnabled":false,"singleSignOnSettings":null,"verifiedCustomDomainCertificatesMetadata":null,"verifiedCustomDomainKeyCredential":null,"verifiedCustomDomainPasswordCredential":null}} 
2024.12.27 01:51:17.230         10333 [Client=UX, fn=SendAsync, Thread=19] FINISH Time=184.1ms Results=BadRequest {"error":{"code":"Invalid_App","message":"The application with objectId XXXXXXXX is soft-deleted","innerError":{"date":"2024-12-27T01:51:17","request-id":"XXXXXX","client-request-id":"XXXXXX"}}}

I have used MS Graph explorer to view all the soft deleted apps and this is not one of them. In fact I cannot find any reference to this app ID in our tenancy. I am guessing this is some leftover config from the last time HCW was ran, but I am unsure on how to clear it? My thoughts were maybe a registry Key somewhere.

Any help or suggestions would be appreciated as I have been unable to find any information on this issue.

Shlomi Goldschmiedt 5 Reputation points

2024-12-30T18:19:10.25+00:00

Having the exact same issue on two different servers in a DAG configuration.

Please update if you’ve found a solution.
Mark Ziesemer 11 Reputation points

2024-12-31T21:24:38.0466667+00:00

Just received back from our Microsoft support ticket after nearly a week:

There was a regression on December 16th in Azure which was responsible for deleting the hybrid agent. It was identified and fixed globally by the morning on December 18th. From my understanding, the recommendation to restore it was running the HCW up to the point where it verifies the hybrid agent. But here, you mention that rerunning the HCW is failing with an error about the soft deleted agent.

I won't include the recommended action plan that was provided, because none of it works yet. I'll try to update here once we've found a working solution.

Feel free to try to cross-reference any of your support requests to # 2412250040001836.
Shlomi Goldschmiedt 5 Reputation points

2025-01-06T10:44:35.2766667+00:00

@Mark Ziesemer Any update on your support ticket?

Thanks
Anders Stegelius 0 Reputation points

2025-01-07T08:53:19.88+00:00

Having the exact same issue, any updates would be appreciated.

2 answers

Your answer

Shlomi Goldschmiedt 5 Reputation points

2024-12-30T18:19:10.25+00:00

Having the exact same issue on two different servers in a DAG configuration.

Please update if you’ve found a solution.
Mark Ziesemer 11 Reputation points

2024-12-31T21:24:38.0466667+00:00

Just received back from our Microsoft support ticket after nearly a week:

There was a regression on December 16th in Azure which was responsible for deleting the hybrid agent. It was identified and fixed globally by the morning on December 18th. From my understanding, the recommendation to restore it was running the HCW up to the point where it verifies the hybrid agent. But here, you mention that rerunning the HCW is failing with an error about the soft deleted agent.

I won't include the recommended action plan that was provided, because none of it works yet. I'll try to update here once we've found a working solution.

Feel free to try to cross-reference any of your support requests to # 2412250040001836.
Shlomi Goldschmiedt 5 Reputation points

2025-01-06T10:44:35.2766667+00:00

@Mark Ziesemer Any update on your support ticket?

Thanks
Anders Stegelius 0 Reputation points

2025-01-07T08:53:19.88+00:00

Having the exact same issue, any updates would be appreciated.

Answer 1

OscarNHD 25

Hi All,

Logged a ticket with Microsoft and they were able to restore the missing Application, after which I could remove it properly and re-run the Hybrid agent.

Issue was definitely on the Azure side rather than any local configuration. While waiting for MS support I ended up building a new exchange server and it ended up having the same issue.

Unfortunately this is one only MS support can help with. Very thankful the Tech's I got were able to help though.

Thanks,
Oscar Downing

Answer 2

Anonymous

Hi,@OscarNHD

Thanks for posting your question in the Microsoft Q&A forum.

Based on the information you provided, it appears that the application has been “soft deleted” when using the PATCH operation to update certain configurations.

Here are my suggestions:

1.Access Azure AD for Deleted Applications:-

Go to the Azure portal.
Navigate to Azure Active Directory.
Select "App registrations".
Look for a "Deleted applications" section to see if the application in question is listed there.

2.Clean Up Registry Keys (if applicable):

Sometimes stale entries related to the HCW might reside in the registry.
Open the Registry Editor on the server where HCW was run.
Carefully navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\Hybrid (the actual path may vary slightly depending on your environment).
Look for any keys that might be related to the old configuration and remove them.

If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

OscarNHD 25 Reputation points

2024-12-31T05:44:53.93+00:00

Hi Bruce,

I appreciate the reply.

I have looked at the deleted application section under app registrations and as per my post I also used Graph to view deleted applications, as I have seen discrepancies between the GUI and graph in the past. Neither had an application that matched the relevant ID or looked to be anything related.

Unfortunately, I was not able to find a key called "Hybrid" under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15 on the Exchange server or the server that I was trying to set the agent up on. Also looked around for anything else that could be related to the problem configuration, but wasn't able to find anything obvious.

Thanks,
Oscar
Anonymous

2024-12-31T09:27:48.02+00:00

Hi,@OscarNHD

I'm glad to hear from you.

Have you tried Classic Hybrid?

OscarNHD 25

Hi Bruce,

I just tried Classic topology and it completed with the below messages.

HCW8078 - Migration Endpoint could not be created.  

Microsoft.Exchange.Migration.MigrationServerConnectionFailedException
The connection to the server 'XXXXXXXXXX' could not be completed.

Microsoft.Exchange.MailboxReplicationService.MRSRemoteTransientException
The call to 'XXXXXXXXXXXX' timed out. Error details: The request channel timed out attempting to send after 00:00:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. --> The HTTP request to 'XXXXXXXXXX' has exceeded the allotted timeout of 00:00:00. The time allotted to this operation may have been a portion of a longer timeout.

Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException
The request channel timed out attempting to send after 00:00:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.

Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException
The HTTP request to 'XXXXXXXXXX' has exceeded the allotted timeout of 00:00:00. The time allotted to this operation may have been a portion of a longer timeout.
OriginalFailureType: TimeoutException, WellKnownException: MRSRemote None MRSRemote 
Remote stack trace:
Remote trace:
   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpClientRequestChannel.HttpClientChannelAsyncRequest.SendRequestAsync(Message message, TimeoutHelper timeoutHelper)
   at System.ServiceModel.Channels.RequestChannel.RequestAsync(Message message, TimeSpan timeout)
---------------
Local trace:

----------------------------

Dk-7423 0 Reputation points

2025-01-03T13:38:21.29+00:00

Hi Bruce Jing-MSFT,
changing to classic mode can only be a temporary workaround and even for some customers it is not even a valid temporary solution because there are additional changes that has to be made when you switch to classic. the firewall has to be configured to allow inbound traffic on port 443 and extended protection has to be re-enabled on EWS.
Nonetheless I discovered the same issue "The application with objectId XXXXXXXX is soft-deleted" by serveral customers since December 16th and there is currently no public solution or workaround for this issue. Let me know if there are new ideas and if we should try out something new.
I recommended my customers to open a support case and refering to the existing case of Mark Ziesemer
Anonymous

2025-01-06T09:14:18.5466667+00:00

Hi，@OscarNHD

I'm glad to hear from you.

It is recommended that you have disabled the MRS proxy, enabled it again, and restarted IIS using PowerShell commands.

Microsoft doesn't provide an official solution, but I found third-party documentation, FYI.

https://www.alitajran.com/hcw8078-migration-endpoint-could-not-be-created/

https://www.stellarinfo.com/blog/fix-hcw8078-migration-endpoint-could-not-be-created-error/

Please Note: Since the web site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information.
OscarNHD 25 Reputation points

2025-01-07T01:10:27.5866667+00:00

Thanks Bruce,

Yes, I have seen that documentation and tried disabling and re-enabling MRS proxy more than once as troubleshooting for my original issue, however, it did not help. I just tried it again and re-ran the HCW with classic configuration, however I had the same result.

I also had another look at our firewall, but could not see any related traffic being blocked.

Kind Regards,
Oscar Downing.
Anonymous

2025-01-09T06:53:13.8966667+00:00

Hi，@OscarNHD

Depending on your current complex environment, it is recommended to open a service request to M365 backend team to resolve this issue. thanks for your understanding.
OscarNHD 25 Reputation points

2025-01-10T05:19:05.0233333+00:00

Hi Bruce,

I appreciate your help. I have logged a ticket with our Microsoft partner and will try and get support through them. I will report back with an update once I have a solution.

Kind Regards,
Oscar Downing.
Anonymous

2025-01-10T06:11:43.7766667+00:00

Okay, looking forward to your good news.
MaD 0 Reputation points

2025-02-20T13:45:05.1466667+00:00

Hi, I had same issue with Modern Hybrid agent, not being able to validate in the end/setup. So finally I switch to "Echange Classic Hybrid Typology". Now I can see on-preem users calendar (free/buzy) time., from my online (cloud) mailbox.

Seems there is a lot of issues with the "Modern Hybrid Topology", not recomended to use as I have heard from other reliable sources.

Share via

Unable to Register Exchange Hybrid Agent

2 answers

Your answer