Intermittent failure of Azure AD Connect directory synchronization

Question

I have an installation of Azure AD Connect on an on-premises server that has been running error-free for several years. It has suddenly begun to intermittently fail during directory synchronization attempts.

When I examine the logs in the 'Synchronization Service Manager (miisclient.exe)', I see failures for both import and export using the AAD connector [screenshot attached 01a.PNG]. But it doesn't always fail. In the screenshot, you can see that the 10:15am directory sync succeeded, but the 10:45am directory sync failed.

If I open 'Azure AD Connect (AzureADConnect.exe)' and attempt to sign in using a cloud-only global administrator account, I get the following failure [screenshot attached 02a.PNG]. However, if I sign-in using the /InteractiveAuth command line switch, I can access the program without errors.

When I examine the application log in Windows Event Viewer for the failed import, I see the following [screenshot attached 03a.PNG].

I can verify the following about my system:

1. The use of TLS1.2 is being enforced for all applications on the on-premises server as suggested in (https://learn.microsoft.com/en-us/azure/active-directory/hybrid/reference-connect-tls-enforcement)
2. No passwords have been changed or expired on either the on-premises sync account or the cloud sync account as suggested in (https://learn.microsoft.com/en-us/troubleshoot/azure/active-directory/directory-sync-stop-register)
3. Both the cloud-only global administrator account and the cloud sync account are EXCLUDED from any conditional access policy that would enforce MFA as indicated on this stie (https://www.alitajran.com/conditional-access-mfa-breaks-azure-ad-connect-synchronization/)
4. All of the root CAs have been updated to the ones listed on this page (https://learn.microsoft.com/en-us/azure/security/fundamentals/tls-certificate-changes#what-is-changing)
5. I even followed the suggestion to add the certificate from login.microsoftonline.com to my list of trusted root certificates (https://social.msdn.microsoft.com/Forums/azure/en-US/cac7a98e-f934-4a27-865c-0a8fc99a6d16/connect-to-azure-ad-unable-to-validate-credentials-could-not-establish-trust-relationship-for-the?forum=WindowsAzureAD)

I should also point out that password synchronization is still working properly. Only directory synchronization fails, but it does not fail 100% of the time. Every 4-5 hours a directory synchronization will be successful. I have no idea what my next troubleshooting steps should be. 2: https://learn.microsoft.com/en-us/azure/active-directory/hybrid/reference-connect-tls-enforcement

Answer 1

In our case, the issue with status no-start-ma and stopped-extension-dll-exception was resolved with the link below. The problem was related to TLS 1.2 and registry entries in the registry editor.

https://learn.microsoft.com/en-us/entra/identity/hybrid/connect/reference-connect-tls-enforcement

Answer 2

Hi Aaron,

Thank you for asking this question on the Microsoft Q&A Platform.

It seems this one is a standard error if the proxy settings are changed or a .Net Framework is updated on the server, please check the config file and update the configuration as per this article and this should help.

unable-communicate-windows-service

===
Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Answer 3

Do you have a staging server you can bring up to see if the issues persists?
If not, I recommend building one and set the new server as your export server

https://learn.microsoft.com/en-us/azure/active-directory/hybrid/how-to-connect-sync-staging-server

If that is not possible, consider rebuilding this server and install the latest version (What version of AADConnect is this?)

If you go this route, be sure to export the current settings for re-import

https://learn.microsoft.com/en-us/azure/active-directory/hybrid/how-to-connect-import-export-config

Answer 4

Sorry, this isn't an answer... it's just the OP adding some more information for context.
As I have stated, sometimes an Azure AD Connect directory synchronization is successful. When it is successful, I get the following events in the Windows event log:

EVENT - Directory Synchronization - EventID: 904
MSAL: False MSAL 4.36.0.0 MSAL.Desktop 4.7.2 Windows Server 2019 Standard [08/21 12:45:07.53 - dea1d68c-418a-4eef-85a0-ed97db37a998] === Token Acquisition (UsernamePasswordRequest) started:
Scopes: https://graph.windows.net/user_impersonation
Authority Host: login.microsoftonline.com
EVENT - Directory Synchronization - EventID: 904
MSAL: False MSAL 4.36.0.0 MSAL.Desktop 4.7.2 Windows Server 2019 Standard [08/21 12:45:07.55 - dea1d68c-418a-4eef-85a0-ed97db37a998] Fetching instance discovery from the network from host login.microsoftonline.com.
EVENT - Directory Synchronization - EventID: 904
MSAL: False MSAL 4.36.0.0 MSAL.Desktop 4.7.2 Windows Server 2019 Standard [08/21 12:45:39.63 - dea1d68c-418a-4eef-85a0-ed97db37a998] Token of type 'urn:oasis:names:tc:SAML:1.0:assertion' acquired from WS-Trust endpoint.
EVENT - Directory Synchronization - EventID: 904
MSAL: False MSAL 4.36.0.0 MSAL.Desktop 4.7.2 Windows Server 2019 Standard [08/21 12:45:40.35 - dea1d68c-418a-4eef-85a0-ed97db37a998] Fetched access token from host login.microsoftonline.com.
EVENT - Directory Synchronization - EventID: 904
MSAL: False MSAL 4.36.0.0 MSAL.Desktop 4.7.2 Windows Server 2019 Standard [08/21 12:45:40.35 - dea1d68c-418a-4eef-85a0-ed97db37a998]
=== Token Acquisition finished successfully:

However, 30 minutes after those events were logged there was a failed directory synchronization. These are the events that get logged when a directory synchronization fails:

EVENT - Directory Synchronization - EventID: 904
MSAL: False MSAL 4.36.0.0 MSAL.Desktop 4.7.2 Windows Server 2019 Standard [08/21 13:15:00.34 - f1c354ef-3594-40de-bfaf-22335486404a] === Token Acquisition (UsernamePasswordRequest) started:
Scopes: https://graph.windows.net/user_impersonation
Authority Host: login.microsoftonline.com
EVENT - Directory Synchronization - EventID: 906
Authenticate-MSAL: unexpected exception [Unspecified-Authentication-Failure] - extendedMessage: An error occurred while sending the request. | The request was aborted: Could not create SSL/TLS secure channel.
webException: The request was aborted: Could not create SSL/TLS secure channel.
STS endpoint: HTTPS://LOGIN.MICROSOFTONLINE.COM/[TenantID].ONMICROSOFT.COM
EVENT - Directory Synchronization - EventID: 906
GetSecurityToken: unable to retrieve a security token for the provisioning web service (AWS). An error occurred while sending the request. | The request was aborted: Could not create SSL/TLS secure channel.. extendedMessage: An error occurred while sending the request. | The request was aborted: Could not create SSL/TLS secure channel.
webException: The request was aborted: Could not create SSL/TLS secure channel.
STS endpoint: HTTPS://LOGIN.MICROSOFTONLINE.COM/[TenantID].ONMICROSOFT.COM
EVENT - Directory Synchronization - EventID: 106
Failed to connect to Windows Azure Active Directory during import: Exception: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.

These two sequences occurred 30 minutes apart and no changes had occurred anywhere locally that could cause the different results.

If I had to wildly speculate, I would say that there are multiple Azure endpoints that I could be connecting to during a directory synchronization and one or more of those endpoints are rejecting my connection attempts while one or more endpoints are accepting my connection attempts. I can't think of any other scenario that might explain what I am experiencing.

Answer 5

Well, I can tell you the one thing that did change, version 2.1.16 came out on 8/2/2022.
Might be worth opening a ticket ...
or if you can rebuild and use the previous version of AADConnect...