Directory Sync Event 130 Get Single Object failed

Question

Event 130 errors have started appearing in our 2019 Essentials 'Directory Synchronization' log every 30 minutes. I'll post a sample at the bottom.

The referenced 'User Object' is a departed employee. The "User_79888ec3-c003-42f7-8905-f998e769a84c" matches the Object ID of his Shared Mailbox in Azure AD.

We typically delete the local A/D user, restore it in M365 and convert the mailbox to Shared. We did this for the user 2 months ago. Last month, to clean up other errors, I also set the Immutable ID on the Cloud account to "$null".

Synchronization Service Manager shows no errors.
Azure Active Directory Connect Health shows no errors.
Azure AD Connect version (2.0.28.0)

I read this as something on the server is reaching out to contact the cloud version of the user, but failing to find it.

Advice?

Log Name: Application
Source: Directory Synchronization
Date: 2/22/2022 3:49:10 PM
Event ID: 130
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: ServerName.XXX.local
Description:
Get single object failed to get object User_79888ec3-c003-42f7-8905-f998e769a84c. Tracking Id: 00000000-0000-0000-0000-000000000000, Exception: Exception details =>
Type => Microsoft.Online.Coexistence.AzureADObjectNotFoundException
An error occurred. Error Code: 53. Error Description: An internal error has occurred. This operation will be retried during the next synchronization. If the issue persists for more than 24 hours, contact Technical Support. Tracking ID: 2416b89a-feae-4448-b22d-02361cf6c577 Server Name: .
StackTrace =>
at Microsoft.Online.Coexistence.ProvisionHelper.AdminWebServiceFaultHandler(FaultException1 adminwebFault) at Microsoft.Online.Coexistence.ProvisionHelper.InvokeAwsAPI[T](Func1 awsOperation, String opsLabel, IEnumerable`1 operationHeaders)
at Microsoft.Azure.ActiveDirectory.Connector.ProvisioningServiceAdapter.<>c__DisplayClass83_0.<GetSingleObject>b__0()
at Microsoft.Azure.ActiveDirectory.Connector.ProvisioningServiceAdapter.ExecuteWithRetry(String actionName, Action action)
at Microsoft.Azure.ActiveDirectory.Connector.ProvisioningServiceAdapter.GetSingleObject(SyncReference reference, Byte[] singleObjectCookie, Byte[] readbackCookie, Boolean isFullImport, String[] reasons)
at Microsoft.Azure.ActiveDirectory.Connector.GetImportEntriesTask.FullObjectRefetcher.FetchFullAzureADObject(CaseInsensitiveSchema schema, ProvisioningServiceAdapter provisioningServiceAdapter, SyncReference reference, Byte[] originatingReadbackCookie, Boolean isFullImport, String[] reasons)
at Microsoft.Azure.ActiveDirectory.Connector.GetImportEntriesTask.ReadFullRefetchObjects(Byte[] originatingCookie)

InnerException =>
Type => System.ServiceModel.FaultException`1[[Microsoft.Online.Coexistence.Schema.AdminWebServiceFault, Microsoft.Online.Coexistence.Schema.Ex, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]]
An internal error has occurred. This operation will be retried during the next synchronization. If the issue persists for more than 24 hours, contact Technical Support.
StackTrace =>

Server stack trace:
at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.Online.Coexistence.Schema.IProvisioningWebService.GetSingleObject(SyncReference reference, Byte[] singleObjectCookie, Byte[] readbackCookie, Nullable1 isFullReadBack, String[] getSingleObjectReason) at Microsoft.Online.Coexistence.ProvisionHelper.<>c__DisplayClass67_0.<GetSingleObject>b__0() at Microsoft.Online.Coexistence.ProvisionHelper.InvokeAwsApi[T](Func1 awsOperation, String apiAction, String applicationId, Guid clientTrackingId, String clientVersion, String contextId, String dirSyncBuildNumber, String fimBuildNumber, String machineIdentity, IEnumerable1 operationHeaders) at Microsoft.Online.Coexistence.ProvisionHelper.InvokeAwsAPI[T](Func1 awsOperation, String opsLabel, IEnumerable`1 operationHeaders)

Answer 1

@Steve Deal ,

Thanks for reaching out.

It appears that the Azure AD connect sync server was not aware of the modifications made in Azure AD for recovering users' accounts from the deleted bin, hence the sync engine failed to obtain the object. User 79888ec3-c003-42f7-8905-f998e769a84c.

Therefore, would request you to run full Import and then Sync on both the connectors ( Firstly On-premises AD & then AAD connector) on Azure AD connect server as shown below.

If it does not resolve the issue then this might required deeper investigation , I recommend contacting Azure support. If you have a support plan, please open a support ticket; otherwise, please let us know and we will attempt to assist you get one-time free technical support.

-----
Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Answer 2

Thanks sikumars,
The Import/Synch doesn't change things.
We do have a ticket open with Azure Support now / will update here later!

Answer 3

We looked at the issue with an Azure AD team person.

The error suggests that an object still exists on the server side. We reviewed the server and AzureAD but found no object.

Then strangely, I discovered that the error just quit logging YESTERDAY. We only did an operation on another mailbox yesterday, so neither we nor Microsoft understand why it just started working after all this time.

One thing I did learn from tech support - after deleting a local AD object, do TWO full sync cycles to Azure to make sure it gets deleted properly on both sides. Then you can restore on the Azure side from Deleted Users.