Partilhar via

Event 1035, MSExchangeTransport - Inbound authentication failed with error UnexpectedExchangeAuthBlob for Receive connector

  • Artigo

Scenario

Ran into an interesting case today where mail flow had quit working in one direction between sites. The environment was on Exchange 2010. Basically, Site A and Site B could not send mail to an exchange server in Site C. They could send to each other, and mail was flowing out from Site C, but inbound mail to Site C was queueing up.

The error in the Application event logs looked as follows:

Log Name:      Application
Source:        MSExchangeTransport
Date:          8/24/2016 1:21:43 PM
Event ID:      1035
Task Category: SmtpReceive
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:     SiteCserver.contoso.com
Description:
Inbound authentication failed with error UnexpectedExchangeAuthBlob for Receive connector Default SiteCServer. The authentication mechanism is ExchangeAuth. The source IP address of the client who tried to authenticate to Microsoft Exchange is [IP of Site A exchange server (or site B)].
Event Xml:
<Event xmlns="https://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSExchangeTransport" />
    <EventID Qualifiers="32772">1035</EventID>
    <Level>3</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2016-08-24T17:21:43.000000000Z" />
    <EventRecordID>10489441</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SiteCserver.contoso.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>UnexpectedExchangeAuthBlob</Data>
    <Data>Default SiteCServer</Data>
    <Data>ExchangeAuth</Data>
    <Data>[IP of Site A exchange server (or Site B)]</Data>
  </EventData>
</Event>

There was no time skew in the environment and no backpressure events were showing up (Event IDs 15001 -15007) in the Application event logs.

Resolution

Normally, I would spend a great deal of time working through Protocol logs and network traces to figure out what is happening here, but luckily for me, I have an engineer named Miguel Ortiz on my team and he has seen this before.

In this particular case, what was happening was, the Kerberos token from Site A and Site B destined for Site C had gotten corrupted. Site C had suffered a severe outage and we believe it was during this time that the Kerberos tokens became corrupted. We went through the following steps to remediate the issue:

    1. On the Site C domain controllers, we restarted the Kerberos Key Distribution Center service (KDC).
    2. On the Site C Exchange server we purged the Kerberos tickets using the KList Purge command from an administrative command prompt
      1. klist purge
    3. Next, we restarted the Netlogon service on the Site C Exchange server.
      1. Net Stop Netlogon && Net Start Netlogon
    4. Then we restarted the Transport Service on the Site C Exchange server.
      1. Net Stop MSExchangeTransport && Net Start MSExchangeTransport
    5. After transport came back up on Site C, we then repeated steps 2 through 4 on the Site A and Site B servers.

This process basically clears out the Kerberos tickets and forces Site A and Site B to re-send their Kerberos request to Site C. In other words, they cleared the corrupted token and requested a new one.

Once this process completed, mail flow returned to normal and no further 1035 events were thrown.

Special Thanks

Thanks to Miguel Ortiz for helping me get through this case quickly and working out the steps to resolve it.

Comments

  • Anonymous
    November 15, 2016
    We ran into this exact issue today.. I found we can skip the netlogon restart (that forces a information store restart)We did "klist -li 0x3e7 purge" to clear the system account tickets then just a transport restart..
    • Anonymous
      November 15, 2016
      Excellent find Scott. I appreciate the update.
  • Anonymous
    January 31, 2017
    I ran to this issue , and i still have it ... the symptoms for this error those it include lost connection with exchange i mean the virtual machine were its hosted "freezes"for a minute and comes back ...Error that fallow 1035:106 general,msexchange common2937 ada acces validation
    • Anonymous
      February 01, 2017
      Arthur, If I am understanding you properly, you are saying that you are having the issue where the error, Inbound authentication failed with error UnexpectedExchangeAuthBlob for Receive connector, is reported? And I assume you followed the steps as laid out in the article above. If that is the case then you need to check to ensure you are not having backpressure events and if you are, troubleshoot those. They will show up as Event IDs 15001 -15007 in the Application Event logs. Also make sure you do not have a time skew issue. Your Exchange servers and your Domain Controllers all need to be on the same time (with the time zone adjustments of course). If they are not, you need to get them into sync. Hopefully this helps. If you are still having the issue and you have a premier account, I think it would be in your best interest to call in and open a ticket.
  • Anonymous
    April 03, 2017
    The comment has been removed
  • Anonymous
    October 26, 2017
    We had a similar issue with the same error message. We actually had to restart the domain controller that was serving the bad kerberos token. Once we forced the Exchange server to connect to a different DC, we were able to follow the steps above to get mail flowing again.