Unknown username or bad password - InetInfo.exe – ADVAPI
A few days back I worked on a very interesting case and when I searched on Internet I found that a lot of people are running in to the same problem which prompted me to write this blog entry.
You will run in to this issue only if you have Exchange/SMTP running on the machine.
You keep on getting these failure audits in your event viewer and you dont konw why they are coming. After some time the account listed in the failure audit just gets locked out and you have to go and unlock the account very frequently. In a lot of cases I saw this was happening in less than 30 seconds.
Event Type: Failure Audit
Event Source: Security
Event Category: Logon/Logoff
Event ID: 529
Date: 8/16/2007
Time: 10:13:24 AM
User: NT AUTHORITY\SYSTEM
Computer: <server>
Description:
Logon Failure:
Reason: Unknown user name or bad password
User Name: <USER>
Domain: <Domain>
Logon Type: 8
Logon Process: Advapi
Authentication Package: Negotiate
Workstation Name: <ServerNAme>
Caller User Name: NETWORK SERVICE
Caller Domain: NT AUTHORITY
Caller Logon ID: (0x0,0x3E4)
Caller Process ID: 2464
Transited Services: -
Source Network Address:
Source Port:
Proceed further only if you see the above text in bold in the event viewer entry. The process id 2464 is determined to be InetInfo. If yes then read further...If no you might be able to use some troubleshooting steps from this blog entry.
The interesting thing to note here is that the Logon Process is ADVAPI. ADVAPI is the DLL for advanced Windows api's and is used in a lot of OS related code. The function on which you can concentrate on for now are LogonUser, LogonUserA, LogonUserExW and LogonUserExA. The code which is generating these events is calling one of these functions for sure.
To find out the code, we can use the Debugging Tools For Windows - www.microsoft.com/whdc/devtools/debugging/default.mspx. Install them on your machine and after install just attach to InetInfo.exe (you can attach to a process by going to WinDBG and then selecting File -> Attach to Process. After that select InetInfo.exe from the list.
NOTE: The moment you do this you have stopped InetInfo and every execution is blocked. In other words what this means that InetInfo is waiting for you to do something and once you are done only then it will be able to proceed.
After that run the following commands one by one.
1) .symfix c:\symcache
2) bp ADVAPI32!LogonUserA "k 100;.time;g"
3) g
(You should be able to connect to Internet from the machine where you are Debugging as WinDBG goes to https://msdl.microsoft.com/downloads/symbols to download the PDB files for the DLL's. You will still be able to debug the process but the function names will not be correct)
After that wait for some time till the problem happens. Once you get the failure Audit in Event Viewer, scroll up in the WinDBG window to see the time when the problem happend and if you see a stack like the following it will just confirm that the failure is coming from exchange.
advapi32!LogonUserA+0x23
exps!CExchAuthContext::HrCheckClearTextLogin+0x1af
exps!CExchAuthContext::HrServerNegotiateClearTextAuth+0xb6
exps!CExchAuthContext::HrServerNegotiateAuth+0x18
exps!CSessionContext::OnEXPSInNegotiate+0x14a
exps!CSessionContext::OnSmtpInCallback+0x2ae
smtpsvc!SMTP_CONNECTION::ProcessPeBlob+0xc1
smtpsvc!SMTP_CONNECTION::ProcessInputBuffer+0x12b
smtpsvc!SMTP_CONNECTION::ProcessReadIO+0xb7
smtpsvc!SMTP_CONNECTION::ProcessClient+0x146
smtpsvc!SmtpCompletion+0x16
isatq!AtqpProcessContext+0x1db
isatq!AtqPoolThread+0x1d1
(You might see the different functions if the symbols have not matched but exps.dll in the stack would be enough to point to this issue)
So why is Exchange doing that. From the call stack we can see that we are just trying to process a SMTP message that came to this server. Your next would be to check the SMTP message and get more details around it
Use Ethereal to capture a trace and after the problem has happened, stop the trace and analyze it using Ethereal
Use the following filter in Ethereal - smtp.rsp.parameter contains "Authentication unsuccessful"
and in the list of the packets, right click on one of them and say follow TCP Stream. Confirm that this failure for the same user (The user name and password are base64 decoded)...
So yes, this is the guy...
220 maine.anr.msu.edu Microsoft ESMTP MAIL Service, Version: 6.0.3790.3959 ready at Tue, 14 Aug 2007 14:46:08 -0400 EHLO CYF-162-WILKINS
250-maine.anr.msu.edu Hello [10.10.144.11] <---This is the guy sending the SMTP message
250-TURN
250-SIZE
250-ETRN
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-8bitmime
250-BINARYMIME
250-CHUNKING
250-VRFY
250-X-EXPS GSSAPI NTLM LOGIN
250-X-EXPS=LOGIN
250-AUTH GSSAPI NTLM LOGIN
250-AUTH=LOGIN
250-X-LINK2STATE
250-XEXCH50
250 OK AUTH LOGIN
334 VXNlcm5hbWU6ZmFydXFp
334 UGFzc3dvcmQ6
535 5.7.3 Authentication unsuccessful.
Use a Base64 Decoder to Decode VXNlcm5hbWU6ZmFydXFp and it should out to be a user name and UGFzc3dvcmQ6 would be the password. In our case VXNlcm5hbWU6ZmFydXFp decodes (Base64 decoder) to "Username:faruqi" . Try to find out what is the IP Address 10.10.144.11 which is listed there and diagnose it further as to if it is an Internal IP or if someone is trying to HACK YOUR MACHINE.
Comments
Anonymous
November 11, 2009
wow, it resolved one of my customer's issues..thanks a ton!Anonymous
April 10, 2011
I have a similar issue where a use account is getting locked --------------------------Event Log from DC------------------------------------ A user account was locked out. Subject: Security ID: SYSTEM Account Name:DC01$ Account Domain:mydomain Logon ID:0x3e7 Account That Was Locked Out: Security ID:MJNabc Account Name:abc Additional Information: Caller Computer Name:Exch2 (Hub Cas Server) --------------------------Event log from Exchange Server----------------------- Further If I read the log from the hub cas server i get the below entry An account failed to log on. Subject: Security ID: NETWORK SERVICE Account Name:SBYPRDHCX2$ Account Domain:abc Logon ID:0x3e4 Logon Type: 3 Account For Which Logon Failed: Security ID: NULL SID Account Name:abc Account Domain: Failure Information: Failure Reason:Account locked out. Status: 0xc0000234 Sub Status:0x0 Process Information: Caller Process ID:0x1674 Caller Process Name:D:E2k7BinEdgeTransport.exe Network Information: Workstation Name:Exch2 (Hub Cas Server) Source Network Address:- Source Port:- Detailed Authentication Information: Logon Process:Advapi Authentication Package:MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Transited Services:- Package Name (NTLM only):- Key Length:0
Unable to determine what is the possible cause which is calling the EdgeTransport.exe. Have you come across this scenario.
Anonymous
June 19, 2014
Fantastic, this helped so muchAnonymous
June 23, 2014
HI, Thank you very much for this :D I had been tackling an issue for months thinking it was remote web, RDP etc but this post showed me exactly what it was and also how the security on emails servers really should be better. So with that I am adding white-lists to all my other clients with SMTP to help protect against this. Thank you Once again.