Dynamics CRM Troubleshooting: Kerberos
This is Part 2 of my article on improving CRM performance with Kerberos
Series
http://social.technet.microsoft.com/wiki/contents/articles/6450.aspx
Introduction
Kerberos is not the easiest protocol to troubleshoot due to its complex architecture, I would like to share with you a few tips on how to tackle kerberos issues.
To troubleshoot kerberos I'm going to use the following tools:
- Wireshark (we could use Microsoft network monitor but for Kerberos I think Wireshark is simpler to use and illustrate)
- Fiddler (capture HTTP packets)
- ADSI Edit or Command prompt setspn
- Microsoft Kerbtray to list/purge tickets or command prompt klist.
If you read the first article on how to improve performance with kerberos, you understand that when you use custom service accounts you will need unique SPN's configured to allow authentication to succeed. However you could use built-in accounts, the best practice and recommendations from Microsoft is to always use custom service accounts, and it's here where things get a bit more complicated. I'm assuming you have attempted to configure Kerberos based on my first article and you run Fiddler for the first time and let's assume you see this on the first attempt:
http://1.bp.blogspot.com/-Ni0Yu-_3ncE/UH1tDBR_O6I/AAAAAAAAANI/rMQfbRfMf8U/s1600/FiddlerNTLM.png
This ticket is an NTLM ticket, Kerberos failed somewhere and we will try to find out using wireshark.
Open Wireshark start capturing network traffic and on the search field type: Kerberos
http://2.bp.blogspot.com/-JBEfd7q0OdQ/UH1vyC29srI/AAAAAAAAANg/Ptb1ah3JF6g/s1600/kerberospLookup.png
http://3.bp.blogspot.com/-RdGQpTFootU/UH1vlv0NW-I/AAAAAAAAANY/NAqCl0rZ_ZI/s640/kerberosp4Error.png
Wireshark shows a packet with error: KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN highlight this packet and on the below screen expand the Kerberos packet details and you find that the missing DNS is **HTTP/crm.ha **Kerberos is looking for this and can't find it. The next step is to register an SPN under the service account CRM website is running on.
Configuring SPN
a quick way to create visualize SPN's is using the Microsoft command tool Setspn, open the command prompt and type:
**setspn -L DOMAIN\SERVICE_ACCOUNT **
Replace DOMAIN and SERVICE_ACCOUNT with your own domain and service account. This will list any SPN's configured under that account.
If you can't find the SPN listed under that account create the SPN:
setspn -a HTTP/crm.ha SERVICE_ACCOUNT
Run again the first command and check if the SPN has been created.
Another way to create SPN's is using the ADSI Edit GUID, it's easier to manage multiple SPN's for multiple accounts etc..
To install ADSI Edit follow the following instructions:
On the Server manager under Features add new features and navigate to the following options:
http://2.bp.blogspot.com/-8kya1khLA7k/UH18zTxpWeI/AAAAAAAAAN4/iwBw9vhHnmg/s640/KerberosADSI.png
When installed go to: Start menu > Administrative Tools > ADSI Edit
http://1.bp.blogspot.com/-5XtJK79NO_0/UH180BncChI/AAAAAAAAAN8/yKWpdJ4p1jQ/s640/KerberosADSIp2.png
Open ADSI Edit and right click and choose connect to...
On the below screen type the name of the object you want to use, and leave default naming context and type the SERVER_NAME click ok. replace SERVER_NAME with the name of Domain controller in your network.
http://4.bp.blogspot.com/-LuY4pGK7_dM/UH180xO28_I/AAAAAAAAAOE/FuE2nLsrT3I/s1600/KerberosADSIp3.png
http://3.bp.blogspot.com/-Su-QWG3Opao/UH182D7RNtI/AAAAAAAAAOM/K6XCX7IzSx0/s1600/KerberosADSIp4.png
Below Expand the folders until you see the service account you want to configure the SPN's
http://2.bp.blogspot.com/-dRZ2xpCxxts/UH183DmfcCI/AAAAAAAAAOU/ASUPa5LPxHE/s1600/KerberosADSIp5.png
When you find the account, right-click and properties. The below screen appears. Scroll down to servicePrincipalName and click Edit.
http://2.bp.blogspot.com/-gbU8Jzs0GjA/UH184I6YX8I/AAAAAAAAAOc/APkxLllwFx8/s1600/KerberosADSIp6.png
On the below screen you can manage the SPN's and add the needed SPN, you can type HTTP/crm.ha and click OK and Apply and it's done.
Looking for Kerberos tickets
We have configured the missing SPN, the next step is to run wireshark and look for the kerberos ticket.
Highlighting the first packet show us what request's are going through in the process.
http://1.bp.blogspot.com/-fN8qAJr5ZhI/UH1_jgQ5CEI/AAAAAAAAAOo/_zjURj1n_mA/s640/kerberosp1.png
I highlight the first HTTP packet and I already see the first negotiation using Kerberos tickets look for Authorization: Negotiate YIIUM this is a Kerberos ticket.
http://2.bp.blogspot.com/-NxaoWsxStQg/UH6JbXUlUaI/AAAAAAAAAQw/QHLnU5eZRZg/s640/kerberosp2.png
Below I highlight the next in line HTTP packet HTTP/1.1 found and you see a reply with a kerberos ticket again.
http://3.bp.blogspot.com/-WuuqVXOV9lg/UH1_lUBmolI/AAAAAAAAAO4/Cx4wCVN4bw4/s640/kerberosp3.png
It looks like we are now using Kerberos. if we look back at Fiddler now, we should see the following:
http://2.bp.blogspot.com/-LiSa10vudnY/UH56cOX7pOI/AAAAAAAAAQA/cJ-yPExqjVk/s1600/kerberosFiddlerp1.png
http://3.bp.blogspot.com/-ILg1H3aofNY/UH56dOh7v3I/AAAAAAAAAQE/FLw11e3stQs/s1600/kerberosFiddlerp2.png
Also another way to check your kerberos tickets it's using Microsoft Kerbtray or the command line prompt klist.
Usage: Klist.exe [-lh ] [-li ] tickets | tgt | purge
On the command line run:
klist [ENTER] This should show you all kerberos tickets issued to your machine and user account, you may see a number of them.
To purge the tickets do:
klist purge
You can also use a GUID tool called kerbtray you can download kerbtray from here:
- http://www.microsoft.com/en-us/download/details.aspx?id=17657
- http://www.microsoft.com/en-us/download/details.aspx?id=7887
Kerbtray will appear as a green icon on your tray area, right-click this icon and you can purge tickets or list the current tickets. This is the same functionality as the klist command.
http://3.bp.blogspot.com/-dhzNNdt4rrQ/UH57K5bE9SI/AAAAAAAAAQU/8ATomgFpO50/s1600/kerbtrayPurgeTickets.png
http://1.bp.blogspot.com/-1aNyi0apB00/UH57KAZlarI/AAAAAAAAAQQ/aXhZ1tqzucA/s1600/kerbtray.png
Event Logs
You could also look for kerberos authentication issues by enabling Kerberos debug logging on the server, this would generate a number of detailed event logs on your server which can be quite handy:
http://support.microsoft.com/kb/262177
Summary
I hope this was helpful, the article is designed to give an overview of the tools available for troubleshooting kerberos authentication issues and how to use this tools to look for the right information.
References
If you interested in learning more about Kerberos and its architecture, I would recommend you read this article which gives a great overview on this. You may also find loads of high-level articles on this.