Partager via

Appendix D: Kerberos and LDAP Troubleshooting Tips

  • Article
On This Page

Kerberos Troubleshooting Tips Kerberos Troubleshooting Tips
LDAP Troubleshooting Tips LDAP Troubleshooting Tips

Kerberos Troubleshooting Tips

This section will help you troubleshoot Kerberos authentication problems in a heterogeneous UNIX and Microsoft® Windows® operating system environment.

A good place to start is with the following white paper, “Troubleshooting Kerberos Errors,” which provides background and Microsoft-specific guidance and is available at https://www.microsoft.com/technet/prodtechnol/windowsserver2003/technologies/security/tkerberr.mspx.

The following appendices also contain information about Kerberos error messages and troubleshooting tools: Appendix C: “Kerberos and LDAP Error Messages” and Appendix E: “Relevant Windows and UNIX Tools.”

The remainder of this section addresses issues that are common to heterogeneous environments or are UNIX-specific.

See also Volume 2: Chapter 5, “Stabilizing a Custom Solution” on testing the KDC.

Common Problems

When you begin troubleshooting a Kerberos problem, there are a few common trouble-spots that you should check first:

  • Clock skew

  • Encryption types

  • Key tables

  • Domain/realm mapping

  • Name resolution

In addition, there are a couple of common trouble spots that are specific to the solutions discussed in this guide:

  • PAM configuration

  • Credentials cache permissions

In some cases, it will be obvious when troubleshooting which of these, if any, is the cause of the problem. For instance, when there is a clock skew problem, you may see a clock skew error. In other cases, one of these may be the root of the problem but with no obvious indications that this is the case. For example, issues that are the result of name resolution problems often appear with symptoms that seem to have no relation to name resolution. Do not rule out one of these issues just because there is not an obvious pointer to it. The effect of a problem may be subtle.

Clock Skew

Time differences are a common factor when dealing with Kerberos configuration. Kerberos requires that all the computers in the environment have system times within 5 minutes of one another. If computers that a client is attempting to use for either initial authentication (the Kerberos server) or resource access (including both the application server and, in a cross-realm environment, an alternate Kerberos server) have a delta greater than 5 minutes from the client computer or from one another, the Kerberos authentication will fail.

Time Sync Error Messages

Time synchronization problems can be identified when an error similar to “Clock skew too great” is returned, although other more obscure errors may also indicate time synchronization problems. Windows-based computers may generate Event ID 11 from w32time in their event log if the computer is having trouble synchronizing its time.

Common Time Sync Issues

  • Basic time syncing. Is each computer in the environment within 5 minutes of all the others? Note that an environment where the client is 3 minutes slower than the Kerberos server and the application server is 3 minutes faster than the Kerberos server represents a time syncing problem, as the total skew is 6 minutes.

  • Time zone inconsistencies. Clocks may appear to be in sync and still create problems if time zones on either computer are not set correctly. On UNIX-based computers the date -u command can be used to check the absolute time of each computer.

See also Appendix H: “Configuring Time Services for a Heterogeneous UNIX and Windows Environment.”

Encryption Types

Each Kerberos implementation supports a set of encryption types used to encrypt part of the Kerberos traffic. The set of supported encryption types varies slightly by implementation, so in building a heterogeneous environment encryption types that are supported for all involved implementations must be selected. For example, Active Directory® directory service supports the RC4-HMAC encryption type, but native UNIX and older MIT implementations do not. Many UNIX implementations support the SHA1 encryption type, but Active Directory does not. Most implementations support DES-CRC and DES-MD5. Although these encryption types are not as secure as RC4-HMAC and SHA1, they have been selected for this document because of their universal support.

Common Encryption Type Issues

  • Missing entries. The default encryption type entries are missing from the krb5.conf file on the UNIX computers. This can cause the request to be made using the sha1 encryption type, which is not supported by Active Directory.

  • Service key table problems. A service key table contains an incorrect or incompatible encryption type. This can occur when a key table is created using css_adkadmin without using the DES flag or when a key table is created using ktpass for an environment configured to use rc4-hmac.

Possible Symptoms of an Encryption Type Problem

  • If authentication is failing and a network trace shows a Kerberos preauthentication request sent from the client and another returned by the Active Directory server with no further requests, this can indicate an encryption type problem. (This behavior can be the result of a number of problems, not just encryption type.)

  • For End State 2 on Red Hat, a key table with an rc4-hmac encryption type may cause the native kinit tool to hang or core dump on requesting a TGT based on the key table.

Key Tables

In a Kerberos environment, both a client (a user) and a server (the server side component of an application) must have a key (a password). On an application server, this key is stored in a key table (by default a krb5.keytab file). If the key stored in the key table on the application server does not match the key for this service stored in the Kerberos database, or if the application does not have access to read the key from the table, the application will not be capable of completing its side of the Kerberos transaction.

  • Key table entry not found.

  • Key table I/O error.

Common Key Table Issues

  • By default when key tables are created, they are owned by root and readable only by root. If a key table is created on Windows using ktpass and copied to the UNIX computer, care must be taken to ensure it has the appropriate file permissions. If a Kerberos application runs as an account other than root, the key table permissions must be modified to allow the application to read the table.

  • The key, key version number, and key encryption type stored in the key table must match the data for this service stored in Active Directory. To check the validity of the key, use the kinit tool to attempt to acquire an initial ticket because this service is based on the key stored in the key table.

  • Different operating systems have different default locations for the key table file. For example, the Red Hat default is /etc/krb5.keytab, and the Solaris default is /etc/krb5/krb5.keytab. The path to the key table can be specified in the krb5.conf file.

If in doubt about the validity of the key table, move (rename) the existing one and create a new file. Remember that generating a new key table will change the password of that account and increment the key version number. If the same key table is used on multiple computers, it will have to be redistributed to the other computers as well.

See also the "Encryption Types" and "Name Resolution" sections in this appendix.

Key Table Troubleshooting Tools

  • kinit. The primary tool used for checking service tables is kinit. The syntax of the command may vary for different versions of kinit and on different platforms, but it typically uses the -k switch to read the key from the key table, instead of prompting the user for the password, and the optional -t switch to specify the name and location of the key table.

  • klist. The klist tool can be used to display the contents of the key table. The syntax of the command may vary for different versions of klist and on different platforms, but it typically uses the -k switch to display the key table contents instead of the contents of the user's credentials cache. The -t switch to specify the name and location of the key table and the -e switch to display the encryption type of the stored key may also be used.

  • ktutil. The ktutil tool is used to manage key tables. It can also be used to list the contents of a key table although it does not display the key encryption type.

For more information about troubleshooting tools, see Appendix E, “Relevant Windows and UNIX Tools.”

Domain/REALM Mapping

To access Kerberized services, the client computer must be capable of resolving the DNS domain of the target computer to the correct Kerberos REALM. This becomes an issue when the DNS domain name does not match the Kerberos REALM name. Because mapping does not become an issue until the client computer tries to access a service, domain to REALM mapping problems do not affect initial ticket requests (TGTs). When mapping problems exist, service ticket requests may fail or access to Kerberized services may fail. With Active Directory, the REALM name is always the uppercase equivalent of the DNS domain name.

Look in your krb5.conf file to see if the [realms] section and the [domain_realm] section are correct for your environment.

See Volume 2: Chapter 4, “Developing a Custom Solution” for more information on the krb5.conf file.

Name Resolution

Problems with Kerberos are often related to name resolution or Domain Name System (DNS) problems. Active Directory domain controllers, Windows clients, UNIX clients, and application servers must all have a shared understanding of the correct host names and IP addresses for each computer within the environment. DNS is the typical way of computers doing name resolution; however, this might be combined with hosts files, LDAP queries, or other means. DNS will be the focus of this section. Configuration problems with DNS can be subtle but still affect the functionality of Kerberos.

Investigate DNS issues if you are experiencing error messages similar to those listed as follows:

  • Host name cannot be canonicalized.

  • Incorrect net address.

  • Server not found in Kerberos database.

  • Cannot contact KDC for requested realm.

Common DNS Issues

  • DNS problems are often encountered only during a service ticket request after a successful TGT request. If a client can successfully authenticate initially but is then unable to acquire a service ticket or access services, then DNS problems are the likely cause.

  • The error “Server not found in Kerberos database” is common and can be misleading because it often appears when the service principal is not missing. The error can be caused by domain/realm mapping problems or it can be the result of a DNS problem where the service principal name is not being built correctly. Server logs and network traces can be used to determine what service principal is actually being requested.

  • Kerberos recognizes short host names as different from long host names. For example, problems may occur if a client computer knows an application server as appserver1.example.com, but the Kerberos server knows the same computer as appserver1. Check that each host in the environment knows the others by using a consistent naming pattern.

  • Kerberos is case sensitive. Problems can occur in an environment using host names with mixed case. In the world of Kerberos, appserver1.EXAMPLE.COM and appserver1.example.com are not the same. Check that DNS resolves host names with consistent case.

  • Kerberos relies on the presence of both forward and reverse lookup entries in DNS. Check that the host name of each computer can be resolved to its IP address and that its IP address can be resolved to its host name.

  • DNS domain name ambiguities in a multidomain environment can result in subtle DNS issues. Check that each computer knows the others using the same domain name. Avoiding the use of short host names is particularly important in a multidomain environment.

  • Look carefully at the configuration of any multihomed hosts. You might need to perform network traces to determine which interfaces and what names are being used in requests to or from computers with multiple network cards.

DNS Troubleshooting Tools

  • The nslookup tool can be used to validate DNS configuration, checking for host name and IP address mismatches. Use nslookup on the client, Kerberos server, and application server to confirm that each computer in the environment can resolve the other computers by both host name and IP address.

    Note   Some implementations of nslookup may use only DNS servers for name resolution while others may also check files, LDAP, or other configured name resolver sources.

  • The ping tool can help confirm that each computer can contact the others using long name (appserver.example.com), short name (appserver), and IP address.

  • Subtle DNS configuration problems that cannot be found with ping and nslookup can often be found with tools using the getservbyaddr and getservbyname functions.

  • The traceroute (tracert on Windows) tool can help diagnose networking issues between the clients and the DNS server.

  • The pathping tool on Windows can also help diagnose network and latency issues between the clients and the DNS server.

  • The dnslist Windows tool may be helpful in diagnosing DNS errors or performing bulk DNS lookups.

  • Ethereal (https://www.ethereal.com/) is a network protocol analyzer that can be used to capture and analyze Kerberos traffic.

  • The netdiag.exe tool may also be capable of gleaning useful information. This tool is included in the Windows Server 2003 support tools.

  • The Certified Security Solutions gettkt tool can be used to manually request a service ticket for any service, which can be helpful when initial ticket requests succeed but logon or application access fails.

See also Appendix E: “Relevant Windows and UNIX Tools” for more information.

PAM Configuration

The entries in the PAM configuration files can be a common source of problems. Incorrect PAM configuration can lead to loss of access to the host, so caution should be used when configuring or troubleshooting. Common PAM configuration issues include:

  • Incorrect configuration of the control_flag. For instance, use of required instead of sufficient, can cause logon failures and, potentially, total loss of access to the host.

  • If the "use_first_pass" option is missing from the PAM configuration, behavior at logon may be unexpected or confusing.

See the operating system man pages for more information.

Credentials Cache Permissions

For open source End State 2 on Solaris, the permissions on the credentials cache acquired for the LDAP proxy users (/var/tmp/proxycreds) must be readable by all users and writable by owner (644). If the permissions are too restricted (for instance, 640), attempts to log on using ssh may fail. Logon using other access methods (console logon, for instance) may succeed but then requests for group membership or other attributes may fail. For instance, the "Client not found in Kerberos database" error might appear at the command line or in the UNIX syslog, or a network trace may show the GSS-API equivalent code for this error (KRB5KDC_ERR_C_PRINCIPAL_UNKNOWN).

For open source End State 2 on Red Hat, the permissions on the LDAP proxy credentials cache should be readable by owner and group and writable by owner (640).

Service Principal Name (SPN) Errors and Duplicates

If the computer or service accounts have incorrect SPNs associated with them, attempts to acquire a service ticket for that SPN will fail. Look at the LDAP attribute servicePrincipalName of the account in question to see the SPNs associated with the account. This can be done with the ADSI Edit tool or a similar tool (see Appendix E: “Relevant Windows and UNIX Tools”).

Duplicate SPNs can also cause either failure or possibly intermittent failure. For details see “Event ID 11 in the system log of domain controllers” at
https://support.microsoft.com/default.aspx?scid=kb;EN-US;321044.

Error Messages

Following are some Kerberos-related error messages and their potential causes and solutions. Errors associated with Kerberos request failures may appear at the UNIX command line, in the UNIX syslog, in the Active Directory event log, and/or in a network trace. Often, the same or similar error message will be seen in more than one place.

Although we have indicated as follows a specific location for each error message, you may find the same error or similar error message will appear elsewhere caused by the same problem.

UNIX Command-Line Error Messages

No credentials cache found when initializing cache

Application/Function: Message appearing at the command line while trying to execute css_adkadmin.

Potential Cause and Solution: Can indicate that the credentials cache environment variable is set incorrectly. Check the setting for the KRB5CCNAME variable. If it is set, clear it (remove the entire variable—not set the variable to null) and try again.

Set password for principal failed: Authentication error
Failed to add entry to key table

Application/Function: Message appearing at the command line or in the css_adkadmin interface while trying to execute the css_adkadmin ktadd command to add an entry to the key table.

Potential Cause and Solution: Can indicate that principal name specified to be added to the key table does not exist in the Active Directory database.

Client not found in Kerberos database

Application/Function: Anything that makes an initial ticket request.

Potential Causes and Solution: The account for the user name being requested doesn't exist in Active Directory or is incorrect in Active Directory. This could also indicate a DNS problem.

Server not found in Kerberos database

Application/Function: Anything that makes a service ticket request.

Potential Causes and Solution: The account for the service principal name being requested doesn't exist in Active Directory or is incorrect in Active Directory. This could also indicate a DNS problem.

Cannot resolve network address for KDC in requested realm while getting initial credentials

Application/Function: Anything that makes an initial ticket request.

Potential Cause and Solution: This could indicate that the KDC entry in krb5.conf is misconfigured or that there is a DNS problem.

Cannot establish a session with the Kerberos administrative server for realm EXAMPLE.COM. Cannot resolve network address for KDC in requested realm.

Application/Function: Password change request with kpasswd using the native Solaris 9 kpasswd tool.

Potential Cause and Solution: This could indicate that the KDC entry in krb5.conf is misconfigured or that there is a DNS problem.

Unable to get host-based service name for realm EXAMPLE.COM

Application/Function: Password change request with kpasswd using the native Solaris 9 kpasswd tool.

Potential Cause and Solution: Can indicate that the admin_server setting in krb5.conf is missing or incorrect. This may not appear if the admin_server entry exists with an incorrect host name for the admin server. This could also indicate that the default_realm setting in krb5.conf is incorrect.

Cannot establish a session with the Kerberos administrative server for realm EXAMPLE.COM. Client/server realm mismatch in initial ticket request.

Application/Function: Password change request with the native Solaris 9 kpasswd tool.

Potential Cause and Solution: Can indicate that the kpasswd_protocol setting in krb5.conf is missing or incorrect.

Cannot establish a session with the Kerberos administrative server for realm EXAMPLE.COM. Preauthentication failed.

Application/Function: Password change request with kpasswd using the native Solaris 9 kpasswd tool.

Potential Cause and Solution: Can indicate that the incorrect old password was entered for the user.

Preauthentication failed getting initial ticket

Application/Function: Password change request with kpasswd using the native Red Hat 9 and open source kpasswd tool.

Potential Cause and Solution: Can indicate that the incorrect old password was entered for the user.

Preauthentication failed while getting initial credentials

Application/Function: Initial ticket request with kinit.

Potential Cause and Solution: Can indicate that the incorrect password was entered for the user.

Cannot find KDC for requested realm while getting initial credentials

Application/Function: Initial ticket request with kinit using the -k switch to request an initial ticket based on the key stored in a key table.

Potential Cause and Solution: Under different circumstances, this error generally indicates that there is a DNS problem. However, with this specific usage of kinit, it can indicate that the key in the key table doesn't match the key for this principal in the Active Directory database.

Password has expired while getting initial credentials

Application/Function: Anything that makes an initial ticket request.

Potential Cause and Solution: Indicates that the user's password is expired or set to require password change.

Requested effective lifetime is negative or too short while getting initial credentials

Application/Function: Anything that makes an initial ticket request.

Potential Cause and Solution: Can indicate a clock skew problem. Sync the clocks between the UNIX client and the Active Directory server and try again.

No more memory to allocate (in credentials cache code) while retrieving principal name

Application/Function: klist

Potential Cause and Solution: Can occur when klist is executed specifying a key table without using the -k flag. This causes klist to try and interpret the key table as a credentials cache. Try again specifying the -k switch:

klist –k /etc/krb5/krb5.keytab

No credentials cache file found while setting cache flags (ticket cache /tmp/filename)

Application/Function: klist

Potential Cause and Solution: Can occur when klist is executed for a specified credentials cache and the credentials cache file doesn't exist.

Unsupported credentials cache format version number while setting cache flags (ticket cache /tmp/filename)

Application/Function: klist

Potential Cause and Solution: Can occur when klist is executed for a specified credentials cache and the credentials cache file isn't valid or is in a format that is not supported by the version of klist.

Clients’ credentials have been revoked while getting initial credentials

Application/Function: kinit

Potential Causes and Solution: Can indicate that the user's account is locked or expired (account expired, not password expired).

UNIX System Log File (syslog) Error Messages

CROND[11772]: GSSAPI Error: The context has expired (No error)

Application/Function: Message appearing in syslog related to Kerberos authentication for the LDAP authorization connection to the Active Directory server.

Potential Cause and Solution: The Kerberos credential used to make the LDAP connection to the Active Directory server has expired and has not or could not be renewed. Confirm that the cron job to acquire the credential for the proxy/service user is correct. Confirm that the key table containing the stored key for the proxy/service user is correct. Attempt to manually acquire a credential for the proxy/service user using this command (where /etc/proxy.keytab is the key table containing the key for the proxy user and proxy/service is the name of the proxy user):

/usr/bin/kinit -k -t /etc/proxy.keytab proxy/service

(Only applicable to 2B open source solutions)

GSSAPI Error: Miscellaneous failure (Credentials cache permissions incorrect)

Application/Function: Error message appearing in the UNIX syslog upon logon attempt with End State 2 open source solution.

Potential Cause and Solution: Can indicate the permissions on the credentials cache for the LDAP proxy user (/var/tmp/proxycreds) are incorrect.

/usr/dt/bin/ttsession[541]: [ID 848021 daemon.error] _Tt_iceauth::make_auth_cookie(): timeout in locking authority file ' /home/testuser01 /.TTauthority'

Application/Function: Error message appearing in the UNIX syslog upon GUI logon attempt with End State 2. Logon attempt fails.

Potential Cause and Solution: This can indicate that the permission or ownership on the user's home directory is wrong.

pam_krb5: error reading keys for host/ hostname.example.com from /etc/krb5/krb5.keytab: Key version number for principal in key table is incorrect

Application/Function: Logon attempt using pam_krb5.

Potential Causes and Solution: Can indicate that the key for the computer account (host/hostname principal) in Active Directory doesn't match the key stored in the key table. Delete or name off the krb5.keytab and generate a new one.

PAM-KRB5 (auth): krb5_verify_init_creds failed: Key version number for principal in key table is incorrect

Application/Function: Logon attempt using pam_krb5.

Potential Causes and Solution: For native Solaris End States 1 and 2, this can indicate that the key for the computer account (host/hostname principal) in Active Directory doesn't match the key stored in the key table. Delete or name off the krb5.keytab and generate a new one.

PAM-KRB5 (auth): krb5_verify_init_creds failed: Unknown code 2

Application/Function: Logon attempt using pam_krb5.

Potential Causes and Solution: For native Solaris End States 1 and 2, this can indicate that the key table is missing or damaged. Delete or name off the krb5.keytab, if it exists, and generate a new one.

pam_krb5: unable to determine uid/gid for user

Application/Function: Logon attempt using pam_krb5.

Potential Causes and Solution: The account for the user name being requested doesn't exist in Active Directory or is incorrect in Active Directory or the Active Directory database could not be queried for this information (due, for instance, to a DNS problem).

pam_krb5: authenticate error: Clients credentials have been revoked (-1765328366)

Application/Function: Logon attempt using pam_krb5

Potential Causes and Solution: Can indicate that the user's account is locked or expired (account expired, not password expired).

pam_krb5: authentication fails for ` testuser01'
pam_krb5: pam_sm_authenticate returning 7 (Authentication failure)

Application/Function: Logon attempt using pam_krb5

Potential Causes and Solution: These messages can be seen in conjunction with other failure messages such as "credentials have been revoked" and "preauthentication failed." In this case, the accompanying message is the most significant. Check the user account settings.

Network Trace Error Messages

One of the best methods for investigating Kerberos errors using network traces is to get two traces: one showing a situation where the action or a similar action is being performed successfully and the second should be the situation you are investigating that is failing. Careful examination of the differences between the Kerberos packets will usually give insight into the problem. A network protocol analyzer such as Ethereal is very helpful in this case for decoding the Kerberos packets.

Kerberos errors that appear during a network trace are the GSS-API base error codes instead of the English translation of these codes. See Appendix C, “Kerberos and LDAP Error Messages” for error codes.

Note   When the solution is configured to do Kerberos for LDAP (Solaris and Red Hat End State 2 open source solutions), a network trace of a connection will show the binddn from the ldap.conf or null if no binddn is set in ldap.conf. This binddn is not relevant and does not reflect the user that is actually doing the bind.

Windows Command-Line Error Messages

Very few tools related to this solution are used at the command line in Windows.

DsCrackNames returned 0x2 in the name entry for host_hostname

Application/Function: Attempt to use ktpass to map a service principal name to an Active Directory user name and generate a key table.

Potential Causes and Solution: Can indicate that the user account specified (host_hostname in this example) does not exist.

Windows Event Log Error Messages

See “Troubleshooting Kerberos Errors” at
https://www.microsoft.com/technet/prodtechnol/windowsserver2003/technologies/security/tkerberr.mspx.

Error Behaviors

Some errors may occur with no error message provided to assist in troubleshooting.

User is provided with a message that the user's password must be changed , but the user is allowed to log on without changing the password.

Application/Function: Logon attempt using pam_krb5.

Potential Causes and Solution: This can indicate that the admin_server entry in the krb5.conf file is missing or incorrect.

Troubleshooting

The following are some actions you can take when troubleshooting Kerberos issues. Some actions may be more difficult to perform in your environment than others. For instance, to enable Active Directory logging, you must restart the Active Directory server after configuring the registry. This may not be practical in your environment. Start with actions that are quick and easy, such as using the UNIX Kerberos kinit, klist, and kpasswd tools, before attempting to enable extended logging or debugging.

  • Use kinit to acquire an initial credential for the UNIX user defined in Active Directory:

    kinit testuser01

    After acquiring an initial credential for the test user using kinit, use klist with the –e switch to confirm that a credential has been stored in an accessible credentials cache and that the encryption type for the credential is correct:

    klist –e

    If this succeeds, you have confirmed that:

    • The UNIX-based computer can communicate with the Active Directory server.

    • The krb5.conf file is correctly configured for Kerberos authentication against the Active Directory server.

    • The UNIX user is correctly defined for Kerberos authentication in Active Directory.

    • The encryption types defined in the krb5.conf for initial ticket requests are correct for interoperating with Active Directory.

    • The clocks are in sync between the UNIX-based computer and the Active Directory server.

Note This test does not necessarily confirm that DNS is configured correctly. Subtle DNS problems may not become apparent until a service ticket request is made.

  • Use klist with the –k and –e switches to confirm that the key table for the standard computer account has been created and contains a key with the correct encryption type:

    klist –ke

  • Use kinit to acquire an initial credential for the computer account (host/hostname principal) stored in the standard key table:

    kinit –k host/hostname

    If this succeeds, you have confirmed that the key for this computer account stored in the key table matches the key stored in Active Directory.

Note This test does not confirm that a service ticket request for this computer account will succeed.

  • Use a tool, such as the gettkt tool from Certified Security Solutions (www.css-security.com), to acquire a service ticket for the computer account (host/hostname principal) in Active Directory:

    gettkt –s host/hostname getsrvtkt

Note Before executing this test, you must first acquire an initial credential (TGT), using kinit, for the user executing the test.

If this succeeds, you have confirmed that:

  - The UNIX-based computer account is correctly defined in Active Directory.

  - DNS is correctly configured in the environment.

  - The encryption types defined in the krb5.conf for service ticket requests are correct for interoperating with Active Directory.

Note This test does not confirm that the key table containing the key for this computer account on the UNIX-based computer is correct.

  • Use kpasswd to change the password of a UNIX user defined in Active Directory:

    kpasswd testuser01

    If this succeeds, you have confirmed that:

    • The password change settings in the krb5.conf file (kpasswd_server and kpasswd_protocol) are configured correctly for password changes against Active Directory.

    • For some solutions and some versions of kpasswd, the administration server setting (admin_server) in the krb5.conf file is configured correctly. (For instance, the open source kpasswd tool does not make use of this setting.)

  • Use an administration tool, such as the css_adkadmin tool from Certified Security Solutions (www.css-security.com), to make an administration data request to Active Directory:

    css_adkadmin –p adminuser1 –q "getprincs"

    The getprincs command outputs information about all the users in the Active Directory database to the screen. If your database is large, you may prefer to use the getprinc command and specify a user name to retrieve:

    css_adkadmin –p adminuser1 –q "getprinc testuser01"

    If this succeeds, you have confirmed that:

    • A service ticket for the ldap/hostname principal can successfully be acquired.

    • DNS is correctly configured in the environment (because a service ticket can successfully be acquired—see earlier note about using gettkt).

    • The encryption types defined in the krb5.conf for service ticket requests are correct for interoperating with Active Directory.

    • LDAP read requests against Active Directory are succeeding.

Note The standard Kerberos kadmin tool is not compatible with Active Directory and cannot be used for this test.

  • Use Ethereal to trace packets sent from the UNIX client to the Active Directory server and review the KRB5 or LDAP packets.

  • Enable extended logging on Active Directory server and review the System event log.

  • Enable debug mode, if available, on pam_krb5. The CSS pam_krb5 supports the debug=true flag in /etc/pam.conf. For example:

    login    auth sufficient        pam_krb5.so use_first_pass debug=true

  • Enable auditing of failed logons on the Active Directory domain controller. In Windows Server 2003, successful logons are audited by default. Auditing is set in Group Policy.

Note See Appendix E: “Relevant Windows and UNIX Tools” for more information about troubleshooting tools.

Note For open source solutions, each computer may have more than one set of standard Kerberos client tools, such as kinit and klist, installed. The native tools may not support the encryption types defined in the krb5.conf. For best accuracy in troubleshooting pam_krb5 problems with the open source solutions, use the open source tools.

Using pam_krb5 Debugging

Enabling debugging on the pam_krb5 library in the PAM configuration can sometimes help to troubleshoot difficult problems. When debug is enabled, debug output is sent to the system log (syslog) file.

When interpreting pam_krb5 debug output, look for messages similar to those identified in the “UNIX Command-Line Error Messages” section. Also look for references to the key table or, for End State 2, the proxy LDAP user.

Debug error messages are sometimes very clear and sometimes misleading. For instance, the following straightforward debug error message indicates that the key table containing the computer account (host/hostname principal) for the UNIX-based computer is missing:

Note This command is shown on multiple lines for better readability; enter them as a single line.

Dec 12 15:28:02 server01 login: [ID 467052 auth.debug] 
pam_krb5: TGT not verified because keytab file 
/etc/krb5/krb5.keytab doesn't exist

However, the following set of error messages can, among other things, indicate either that the user attempting to log on doesn't have an account in Active Directory or that there is a DNS problem in the environment. The former is straightforward from looking at the output but the latter is not at all obvious.

Note   Some parts of the following code snippet have been displayed in multiple lines only for better readability. These should be entered in a single line.

Dec 12 15:28:02 server01 login: [ID 467052 auth.crit] 
pam_krb5: unable to determine uid/gid for user
Dec 12 15:28:02 server01 login: [ID 467052 auth.info] 
pam_krb5: authentication fails for `testuser01'
Dec 12 15:28:02 server01 login: [ID 467052 auth.debug] 
pam_krb5: pam_sm_authenticate returning 13 
(No account present for user)
Dec 12 15:28:07 server01 login: [ID 219349 auth.debug] 
pam_unix_auth: user testuser01 not found

For the native End State 2 solution, it may be difficult to determine whether references in the syslog are because of the user attempting to log on or the proxy user making LDAP requests through Kerberos.

For example, the following messages make no reference to the credentials cache to which they refer but in this case are for the proxy user (the first indicates that the /var/tmp/proxycreds file does not exist and the second indicates that the file exists but has the wrong permissions):

Note   Some parts of the following code snippet have been displayed in multiple lines only for better readability. These should be entered in a single line.

Dec 12 15:30:04 server01 login: [ID 702911 auth.notice] GSSAPI 
Error: Miscellaneous failure (No credentials cache found)
Dec 12 15:32:27 server01 mail[468]: [ID 702911 auth.notice] GSSAPI 
Error: Miscellaneous failure (Credentials cache permissions incorrect)

Debug output varies depending on the operation being attempted and the pam_krb5 library in use.

The syslog file must be configured to capture debug data in order for the pam_krb5 debug data to be written to the log. The syslog is configured for debugging with a line similar to the following in the /etc/syslog.conf file (the name of the log file varies by platform and is user-configurable):

*.debug         /var/adm/messages

To enable debugging on pam_krb5 for native Solaris, add "debug" to the options at the end of any auth setting for pam_krb5 in the /etc/pam.conf file (the entry can be added before or after the use_first_pass entry). For example:

other  auth sufficient  pam_krb5.so use_first_pass debug

To enable debugging for pam_krb5 for the open source solution on Solaris, add "debug=true" to the options at the end of any auth setting for pam_krb5 in the /etc/pam.conf file (the entry can be added before or after the use_first_pass entry). For example:

other  auth sufficient  pam_krb5.so use_first_pass debug=true

To enable debugging for pam_krb5 for the native and open source solutions on Red Hat, add "debug=true" at the end of the pam_krb5 setting in the /etc/pam.d/system-auth file. For example:

auth  sufficient  /lib/security/$ISA/pam_krb5.so debug=true

Warning Enabling debugging for pam_krb5 can significantly delay logon and logout operations.

Sample Debug Output for Open Source and Native Red Hat pam_krb5 for a Successful Logon

Note   Some parts of the following code snippet have been displayed in multiple lines only for better readability. These should be entered in a single line.

Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5:
get_config() called
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5:
Creating a ticket with addresses
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
krb4_convert false
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
password-changing banner set to `Kerberos 5'
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
ccache directory set to `/tmp'
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
making tickets forwardable
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
setting initial timeout to 1
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
setting maximum timeout to 30
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
will only attempt to authenticate users when UID >= 0
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
making tickets proxiable
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
setting renewable lifetime to 36000
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
required_tgs set to `host/server01.example.com'
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
setting ticket lifetime to 36000
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
setting timeout shift to 2
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
use_authtok false
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
user_check true
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
validate true
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
warn_period 604800
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
pam_sm_authenticate() called (prc = Success)
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
default Kerberos realm is `EXAMPLE.COM'
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
pam_get_user returned `testuser01'
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
user is `testuser01'
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
`testuser01' has uid 10004, gid 10000
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
attempting to authenticate `testuser01'
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
pam_krb5_auth_password returned Success
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
validating TGT
Dec 12 14:52:06 server01 login: [ID 467052 auth.info] pam_krb5: 
TGT for testuser01 successfully verified
Dec 12 14:52:06 server01 login: [ID 467052 auth.info] pam_krb5: 
authentication succeeds for `testuser01'
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
credentials saved for `testuser01@EXAMPLE.COM' 
(pam_krb5_testuser01@EXAMPLE.COM_cred_stash)
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
saved return code (0) for later use 
(pam_krb5_testuser01@EXAMPLE.COM_ret_stash)
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
pam_sm_authenticate returning 0 (Success)
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
get_config() called
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
Creating a ticket with addresses
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
krb4_convert false
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
password-changing banner set to `Kerberos 5'
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
ccache directory set to `/tmp'
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
making tickets forwardable
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
setting initial timeout to 1
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
setting maximum timeout to 30
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
will only attempt to authenticate users when UID >= 0
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
making tickets proxiable
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
setting renewable lifetime to 36000
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
required_tgs set to `host/server01.example.com'
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
setting ticket lifetime to 36000
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
setting timeout shift to 2
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
use_authtok false
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
user_check true
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
validate true
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
warn_period 604800
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
pam_sm_setcred() called
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
credentials retrieved
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
KRB5CCNAME=FILE:/tmp/krb5cc_10004_GgaG1a
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
setting ownership on `/tmp/krb5cc_10004_GgaG1a' to 10004/10000
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
setting permissions on `/tmp/krb5cc_10004_GgaG1a' to 0600
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
recovered return code 0 from prior call to pam_sm_authenticate()
Dec 12 14:52:06 server01 login: [ID 467052 auth.debug] pam_krb5: 
pam_sm_setcred returning 0 (Success)
Sample Debug Output for Native Solaris pam_krb5 for a Successful Logon

Note   Some parts of the following code snippet have been displayed in multiple lines only for better readability. These should be entered in a single line.

Dec 12 14:30:21 server01 login: [ID 655841 auth.debug] PAM-KRB5 
(auth): pam_sm_authenticate flags=0
Dec 12 14:30:21 server01 login: [ID 549540 auth.debug] PAM-KRB5 
(auth): attempt_krb5_auth: start: user='sarahd01'
Dec 12 14:30:21 server01 login: [ID 704353 auth.debug] PAM-KRB5 
(auth): Forwardable tickets requested
Dec 12 14:30:21 server01 login: [ID 912857 auth.debug] PAM-KRB5 
(auth): Renewable tickets requested
Dec 12 14:30:21 server01 login: [ID 179272 auth.debug] PAM-KRB5 
(auth): attempt_krb5_auth: krb5_get_init_creds_password returns: 
SUCCESS
Dec 12 14:30:21 server01 login: [ID 833335 auth.debug] PAM-KRB5 
(auth): attempt_krb5_auth returning 0
Dec 12 14:30:21 server01 login: [ID 914654 auth.debug] PAM-KRB5 
(auth): pam_sm_auth finalize ccname env, result =0, env 
='KRB5CCNAME=FILE:/tmp/krb5cc_10004', age = 0, status = 0
Dec 12 14:30:21 server01 login: [ID 525286 auth.debug] PAM-KRB5 
(auth): end: Success
Dec 12 14:30:22 server01 login: [ID 712548 auth.debug] PAM-KRB5 
(acct): exp_warn start: user = 'testuser01'
Dec 12 14:30:22 server01 login: [ID 734734 auth.debug] PAM-KRB5 
(acct): fetch_princ_entry: non-RPCSEC_GSS chpw server, 
can't get princ entry
Dec 12 14:30:22 server01 login: [ID 615945 auth.debug] PAM-KRB5 
(acct): exp_warn: fetch_pr failed 4
Dec 12 14:30:22 server01 login: [ID 748222 auth.debug] PAM-KRB5 
(acct): exp_warn end: err = 4
Dec 12 14:30:22 server01 login: [ID 712902 auth.debug] PAM-KRB5 
(acct): end: Success
Dec 12 14:30:22 server01 login: [ID 629253 auth.debug] PAM-KRB5 
(setcred): start: nowarn = 0, flags = 0x1
Dec 12 14:30:22 server01 login: [ID 586274 auth.debug] PAM-KRB5 
(setcred): kmd auth_status: Success
Dec 12 14:30:22 server01 login: [ID 735350 auth.debug] PAM-KRB5 
(setcred): end: Success
Dec 12 14:30:22 server01 login: [ID 490997 auth.debug] PAM-KRB5 
(auth): krb5_cleanup auth_status = 0

The following references will be helpful in understanding Kerberos errors:

LDAP Troubleshooting Tips

This section will help you troubleshoot LDAP authentication and authorization problems in a heterogeneous UNIX and Microsoft Windows environment. A limited number of tools is available for LDAP troubleshooting. That isn’t to say there is not a wide selection of tools to perform LDAP queries; but for the most part, they perform similar functions. This means that when tracking down issues related to LDAP, you tend to be left with three primary tools:

  • Network traces and a protocol analyzer

  • ldapsearch

  • Debug output

Normally, the first step is to ensure connectivity of the LDAP client to the LDAP server and that a successful bind is happening. A network trace is often the easiest way to positively determine both. However, if TLS/SSL or Kerberos authentication for the LDAP bind is enabled, you won't be able to see the actual LDAP traffic. To see the LDAP traffic, you can turn off TLS/SSL or Kerberos authentication for the LDAP, investigate the use of the ssldump tool (but not when using Kerberos to authenticate the LDAP calls), or try to enable debug output.

Another approach is to create LDAP searches. A useful technique is to create an LDAP search that mimics what you think is happening or is a situation that works (or a user that works). Then create another LDAP search that mimics what is failing or queries a user that is failing. Look at the output for the working and nonworking cases and compare. Additional information about LDAP troubleshooting tools is available in Appendix E: “Relevant Windows and UNIX Tools.”

Common Problems

There are several common problem spots to suspect when troubleshooting LDAP issues and the LDAP solutions described in this guide:

  • LDAP data caching

  • LDAP configuration files

  • Name resolution

  • PAM configuration issues

  • TLS certificates

Examining these potential problem spots is a good place to start troubleshooting.

LDAP Data Caching

The LDAP client and Name Service Caching Daemon (NSCD) may cache information. After making LDAP configuration changes, it is best to restart both the LDAP client and NSCD.

If you are experiencing problems, you should also check that NSCD is running and verify the NSCD configuration.

For information about starting the LDAP client and NSCD, see Volume 2: Chapter 4, “Developing a Custom Solution.”

LDAP Configuration Files

LDAP /etc/ldap.conf Configuration File

For the open source and native Red Hat solutions, check the entries in the /etc/ldap.conf file. See Appendix I: “Sample Configuration Files for Custom Solutions.” In particular, look for these easily missed errors:

  • Confirm that the entries for binddn contain cn=Users in addition to the rest of the distinguished name (also known as the DN). The default /etc/ldap.conf file does not contain this. For example:

    binddn cn=proxyuser,cn=users,dc=example,dc=com

  • When configuring TLS for /etc/ldap.conf, confirm that the uri used is a host name instead of an IP address. The default /etc/ldap.conf contains an IP address but TLS will only work with a host name in this entry. For example:

    uri ldaps://server1.company.com/

  • Confirm that the nss_base entries contain "?sub" instead of the default "?one" at the end of each line. This will cause LDAP searches and other operations to look in all subcategories instead of just one layer deep. For example:

    nss_base_passwd ou=unix,dc=example,dc=com?sub
LDAP /var/ldap/ldap_client_file Configuration File

For the Solaris solution, check the entries in the /var/ldap/ldap_client_file file. See Appendix I: “Sample Configuration Files for Custom Solutions.”

Name Resolution

Logon problems on UNIX-based computers are often related to name resolution or Domain Name System (DNS) problems. DNS is the typical choice for performing name resolution; however, this might be combined with hosts files, LDAP queries, or other means. DNS will be the focus of this section. Active Directory domain controllers, Windows clients, UNIX clients, and application servers must all have a shared understanding of the correct host names and IP addresses for each computer within the environment. Although LDAP is not as sensitive to subtle DNS configuration problems as Kerberos, DNS problems may also affect LDAP functionality.

Common DNS Issues

  • When using TLS, referring to the short name instead of the long name can sometimes cause problems. One source of problems can be the X509 certificate used by the server for SSL. This certificate is bound to a particular name; this name must be the one used when the TLS/SSL channel is established. A similar problem can be experienced when using Kerberos to help secure the LDAP channel.

  • DNS domain name ambiguities in a multidomain environment can result in subtle DNS issues. Check that each computer knows the others using the same domain name. Avoiding the use of short host names is particularly important in a multidomain environment.

DNS Troubleshooting Tools

  • The nslookup tool can be used to validate DNS configuration, checking for host name and IP address mismatches. Use nslookup on the client, the Active Directory server, and, if applicable, the application server to confirm that each computer in the environment can resolve the other computers by both host name and IP address.

  • The ping tool can help confirm that each computer can contact the others using long name (appserver.example.com), short name (appserver), and IP address.

  • Subtle DNS configuration problems that cannot be found with ping and nslookup can often be found with tools using the getservbyaddr and getservbyname functions.

  • Ethereal (https://www.ethereal.com/) is a network protocol analyzer that can be used to capture and analyze traffic.

  • The netdiag.exe tool may also be capable of gleaning useful information. This tool is included in the Windows Server 2003 support tools.

PAM Configuration Issues

The entries in the PAM configuration files can be a common source of problems. Incorrect PAM configuration can lead to loss of access to the host, so caution should be used when configuring or troubleshooting. Common PAM configuration issues include:

  • Incorrect configuration of the control_flag. For instance, use of required instead of sufficient can cause logon failures and, potentially, total loss of access to the host.

  • If the "use_first_pass" option is missing from PAM configuration entries, behavior at logon may be unexpected or confusing.

See the operating system man pages for more information.

TLS Certificates

If you are using TLS to authenticate or protect the LDAP traffic, then the Active Directory server must have an appropriate certificate. The LDAP client must also trust the root certification authority, which issued the certificate to Active Directory.

Problems that may be encountered when using TLS include:

  • A missing certificate on the domain controller.

  • A revoked, expired, or otherwise invalid certificate on the domain controller.

The following document, "Requirements for Domain Controller Certificates from a Third-Party CA," describes the requirements for the certificate used by Active Directory and is available at https://support.microsoft.com/default.aspx?scid=kb;en-us;291010.

These requirements state that the fully qualified domain of the domain controller must appear in one of the following places:

  • The Common Name (CN) in the subject field.

  • DNS entry in the Subject Alternative Name extension.

However, we recommend that you use the FQDN in the subject field. A blank subject field may cause malfunctions on the UNIX LDAP clients.

Autoenrollment

When you add a certification authority to your domain, each of your domain controllers should receive a server certificate through autoenrollment. In some cases, however, this automatic process does not complete correctly and you may not see a certificate on the domain controller.

You can acquire a domain controller certificate by using the Certificates console on each of your domain controllers. If you have not done so already, add the Certificates console to each domain controller.

To add the Certificates console to each Active Directory domain controller

  1. Click Start, click Run, type mmc, and then click OK.

  2. Click File, click Add/Remove Snap-in, and then click Add.

  3. Click Certificates, and then click Add.

  4. Select the Computer account option, click Next, and then click Finish.

  5. Click Close, and then click OK.

In the console tree, expand Certificates (Local Computer) and click Personal. You should see a certificate with the FQDN of your domain controller. If there is no certificate, your first troubleshooting step is to force a Group Policy update by executing the following command on one of your domain controllers:

C:\>gpupdate /force

After the Group Policy update, in the Certificates console, refresh the screen and check for the presence of a certificate again. If there is still no certificate, use the following steps on the CA server to check the certificate template and permissions setting.

To check the certificate template and permissions settings

  1. Open Certification Authority in Administrative Tools.

  2. Expand the root name, and then click Certificate Templates.

  3. Confirm that Domain Controller is among the listed templates.

  4. Right-click Certificate Templates, and then click Manage.

  5. In Certificate Templates, right-click Domain Controller template, and then click Properties.

  6. On the Security tab, confirm that Domain Controllers have Enroll permissions.

If the Domain Controller template is missing, you will need to enable this certificate template. Please refer to the certificate services Help for more information.

If the Enroll permission is not enabled, check the Enroll box to enable it.

Return to your domain controllers, run the gpupdate command again and, in the Certificates console, refresh the screen and check for certificates. If there are still no certificates, confirm that autoenrollment is enabled for the domain.

To confirm that autoenrollment is enabled for the domain

  1. On one of your domain controllers, click Start, click Run, type mmc, and then click OK.

  2. Click File, click Add/Remove Snap-in, and then click Add.

  3. Click Group Policy Object Editor, and then click Add.

  4. In the Group Policy Wizard, click Browse.

  5. Select Default Domain Policy, click OK, and then click Finish.

  6. Click Close on the Add Standalone Snap-in dialog box, and then click OK on the Add/Remove Snap-in dialog box.

  7. In the console tree, expand Default Domain Policy [ServerName.example.com] Policy, Computer Configuration, Windows Settings, and Security Settings.

  8. Click Public Key Policies, and then, in the Object Type window, double-click Autoenrollment Settings.

  9. Confirm that Enroll certificate automatically is selected.

If Enroll certificate automatically is not checked, check it. Then, run the gpupdate command again and, in the Certificates console on the domain controller, refresh the screen and check for certificates. You may need to choose Action from the menu and Refresh to update.

If the certificate still does not appear, refer to the following troubleshooting resources: "Domain controllers are not obtaining a domain controller certificate" and "Clients are unable to obtain certificates through autoenrollment" in "Certificate Templates Troubleshooting" at https://www.microsoft.com/technet/prodtechnol/windowsserver2003/library/ServerHelp/43881ad5-aa6b-4527-ad59-cd2218bd9934.mspx.

For more information about using LDAP and TLS/SSL, see:

Error Messages

Error messages can be very helpful when troubleshooting the solutions described in this guide, but LDAP-specific failures frequently do not provide very helpful error messages.

UNIX Command-Line Error Messages

Unfortunately the LDAP tools rarely give error messages on the command line that are especially useful for troubleshooting LDAP problems. The command-line ldapsearch tools do not use the same configuration files as the LDAP clients that are performing the LDAP connections during logon. This means that they cannot be used to verify the LDAP configuration.

The ldapsearch tool is useful for verifying that you have connectivity to the LDAP server (Active Directory), verifying proxy user or end-user passwords (a successful bind means the password is good), and looking at the returned user attributes. (Refer to the ldaplist entry in Appendix E: “Relevant Windows and UNIX Tools.”)

UNIX System Log File (syslog) Error Messages

Unfortunately, the LDAP tools rarely produce syslog error messages that are especially useful for troubleshooting LDAP problems.

Network Trace Error Messages

One of the best methods for investigating LDAP errors using network traces is to get two traces: one showing a situation where the action or a similar action is being performed successfully and the second should be the situation you are investigating that is failing. Careful examination of the differences between the LDAP packets will usually provide insight into the problem. A network protocol analyzer such as Ethereal is very helpful in this case for decoding the LDAP packets.

When TLS/SSL or Kerberos authentication is enabled for the LDAP connection to Active Directory, a protocol analyzer may not be capable of decrypting the packets and so may not show useful information. You may need to disable TLS/SSL or Kerberos authentication for the LDAP connection in order to troubleshoot problems with authentication through LDAP (End States 3 and 4) or authorization through LDAP (End States 2 and 4).

Troubleshooting

  • For authorization through LDAP, use the UNIX chown command to attempt to change the ownership of a UNIX file to an Active Directory user who does not have a local account:

    # touch /tmp/testfile

chown testuser01 testfile

ls –l /tmp/testfile

The **chown** command should complete without an error, and the output of the **ls** command should show that the user testuser01 is the owner of the file. For example:

<pre IsFakePre="true" xmlns="https://www.w3.org/1999/xhtml">-rw-r--r--   1 testuser01   root      0 Dec 13 11:26 testfile</pre>


If the output shows the user's UID instead of user name (10001 instead of testuser01), the command has not completed successfully.

  • For authorization through LDAP, use the UNIX id and groups commands to attempt to list the user and group information for an Active Directory user who does not have a local account:

    # id testuser01

groups testuser01

These commands should complete without an error, and the output of the commands should show the UID and user name of the Active Directory user and the group IDs and group names of the groups of which the user is a member. If only the ID numbers appear, the command has not completed successfully.

For example, on Red Hat the output of the **id** command should be similar to:

**Note**   This command is shown on multiple lines for better readability; enter them as a single line.

<pre IsFakePre="true" xmlns="https://www.w3.org/1999/xhtml">uid=10001(testuser01) gid=10003(tstgrp04) groups=10003(tstgrp04),

10001(tstgrp02),10002(tstgrp03),10004(tstgrp06)

On Solaris, the output of the **id** command should be similar to:

<pre IsFakePre="true" xmlns="https://www.w3.org/1999/xhtml">uid=10001(testuser01) gid=10003(tstgrp04)</pre>


On Red Hat, the output of the **groups** command should be similar to:

<pre IsFakePre="true" xmlns="https://www.w3.org/1999/xhtml">testuser01 : tstgrp04 tstgrp02 tstgrp03 tstgrp06</pre>


On Solaris, the output of the **groups** command should be similar to:

<pre IsFakePre="true" xmlns="https://www.w3.org/1999/xhtml">tstgrp04 tstgrp02 tstgrp03 tstgrp06</pre>

  • Use the UNIX ldapsearch tool to confirm that Active Directory will correctly return LDAP data to the UNIX-based computer.

    For example, the open source ldapsearch command on Solaris and Red Hat and the native ldapsearch command on Red Hat should return a list of attributes for each object in the ou=unix container that has a "cn" beginning with "test" with the following command:

    Note   This command is shown on multiple lines for better readability; enter them as a single line.

    # ldapsearch –x -h 10.9.8.1 -D cn=proxyuser,cn=users,dc=example,

dc=com -w Password1 –b ou=unix,dc=example,dc=com -s sub '(cn=test*)'

The equivalent of this command for the native Solaris **ldapsearch** tool is:

**Note**   This command is shown on multiple lines for better readability; enter them as a single line.

<pre IsFakePre="true" xmlns="https://www.w3.org/1999/xhtml"># ldapsearch -h 10.9.8.1 -D cn=proxyuser,cn=users,dc=example,dc=com

-w Password1 –b ou=unix,dc=example,dc=com -s sub '(cn=test*)'

If this command succeeds, you have verified that a connection can be made for LDAP to a specific Active Directory server, that a bind can be successfully completed for the specified proxy user, and that the requested set of data can be retrieved. This can be useful in troubleshooting broad connectivity problems such as DNS.

An **ldapsearch** command does NOT validate the LDAP configuration on the UNIX client because all contact information (for example, the server to contact or the proxy user) is specified as part of the command.

The output of this command should be similar to the following, with one block of attributes returned for each "test\*" user:

<pre IsFakePre="true" xmlns="https://www.w3.org/1999/xhtml">version: 2

 

filter: (cn=testuser01)

requesting: ALL

 

testuser01, users, unix, example, com

dn: CN=testuser01,OU=users,OU=unix,DC=example,DC=com objectClass: top objectClass: person objectClass: organizationalPerson objectClass: user cn: testuser01 description: A test user givenName: testuser01 distinguishedName: CN=testuser01,OU=users,OU=unix,DC=example,DC=com instanceType: 4 whenCreated: 20051107165145.0Z whenChanged: 20051213160924.0Z displayName: testuser01 uSNCreated: 14272 uSNChanged: 50132 name: testuser01 objectGUID:: bvKdKKV2vE6y1ajqv1vrOQ== userAccountControl: 512 badPwdCount: 0 codePage: 0 countryCode: 0 badPasswordTime: 127788800531975113 lastLogoff: 0 lastLogon: 127789637647575258 pwdLastSet: 127789637647274829 primaryGroupID: 513 objectSid:: AQUAAAAAAAUVAAAAxJqR/5FHfX/C552eXwQAAA== accountExpires: 9223372036854775807 logonCount: 100 sAMAccountName: testuser01 sAMAccountType: 805306368 userPrincipalName: testuser01@example.com lockoutTime: 0 objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=example,DC=com lastLogonTimestamp: 127784593203378594 msSFU30HomeDirectory: /home/testuser01 msSFU30Name: testuser01 msSFU30UidNumber: 10001 msSFU30GidNumber: 10001 msSFU30LoginShell: /bin/sh msSFU30Password: VySPj4Cn3IbgQ msSFU30NisDomain: example  

search result

search: 2 result: 0 Success  

numResponses: 2

numEntries: 1

  • For authorization through LDAP, use the ldaplist command on Solaris to test the native LDAP configuration files. The following command should return the attributes for each nongroup object (users and computers) starting with "test" found in the base domain name defined in the /etc/ldap/ldap_client_file (ou=unix,dc=example,dc=com). (The "\" is needed to prevent the shell from misinterpreting the "*" wildcard character):

    ldaplist -l passwd test\*

    The output of this command should be similar to that shown earlier for ldapsearch, with one block of attributes returned for each "test*" user.

    The following command should return the attributes for each group object starting with "tst":

    ldaplist -l group tst\*

    The output of this command should be similar to the following, with one block of attributes returned for each "tst*" group:

    dn: CN=tstgrp01,OU=groups,OU=unix,DC=example,DC=com

objectClass: top objectClass: posixGroup cn: tstgrp01 distinguishedName: CN=tstgrp01,OU=groups,OU=unix,DC=example,DC=com instanceType: 4 whenCreated: 20051107165009.0Z whenChanged: 20051107170806.0Z uSNCreated: 14227 uSNChanged: 16431 name: tstgrp01 objectGUID: WPQ?\202L`C´\213ËW^Bv^R objectSid: ^A^E sAMAccountName: tstgrp01 sAMAccountType: 268435456 groupType: -2147483646 objectCategory: CN=Group,CN=Schema,CN=Configuration,DC=example,DC=com memberuid: test03 memberuid: test02 msSFU30Name: tstgrp01 gidnumber: 10000 msSFU30NisDomain: example

  • Use the ldap_cachemgr command on Solaris to check the status of the native LDAP cache manager daemon.

    /usr/lib/ldap/ldap_cachemgr -g

    The output of this command should be similar to the following:

    cachemgr configuration:

server debug level          0 server log file "/var/ldap/cachemgr.log" number of calls to ldapcachemgr       1107   cachemgr cache data statistics: Configuration refresh information:   Previous refresh time: 2005/11/28 07:06:47   Next refresh time:     2005/12/13 13:03:10 Server information:   Previous refresh time: 2005/12/13 11:55:55   Next refresh time:     2005/12/13 12:55:55   server: 10.9.8.1, status: UP Cache data information:   Maximum cache entries:          256   Number of cache entries:          0

  • You can test the configuration of nss_ldap using the getent passwd command. This command should return the user entries from the /etc/passwd file followed by any UNIX user accounts stored in Active Directory.
    Use the getent command for the open source and native solutions on Red Hat and the open source Solaris solution to retrieve a list of valid users or groups.

    For example:

    # getent passwd

getent group

If LDAP is working correctly, both local and Active Directory users and groups should be returned.

You can also use **getent** to retrieve information about a single user:

**getent passwd** *username*

where username is the user name of a user within Active Directory that has UNIX attributes.

For example, the **getent** command for users (the "passwd" option) should produce output similar to the following. (This example shows the standard local users on Red Hat. The standard local users for Solaris may be different.)

<pre IsFakePre="true" xmlns="https://www.w3.org/1999/xhtml">root:x:0:0:root:/root:/bin/bash

bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin adm:x:3:4:adm:/var/adm:/sbin/nologin lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin sync:x:5:0:sync:/sbin:/bin/sync shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown halt:x:7:0:halt:/sbin:/sbin/halt mail:x:8:12:mail:/var/spool/mail:/sbin/nologin news:x:9:13:news:/etc/news: uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin operator:x:11:0:operator:/root:/sbin/nologin games:x:12:100:games:/usr/games:/sbin/nologin gopher:x:13:30:gopher:/var/gopher:/sbin/nologin ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin nobody:x:99:99:Nobody:/:/sbin/nologin rpm:x:37:37::/var/lib/rpm:/bin/bash vcsa:x:69:69:virtual console memory owner:/dev:/sbin/nologin nscd:x:28:28:NSCD Daemon:/:/sbin/nologin sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin rpc:x:32:32:Portmapper RPC user:/:/sbin/nologin rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin mailnull:x:47:47::/var/spool/mqueue:/sbin/nologin smmsp:x:51:51::/var/spool/mqueue:/sbin/nologin pcap:x:77:77::/var/arpwatch:/sbin/nologin xfs:x:43:43:X Font Server:/etc/X11/fs:/sbin/nologin ntp:x:38:38::/etc/ntp:/sbin/nologin gdm:x:42:42::/var/gdm:/sbin/nologin localuser01:x:500:500::/home/localuser01:/bin/csh localuser02:x:501:501::/home /localuser02:/bin/csh localuser03:x:502:502::/home /localuser03:/bin/csh test01:x:10000:10000:testuser01:/home/testuser01:/bin/sh test02:x:10001:10001:testuser02:/home/testuser02:/bin/sh test03:x:10002:10002:testuser03:/home/testuser03:/bin/sh test04:x:10003:10003:testuser04:/home/testuser04:/bin/sh

The **getent** command for groups should produce output similar to the following. (This example shows the standard local groups on Red Hat. The standard local groups for Solaris may be different.)

<pre IsFakePre="true" xmlns="https://www.w3.org/1999/xhtml">root::0:root

other::1: bin::2:root,bin,daemon sys::3:root,bin,sys,adm adm::4:root,adm,daemon uucp::5:root,uucp mail::6:root tty::7:root,adm lp::8:root,lp,adm nuucp::9:root,nuucp staff::10: daemon::12:root,daemon sysadmin::14: smmsp::25:smmsp nobody::60001: noaccess::60002: nogroup::65534: tstgrp01:x:10000:testuser03,testuser02 tstgrp02:x:10001:testuser06,testuser10,testuser05,testuser04 tstgrp03:x:10002:testuser05,testuser07,testuser09, testuser01 tstgrp04:x:10003:testuser08,testuser10,testuser02,testuser01 tstgrp06:x:10004:testuser07,testuser09,testuser04,testuser02 tstgrp07:x:10005:testuser10,testuser03,testuser02

Using LDAP Debugging

Enabling debugging for LDAP can sometimes help to troubleshoot difficult problems. When debug is enabled, debug output is sent to the screen and, optionally, logged to a file. The debugging option for LDAP is probably most useful for determining whether class and attribute mappings are configured correctly. It can also help with host name mismatches with the server certificate and the uri used to contact the server.

To enable general LDAP debugging for the open source solution on Solaris and native and open source solutions on Red Hat, add lines for "debug ####" and, optionally, "logdir /output_dir" at any point in the/etc/ldap.conf file, where #### is the debugging level and /output_dir is the path to a directory where the log data can be written. For example:

debug 65535
logdir /tmp/ldaplog

During a logon operation, debug data is output to the screen and to a file in the directory specified by the logdir setting in the /etc/ldap.conf file. An output file is generated for each process ID (PID).

There are many debugging levels—bit strings corresponding to different kinds of debugging information. Interpreting this data is a challenge. Table D.1 lists four of the log levels and the useful data they provide for logon operations.

Table D.1. Data Provided by Log Levels

Log Level

Data

1

TRACE data.

2

PACKET data: This includes hexadecimal and ASCII renditions of the data in the LDAP packets sent to and received from the Active Directory server. In this data you can see the attributes for which data is being requested and the data returned.

16

BER data.

65535

All data: At this debugging level, TRACE, PACKET, and BER data is returned.

Data that you find in debug output can also be captured using a network trace tool such as Ethereal. It is generally much simpler to analyze data captured in a trace with a packet analyzer than to analyze raw LDAP debug data. However, when a session is secured with TLS/SSL or Kerberos, the packet analyzer may not be capable of decoding the packets. For initial troubleshooting, it may be most helpful to disable TLS or Kerberos authentication for the LDAP connection and use a packet analyzing tool to troubleshoot. If you find it necessary to troubleshoot the problem without disabling TLS or Kerberos or if you are unable to use a network trace tool on your network, the debug output from LDAP may be useful.

In addition to the debug levels that can be set for LDAP in general, it is possible to enable debugging specifically for the pam_ldap module for the native Solaris solution. The syslog file must be configured to capture debug data in order for the pam_ldap debug data to be written to the log. The syslog is configured for debugging with a line similar to the following in the /etc/syslog.conf file (the name of the log file is user-configurable):

*.debug         /var/adm/messages

To enable debugging on pam_ldap for native Solaris, add "debug" to the options at the end of any auth setting for pam_ldap in the /etc/pam.conf file (the entry can be added before or after the use_first_pass entry). For example:

other  auth sufficient  pam_ldap.so.1 use_first_pass debug

Warning Enabling debugging for LDAP and pam_ldap can significantly delay logon and logout operations.

The following references will be helpful in understanding LDAP errors:

Download

Get the Windows Security and Directory Services for UNIX Guide

Update Notifications

Sign up to learn about updates and new releases

Feedback

Send us your comments or suggestions