Imagine a scenario where you are analyzing/troubleshooting some server or application issue and you would like to collect all relevant eventlogs from server for your research. However, sometime you may end-up in below few situations which will make this task a nightmare:

Ntds.dit

Ntds.dit is the main AD database file. NTDS stands for NT Directory Services. The DIT stands for Directory Information Tree. The Ntds.dit file on a particular domain controller contains all naming contexts hosted by that domain controller, including the Configuration and Schema naming contexts. A Global Catalog server stores the partial naming context replicas in the Ntds.dit right along with the full Domain naming context for its domain.

Edb.log

Edb.log is a transaction log. Any changes made to objects in Active Directory are first saved to a transaction log. During non-peak times in CPU activity, the database engine commits the transactions into the main Ntds.dit database. This ensures that the database can be recovered in the event of a system crash. Entries that have not been committed to Ntds.dit are kept in memory to improve performance. Transaction log files used by the ESE (Extensible Storage Engine is an Indexed Sequential Access Method (ISAM) data storage technology from Microsoft. ESE is the core of Microsoft Exchange Server and Active Directory.) engine are always 10MB.

Edbxxxxx.log

These are auxiliary transaction logs used to store changes if the main Edb.log file gets full before it can be flushed to Ntds.dit. The xxxxx stands for a sequential number in hex. When the Edb.log file fills up, an Edbtemp.log file is opened. The original Edb.log file is renamed to Edb00001.log, and Edbtemp.log is renamed to Edb.log file, and the process starts over again. Excess log files are deleted after they have been committed. You may see more than one Edbxxxxx.log file if a busy domain controller has many updates pending.

Edb.chk

Edb.chk is a checkpoint file. It is used by the transaction logging system to mark the point at which updates are transferred from the log files to Ntds.dit. As transactions are committed, the checkpoint moves forward in the Edb.chk file. If the system terminates abnormally, the pointer tells the system how far along a given set of commits had progressed before the termination.

Res1.log and Res2.log

Res1.log and Res2.log are reserve log files. If the hard drive fills to capacity just as the system is attempting to create an Edbxxxxx.log file, the space reserved by the Res log files is used. The system then puts a dire warning on the screen prompting you to take action to free up disk space quickly before Active Directory gets corrupted. You should never let a volume containing Active Directory files get even close to being full. File fragmentation is a big performance thief, and fragmentation increases exponentially as free space diminishes. Also, you may run into problems as you run out of drive space with online database defragmentation (compaction). This can cause Active Directory to stop working if the indexes cannot be rebuilt.

Temp.edb

This is a scratch pad used to store information about in-progress transactions and to hold pages pulled out of Ntds.dit during compaction.

Schema.ini

This file is used to initialize the Ntds.dit during the initial promotion of a domain controller. It is not used after that has been accomplished

While troubleshooting any domain related issues or CRM server authentication or name resolution issues, it is good that we also check the DC health and perform some quick test using below basic commands to check if everything is working fine with DC.

Here are few basic tools that can be used for this purpose:

DCDIAG

DCDIAG analyzes the state of domain controllers in a forest or enterprise and reports any problems to help in troubleshooting.

1. Preparing to install or migrate to Exchange new version
   
2. Checking FSMO roles.
   
3. Troubleshooting Group Policy.
   
4. Investigating Active Directory not replicating frssysvol error.
   
5. Running down Kerberos authentication problems.
   
6. Resetting the Directory Service Administrator's password.
   
7. Fixing servers Service Principle Name (SPN) error.
   
8. Other DC issues
   

Example: dcdiag.exe /V /D /C /E > c:\dcdiag.log

DCDIAG tool article https://technet.microsoft.com/en-us/library/cc731968(v=ws.10).aspx

REPADMIN

Repadmin.exe is a Microsoft Windows 2000 Resource Kit tool that is available in the Support Tools folder on the Windows 2000 CD-ROM. It is a command-line interface to Active Directory replication. This tool provides a powerful interface into the inner workings of Active Directory replication, and is useful for troubleshooting Active Directory replication problems.

Example: repadmin.exe /showrepl dc* /verbose /all /intersite > c:\repl.txt

RepAdmin tool article https://technet.microsoft.com/en-us/library/cc770963(v=ws.10).aspx

NETDIAG

This command-line diagnostic tool helps to isolate networking and connectivity problems by performing a series of tests to determine the state of your network client. These tests and the key network status information they expose give network administrators and support personnel a more direct means of identifying and isolating network problems. Moreover, because this tool does not require parameters or switches to be specified, support personnel and network administrators can focus on analyzing the output rather than on training users how to use the tool.

1. Installing Exchange and you wish to check that you can connect to other servers.
   
2. Checking VPN network tunnels on the WAN.
   
3. DNS problems.  Computers cannot 'see' their domain controller on the LAN.
   
4. A quick check on hotfixes.
   
5. Check the Network Card Bindings from the command prompt.
   
6. You are having problems with IPSEC.
   
7. Winsock corruption, wrong version incompatibilities.
   
8. NetDiag checks that Domain Controllers are all able to 'speak' LDAP.
   

Example: netdiag.exe /v > c:\netdiag.log

Note: This command need to be run on each DC of the domain.

NetDiag tool article https://technet.microsoft.com/en-us/library/cc731434(v=ws.10).aspx

It is very important to understand the concept of secure channel between an AD and workstation machine or between 2 domains which are in trust. This article will give you an overall picture of a security channel. Also, it will explain what issues you will experience if the security channel is broken or how to identify issues due to broken channels.

Every computer that's joined to a domain—whether it's a workstation client, a server, or a DC—requires connectivity to DCs in order to fulfill some of the service requirements that AD domains require. For workstations and servers, that connectivity is to the DCs in the domain that they're a member of, as well as trusting domains' DCs. DCs in one domain need connectivity to DCs in other trusting and trusted domains. The name describing the cached values in that inter-domain connectivity is the "domain secure channel." To be clear, there are two kinds of secure channels:

• Secure channels from a domain member to a DC in its domain
• Trust secure channels between a trusted and trusting DC

Secure Channel between Trusted Domains

To validate access to resources in a trusting domain, the trusting domain controller establishes a secure channel with a domain controller in the trusted domain. Pass-through authentication then occurs over this secure channel. However, in WAN environments, the trusted domain's domain controllers may be dispersed over a wide variety of fast and slow links. If a fast link is unavailable when the trusting domain controller wants to establish a secure channel, the secure channel may be established with a domain controller over a slow link. Even when the fast link is reestablished, pass-through authentication may occur over the slow link to the trusted domain's domain controller. The mechanism for establishing a secure channel is very similar to the normal user -logon process. That is, the trusting domain controllers send out logon requests to all known domain controllers in the trusted domain. The trusting domain controllers then set up a secure channel with the first trusted domain controller that responds to this request. Normally, this method is preferred because the first domain controller to respond to a logon request is typically the controller that is located across the fastest communication link. However, if that link is down or the domain controller is unavailable, a domain controller over a slower link may respond first, and all pass-through authentications occur over the slow link.

Here is one of the errors that you will receive if the CRM –AD Security channel is broken. You will not be able to login to CRM Server and receive below error message.

Verifying Trust using NLTEST utility

There is a built-in mechanism in Windows that tracks how long authentication takes over the existing secure channel. If pass-through authentication takes longer than 45 seconds, that fact is noted. If two such authentications exceed that limit, a rediscovery process begins, the current secure channel is broken, and the trusting domain's PDC once again sends out logon requests to all known trusted domain controllers. However, because this mechanism tracks only those communications that last longer than 45 seconds, users may see a 40- second delay every time they attempt to use a resource without a secure-channel reset taking place. You can run the NLTEST utility on the trusting domain controller to break and re-initialize a secure channel (for example, when  the  secure-channel  password  was  last  changed)  and  obtain  information  about  an  existing  trust relationship. You can also use NLTEST to restart the discovery process for a new trusted domain controller. The syntax of NLTEST is:

NLTEST /sc_query:<account_domain>

If Security Channel is good, you should output something similar to below screenshot

If the security channel is broken, you shall receive error message as shown in below screenshot

Also, you can use nltest /sc_query:<domain.com>  to test the security channel status

If CRM-AD security channel is broken then you may see few of the listed problems:

1. Unable to login to CRM
2. Unable to add new users in CRM from trusted domain

   Error-1

   Business Management Error: You are attempting to create a user with a domain logon that does not exist. Select another domain logon and try again.

      

   Unhandled Exception: System.ServiceModel.FaultException`1[[Microsoft.Xrm.Sdk.OrganizationServiceFault, Microsoft.Xrm.Sdk, Version=5.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]]: LookupAccountNameW failed with error Detail:

   <OrganizationServiceFault xmlns:i="https://www.w3.org/2001/XMLSchema-instance" xmlns="https://schemas.microsoft.com/xrm/2011/Contracts">

     <ErrorCode>-2147214038</ErrorCode>

     <ErrorDetails xmlns:d2p1="https://schemas.datacontract.org/2004/07/System.Collections.Generic" />

     <Message>LookupAccountNameW failed with error</Message>

     <Timestamp>2013-12-19T16:24:35.8171875Z</Timestamp>

     <InnerFault i:nil="true" />

     <TraceText i:nil="true" />

   </OrganizationServiceFault>

    

   OR

    

   Error-2

   Unable to find AD object for SID

To know more about NLTEST and Troubleshooting Security Channel issues, please refer below articles:

https://social.technet.microsoft.com/wiki/contents/articles/16067.nltest-to-test-the-trust-relationship-between-a-workstation-and-domain.aspx

https://support.microsoft.com/kb/158148

https://blogs.technet.com/b/asiasupp/archive/2007/01/18/typical-symptoms-when-secure-channel-is-broken.aspx

Windows includes a caching DNS-resolver service, which is enabled by default. For troubleshooting purposes, this service can be viewed, stopped, and started like any other Windows service. The caching resolver reduces DNS network traffic and speeds name resolution by providing a local cache for DNS Queries.

When a name is submitted to DNS, if the resolver is caching names, it first checks the cache. If the name is in the cache, the data is returned to the user. If the name isn't in the cache, the resolver queries the other DNS servers that are listed in the TCP/IP properties for each adapter. It does this in the following order:

1. The resolver first checks the local hosts file (located by default in C:\Windows\System32\drivers\etc) to see whether the required name is listed in host file. The local host address, for example, resolves to 127.0.0.1 through use of the hosts file, not a DNS server.

2. In case the name isn't in the hosts file, the resolver then sends the query to the first server on the preferred adapter's list of DNS servers and waits one second for a response.

3. If the resolver doesn't receive a response from the first server within one second, it se nds the query to the first DNS servers on all adapters that are still under consideration and waits 2 seconds for a response.

4. If the resolver doesn't receive a response from any server within 2 seconds, it sends the query to all DNS servers on all adapters that are still under consideration and waits another 2 seconds for a response.

5. If it still doesn't receive a response from any server, it sends the query to all DNS servers on all adapters that are still under consideration and waits 4 seconds for a response.

6. If it still doesn't receive a response from any server, the resolver sends the query to all DNS servers on all adapters that are still under consideration and waits 8 seconds for a response.

7. If the resolver receives a positive response, it stops querying for the name, adds the response to the cache, and returns the response to the client. If it doesn't receive a response from any server by the end of the 8 seconds, it responds with a time-out. Also, if it doesn't receive a response from any server on a specified adapter, it responds for the next 30 seconds to all queries destined for servers on that adapter with a time-out and doesn't query those servers.

The resolver also keeps track of which servers answer queries more quickly, and it might move servers up or down on the search list based on how quickly they respond. In addition, the resolver also caches negative responses. If the resolver is able to successfully reach a domain controller, but that domain controller is unable to resolve the requested name to an IP address, the result is a negative response. So long as that negative response remains in the cache, the resolver will not try to resolve the address again. You can clear the cache by running the following from a command-line:

IPCONFIG /FLUSHDNS

Doing so forces the resolver to start over the next time any name needs to be resolved.

To learn more about Caching and Negative Caching, please refer below TechNet Article

https://technet.microsoft.com/en-us/library/cc959309.aspx

You can also fix this problem by using the IPCONFIG command. Entering the following command allows you to view the current list of DNS entries that the server has cached:

IPCONFIG /displayDNS

Entering the following command allows you to refresh all DHCP leases and re-register DNS names. (Wait 5 minutes for the DNS entries in the cache to be reset and updated with the RRs in the server's database.)

IPCONFIG /registerDNS

You can also use the IPCONFIG command to dump all of the DNS cache entries.

IPCONFIG /flushDNS

It's worth noting that the DNS server should eventually refresh the cache because each entry has a Time-To-Live (TTL) associated with it. TTL indicates a length of time used by other DNS servers to determine how long to cache information for a record before discarding it.

CRM - SQL Authentication Issue

Event error logs are useful. However, it is huge to go through and find the relevant error message. I am sharing this handy script that will help to identify the logon failure details at SQL side. For instance if CRM App Pool account or any service account is failing to access database or failing to get authenticated by SQL then we can collect all the details using below scripts.

Let's say you are trying to access CRM url https://crm.habib.local:5555 and you are receiving below error message in browser. It's clear from the error message that "Habib\CRMAppPoolSvc" account is failing to login or authentication is failing.

In order to find more details about this error from SQL side, you can use below scripts

• You can use below SP to find all error logs in SQL archived logs

  EXEC sp_readerrorlog

   

• To search particular archived logs for particular string use below script. For example if you want to search current log and archive#1 log for error "Logon Failed", you can use below script

   

EXEC sp_readerrorlog0, 1, 'Login failed'

Please Note: 0 = current, 1 = Archive #1, 2 = Archive #2, etc...

• You can replace your search string accordingly with any service account which you are searching in error logs

   

EXEC sp_readerrorlog0, 1, 'HABIB\CRMAppPoolSvc'

• Now the above script will give either all the error logs or filtered error logs. You can use below master script to pull all the failure details related to "Logon" process, which could fail due to various reasons and with various errors.

  This will create 2 temp tables and drop it after the execution.

DECLARE @TSQL  NVARCHAR(2000)

DECLARE @lC    INT

CREATE TABLE #TempLog (

      LogDate     DATETIME,

      ProcessInfo NVARCHAR(50),

      [Text] NVARCHAR(MAX))

CREATE TABLE #logF (

      ArchiveNumber     INT,

      LogDate           DATETIME,

      LogSize           INT

)

INSERT INTO #logF  

EXEC sp_enumerrorlogs

SELECT @lC = MIN(ArchiveNumber) FROM #logF

WHILE @lC IS NOT NULL

BEGIN

      INSERT INTO #TempLog

      EXEC sp_readerrorlog@lC

      SELECT @lC = MIN(ArchiveNumber) FROM #logF

      WHERE ArchiveNumber > @lC

END

--Failed login counts. Useful for security audits.

SELECT 'Failed - ' + CONVERT(nvarchar(5), COUNT(Text)) + ' attempts' AS [Login Attempt], Text AS Details

FROM #TempLog

where ProcessInfo = 'Logon'

and Text like '%failed%'

Group by Text

DROP TABLE #TempLog

DROP TABLE #logF

There are multiple instances where the CRM was no more accessible right after some database restore operations activity. Also, there are instances where in order to restore the databases to previous state or creating test environment you have accidently restored wrong database.

Let's say you have many automated backup files and manual backups in the backup folder location and you not sure if you or other admin user have restored wrong date backup file and CRM became inaccessible, then you can use below few steps to audit your database restore and backup history and isolate the issue.

Using below script you can keep a track of all the databases that were restored. With restore date-time, it will also show the date-time of the backup file that was used to restore. This will help you identify if you have by mistaken restored any wrong date-time database.

Also, using this report you can audit if any critical database was restored which might have caused the downtime. For example, in CRM all the connection related information is stored in MSCRM_CONFIG Database and each CRM environment will have a unique MSCRM_CONFIG database holding the configuration and setting information. If MSCRM_CONFIG database is restored using backup file of old date or backup of other CRM Environment, then it will break down the whole CRM environment. In such cases below steps will help you isolate and troubleshoot the issue.

Method-1: Using SQL Script

use msdb;

select

    DBRestored = destination_database_name,

    RestoreDate = restore_date,

    SourceDB = b.database_name,

    SourceFile = physical_name,

    BackupDate = backup_start_date

from RestoreHistory h

inner join BackupSet b

      on h.backup_set_id = b.backup_set_id

inner join BackupFile f

      on f.backup_set_id = b.backup_set_id

order by RestoreDate

go

Method-2: You use Database management reports to pull Backup-Restore history

There are multiple instances where user uses either one custom service account to run all CRM Services or uses different dedicated service accounts for each CRM Service. If the service accounts are locked out to due to some issue, then the CRM Services will be in stopped state and it will cause downtime. To isolate/troubleshoot locked out issues, you may find this article very helpful.

Please Note: There are many sysinternal tools available to verify Account Lockouts, I am explaining the built in commands or tools to analyze such problems.

Method-1: Account Lockout Tools

There are many methods and tools to find the Account Lockout status or to unlock a locked account. In this post I have explained about one famous tool and command.

Using the LockoutStatus.exe Tool – This tool comes with Account Lockout Tools package. This package was used earlier in Windows 2003. Account Lockout and Management Tools can be used on Windows Server 2008 as well.

Download Account Lockout Tools from here

https://www.microsoft.com/en-us/download/details.aspx?id=18465

How to use LockoutStatus.exe Tool

To run the LockoutStatus.exe tool and display information about a locked out user account:

• Double-click LockoutStatus.exe.
• On the File menu, click Select target.
• Type the user name whose lockout status on the enterprise's domain controllers you want information about. The below screenshot shows that user account f1 is locked.
  

  
  

• To Unlock the Account, right click and select Unlock

  To know more about Account Lockout Tools, read the below technet article

  https://technet.microsoft.com/en-us/library/cc738772(v=ws.10).aspx

   

Method-2: How to use Repadmin Command to find Account Lockout details

• First we have to find the FQDN of the locked user. You can use DSQUERY command for this.

  

• Then run the below command to displays the domain controller that locked the account

  Repadmin /showmeta "FQDNofUser"

  Here is an example to displays the domain controller that locked the account f1:

Type the command Repadmin /showmeta "CN=f1, OU=Finance,OU=East Sales,cn=habib,cn=local"

You will get the output as below: In this we can only find the details of domain controller that locked the account. It cannot unlock the account from this command line.

There are multiple instances where while troubleshooting/isolating CRM issues, you end up checking details from AD side. In that case you can refer to below article which pretty much explains how to enable audit in AD to track activities.

In Windows Server 2008, you can now set up AD DS (Active Directory Domain Services) auditing with a new audit policy subcategory (Directory Service Changes) to log old and new values when changes are made to AD DS objects and their attributes. This can be done using auditpol.exe tool.

Here is a quick command to check which audit policies are currently active:

auditpol /get /category:*

Command to view the audit policy categories and Subcategories:

How to enable the global audit policy using the Windows interface i.e. gpmc tool

• Click Start, point to Administrative Tools, and then Group Policy Management or run gpmc.msc command
• In the console tree, double-click the name of the forest, double-click Domains, double-click the name of your domain, double-click Domain Controllers, right-click Default Domain Controllers Policy, and then click Edit.
  

  
  

• Under Computer Configuration, double-click Policies, double-click Windows Settings, double-click Security Settings, double-click Local Policies, and then click Audit Policy.
  

  
  

• In the details pane, right-click Audit directory service access, and then click Properties.
• Select the Define these policy settings check box.
• Under Audit these attempts, select the Success, check box, and then click OK.
  

  
  

How to enable the change auditing policy using a command line

• Click Start, right-click Command Prompt, and then click Run as administrator.
• Type the following command, and then press ENTER:
• auditpol /set /subcategory:"directory service changes" /success:enable
  

To verify if the auditing is enabled or not for "Directory Service Changes", you can run below command:

Auditpol /get /category:"DS Access"

How to set up auditing in object SACLs

• Click Start, point to Administrative Tools, and then click Active Directory Users and Computers.
• Right-click the organizational unit (OU) (or any object) for which you want to enable auditing, and then click Properties.
• Click the Security tab, click Advanced, and then click the Auditing tab.
  

  
  

• Click Add, and under Enter the object name to select, type Authenticated Users (or any other security principal), and then click OK.
  

  
  

• In Apply onto, click Descendant User objects (or any other objects).
• Under Access, select the Successful check box for write all properties.
  

  
  

• Click OK until you exit the property sheet for the OU or other object.
  

To Test whether auditing is working or not, try creating or modifying objects in Finance OU and check the Security event logs.

I just created a new user account in Finance OU named f4.

If you check the security event logs you will find eventid 5137 (Create)

Note:
Once the auditing is enabled these eventids will appear in security event logs: 5136 (Modify), 5137 (Create), 5138 (Undelete), 5139 (Move).

To know more about AD DS Auditing read technet article:
AD DS Auditing Step-by-Step Guide

https://technet.microsoft.com/en-us/library/cc731607(v=ws.10).aspx

So what's the big deal with AD Replication? I mean it's obvious that your infrastructure has multiple Domain Controllers and you want to make sure that Active Directory talks to each other directly or indirectly and keep each other up to date. We came up with Replication concept. 

Now it's important that Domain controller talk to right domain controller, I mean there should not be any confusion, everyone should be aware who is sending update and receiving updates from whom. So we came up with KCC concept ( Knowledge Consistency Checker - it helps to design the replication topology automatically)

Now too much talking between AD Servers are also not good, each site is connected using very costly WAN link and I am sure any company don't want any useless replication happening via these costly links. So we came up with concepts like Site Link Cost, Replication Schedule.

Trust me AD Replication is a huge topic. This article one pretty much explains basic replication concept between 2 DC's.



Active Directory (AD) is a multi-master directory, meaning each directory services server—referred to as a domain controller—contains a fully readable and writeable copy of the directory
services database. Because all domain controllers can accept changes to the database, some method is needed to replicate those changes to other domain controllers, ensuring a consistent
database across all domain controllers. This scheme is referred to as AD replication.

AD replication can be broken down into four basic operational components:

WHO

It is a list of servers that participate in replication and the servers with which they replicate. Referred to as a replication topology, this list is generated by a special AD component called the Knowledge Consistency Checker (KCC).
Checking replication Topology -  https://technet.microsoft.com/en-us/library/cc739234(v=ws.10).aspx  

WHAT

It is the information that is being replicated. AD uses attribute-based replication and versioning to determine which information has changed and requires replication.
https://windowsitpro.com/active-directory/active-directory-replication-depth

WHEN

It is a schedule that determines when replication will occur. Separate schedules exist for replication within an AD site and for each link connecting different sites.
Managing Replication between sites- https://technet.microsoft.com/en-us/library/cc961783.aspx

HOW

It defines how the replicated data is physically transported across the network.

Site Replication

• Replication traffic is not compressed to save processor processing.
• Replication partners notify each other when changes need to be replicated, to reduce replication latency.
• Replication partners poll each other for changes on a periodic basis.
• Replication uses the remote procedure call (RPC) transport.
• Replication connections can be created between any two domain controllers located in the same site. Connections can be made with multiple domain controllers to reduce replication latency.

 Inter-site Replication

• Replication traffic is compressed to save bandwidth.
• Replication partners do not notify each other when changes need to be replicated, to save bandwidth.
• Replication partners poll each other for changes on a specified polling interval, during scheduled periods only.
• Replication uses the TCP/IP or SMTP transport.
• Replication connections are only created between bridgehead servers (servers that handles all inter-site replication for that domain). Connections between bridgehead servers using the lowest cost route, according to site link cost. Connections will only be created over higher cost routes if all of the domain controllers in lower cost routes are unreachable.

To know more about AD replication please refer to below articles:

https://technet.microsoft.com/en-us/library/cc961788.aspx

https://technet.microsoft.com/en-us/library/cc755994(v=ws.10).aspx#w2k3tr_repto_how_uvgz

Troubleshooting Replication: https://technet.microsoft.com/en-us/library/cc755349(WS.10).aspx#BKMK_1