With Hyperthreading turned OFF

2011-10-27 12:37:32.74 Server      Detected 40 CPUs. This is an informational message; no user action is required.

2011-10-27 12:37:32.94 Server      Large Page Extensions enabled.

2011-10-27 12:37:32.94 Server      Large Page Granularity: 2097152

2011-10-27 12:37:32.98 Server      Large Page Allocated: 32MB

2011-10-27 12:37:33.03 Server      Large Page Allocated: 32MB

2011-10-27 12:37:33.42 Server      Using locked pages for buffer pool.

2011-10-27 12:37:33.79 Server      Large Page Allocated: 32MB

2011-10-27 12:37:33.90 Server      Large Page Allocated: 32MB

2011-10-27 12:37:34.17 Server      Using dynamic lock allocation.  Initial allocation of 2500 Lock blocks and 5000 Lock Owner blocks per node.  This is an informational message only.  No user action is required.

2011-10-27 12:37:34.19 Server      Lock partitioning is enabled.  This is an informational message only. No user action is required.

2011-10-27 12:37:34.53 Server      Multinode configuration: node 0: CPU mask: 0x00000000000ffc00 Active CPU mask: 0x00000000000ffc00. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.

2011-10-27 12:37:34.60 Server      Multinode configuration: node 1: CPU mask: 0x00000000000003ff Active CPU mask: 0x00000000000003ff. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.

2011-10-27 12:37:34.65 Server      Multinode configuration: node 2: CPU mask: 0x000000003ff00000 Active CPU mask: 0x000000003ff00000. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required.

2011-10-27 12:37:34.70 Server      Multinode configuration: node 3: CPU mask: 0x000000ffc0000000 Active CPU mask: 0x000000ffc0000000. This message provides a description of the NUMA configuration for this computer. This is an informational message only. No user action is required

With Hyperthreading turned ON

2011-10-27 13:13:50.31 Server     Detected 20 CPUs. This is an informational message; no user action is required.

2011-10-27 13:13:50.48 Server      Large Page Extensions enabled.

2011-10-27 13:13:50.48 Server      Large Page Granularity: 2097152

2011-10-27 13:14:04.68 Server      Large Page Allocated: 32MB

2011-10-27 13:14:07.87 Server      Using locked pages for buffer pool.

2011-10-27 13:14:12.25 Server      Using dynamic lock allocation.  Initial allocation of 2500 Lock blocks and 5000 Lock Owner blocks per node.  This is an informational message only.  No user action is required.

2011-10-27 13:14:12.28 Server      Lock partitioning is enabled.  This is an informational message only. No user action is required.

Cause and Explanation: A new feature introduced in Windows Server 2008 R2 called “Processor Groups” was responsible for this behavior. Existing applications such as SQL Server 2005 and 2008, which were released prior to the release of Windows Server 2008 R2, can’t take advantage of this feature thus causing the issue

What is a Processor group? In Windows Server 2008 R2, support for systems that have more than 64 logical processors is based on the concept of a processor group, which is a static set of up to 64 logical processors that is treated as a single scheduling entity. Processor groups are numbered starting with 0. Systems with fewer than 64 logical processors always have a single group, Group 0. The operating system minimizes the number of groups in a system. For example, a system with 128 logical processors would have two processor groups with 64 processors in each group, not four groups with 32 logical processors in each group. Processor Group is a new concept in Windows Server 2008 R2 and is not supported by Windows Server 2008, Windows Vista and Windows Server 2003.

Why does Processor Groups affect SQL Server 2005 and 2008 instances? Following three factors combined together would cause Processor Groups to affect SQL Server 2005 and 2008 instances -

• Both SQL Server 2005 and SQL Server 2008 were released prior to the release of Windows Server 2008 R2 and therefore these versions do not utilize the new APIs that were released to take advantage of Processor Groups in Windows Server 2008 R2.
• Windows Server 2008 R2 creates uneven Processor Groups on machines that has > 64 logical processors. This is because the algorithms implemented in Windows 2008 R2 originally were looking to create as little as possible processor groups and keep the individual processor groups as large as possible. Since there is only one Processor Group on servers that has <= 64 processors, no problems are expected on such machines. However, if a machine has more than 64 logical processors, Windows Server 2008 R2 will create more than one processor group and SQL Server 2005/2008 will be bound to any of the groups on service restart. In our scenario, after Hyperthreading was turned on, the machine had 80 Logical processors and 4 NUMA nodes, which were divided into 2 groups by the Windows Server 2008 R2 operating system. The first group had 3 NUMA nodes and hence 60 logical processors in it while the second groups had 1 NUMA node with 20 logical processors in it. When SQL instance was restarted, it was bound to the first group with 20 logical processors in it.
• Windows Server 2008 R2 will randomly assign such applications that are not aware of Processor Groups to one of the existing Processor Groups.

Workaround/Fix: A couple of things to consider -

• How to utilize > 64 logical processors for SQL instances? SQL Server 2005/2008 versions cannot utilize more than 64 logical processors and therefore if you would like your SQL server instance to be able to utilize more than 64 processors, the only option is to upgrade to SQL Server 2008 R2.
• How to force a SQL Server instance to use the Processor Group with maximum number of logical processors in it? For example, on a machine with 96 CPUs you could have two processor groups, Group # 1 with 64 CPUS and Group # 2 with 32 CPUs.  If you have SQL Server 2005/2008 Enterprise installed it can use up to 64 CPUs.  In this scenario, you don’t want  your SQL Server instance to be associated with Group # 2 that has 32 CPUs. If you would like your SQL Server 2005 and 2008 instances to be able to utilize the processor group that has maximum available processors in it, you can use the sc.exe utility as described in this blog article
• How to resolve issues with uneven processor group assignment? Apply Windows Server 2008 R2 hotfix as described in KB 2510206
• How to manually assign processor groups to specific NUMA nodes? Follow instructions in KB 2506384

Additional References:

• Processor Groups
• Uneven Windows Processor Groups
• Performance issues when more than 64 logical processors are used in Windows Server 2008 R2
• How to manually configure K-Group assignment on multiprocessor machines
• Changes in affinity settings of SQL Server 2008 R2 to support > 64 Logical Processors
• Windows 2008 R2 - Groups, Processors, Sockets, Cores Threads, NUMA nodes what is all this?
• ALTER SERVER CONFIGURATION
• SQL Server on Windows 7 / Windows 2008 R2 with more than 64 CPUs
• How many logical processors does SQL Server 2008 R2 Enterprise Edition support

Would love to hear thoughts from folks who have experienced this issue firsthand. Please leave your feedback below if you have anything to share

Cheers!

Installation Failure:

Scenario: The first time I tried the installation with mount points, the installation failed with the following error – “Wait on the Database Engine recovery handle failed. Check the SQL Server error log for potential causes”. Consider the following scenario -

• During the installation, on the Data Directories page, you select separate mount points for system, data, log, Tempdb and Backup directories, but do not make any changes to the default folder structures
• The SQL Server service account does not have sufficient permissions on the mounted drive.
• The installation fails with the following error message – “Wait on the Database Engine recovery handle failed. Check the SQL Server error log for potential causes”

Cause: I discovered that this is a known issue in SQL Server 2008 / R2. If you proceed with the default folder hierarchy after selecting your Mount Points, the default location points to the root of the mounted volumes. If your SQL Server service does not have sufficient permissions to write to the root of the mounted volumes, installation will fail.

Workarounds: You can use one of the following two workarounds to fix this problem -

• Workaround # 1: Change the default folder path to use a sub directory under the root drive. For example, if T:\mountX is a mount points of volume X. Then specify T:\mountX\data as the directory to store database files.
• Workaround # 2: Grant SQL server service account full control permissions to the root volume, such as mountX by using the disk management console

References:

• Installation of SQL Server 2008 R2 on Mount Points fails
• Gotchas with mounted drives or mount points on Microsoft Windows Server
• SQL Server 2008 Analysis Services failover cluster setup fails to install on a Windows Server 2008-based cluster mount point
• Microsoft SQL Server 2008 may not start after you install it in the root directory of an NTFS volume

Mount Point Dependency Issues:

Scenario: After you have successfully installed a SQL Server 2008/R2 failover cluster with mount points, data corruption could occur because of incorrect resource dependencies in the SQL server instance group

Cause: This is a known issue in SQL Server 2008/R2. SQL Server setup will not create the required dependencies automatically and you will need to add the required dependencies manually.

Solution: Add the following dependencies manually -

• Make sure that each of the mounted volume is dependent on the root (or host) drive.
• Make sure that SQL Server is dependent on not just the root (or host) drive, but also on each of the mounted volumes.

References:

• After you install a SQL Server 2008 failover cluster in a disk that contains a mounted volume, no dependencies are created between the mounted volume and the disk
• Failover cluster resource dependencies in SQL Server

Best Practices when using Mount Points in SQL Server -

Follow the following best practices when using mount points with SQL Server standalone and failover cluster instances -

• Use the root (host) volume exclusively for mount points. The root volume is the volume that is hosting the mount points. This greatly reduces the time that it takes to restore access to the mounted volumes if you have to run a chkdsk. This also reduces the time that it takes to restore from backup on the host volume.
• If you use the root (host) volume exclusively for mount points, the size of the host volume only has to be several MB. This reduces the probability that the root volume will be used for anything other than the mount points.
• During failover cluster installation, use subdirectories under the root of mounted volumes to store database and backup files. For example, say you have a mounted volume F:\SQL1. This is the root of the mount point and you shouldn't use this location directly to store your database files. You should instead create a subdirectory/subfolder such as d:\SQL1\USERDATA and use this location to store your database files.
• Add missing dependencies after installing SQL Server 2008 / R2 failover cluster with mount points. Make sure that each of the mounted volume is dependent on the root (or host) drive. Additionally, make sure that SQL Server is dependent on not just the root (or host) drive, but also on each of the mounted volumes.
• If configuring MSDTC on failover cluster, do not use mount points as storage for the MSDTC service. MSDTC currently does not supported mount points.

References:

• How to configure volume mount points on a Microsoft Cluster Server in Windows Server 2003
• How to configure volume mount points on a server cluster in Windows Server 2008
• SQL Server support for mounted volumes

Hope you find this useful!

The SQLNexus tool can be used to analyze SQL server performance data such as blocking, wait stats and SQL traces. The PAL tool can be used to analyze the Perfmon counters collected in a .BLG file . Both of these tools are open source tools and are publicly available for download on the www.codeplex.com website. Please see the SQL Server Downloads section on the Blog Home for the download location of these tools and their pre-requisites components.

What sort of diagnostic data can I collect by using the SQLDiag tool?

The SQLDiag utility is capable of collecting the following sets of diagnostic information -

• Windows Event Logs (Application, System, and Security) in .CSV files
• System Monitor (aka Perfmon) performance counters in .BLG file/s
• SQL Server Profiler traces in .TRC file/s
• SQL Server Blocking chains in a _BLK.TRC file
• SQL Server error logs, configuration, point-in-time snapshots of several DMVs in a .OUT file
• System information via the Msinfo32 utility in a .TXT file

Additionally, SQLDiag can be configured to collect custom data such as the output of a .CMD batch file or a .EXE utility.

What sort of performance issues can I troubleshoot with the SQLDiag, SQLNexus and the PAL tools?

SQLDiag, SQLNexus and PAL tools can be used to troubleshoot a variety of performance issues in SQL Server. I have used these tools on many occasions to assist my customers while troubleshooting performance problems, mostly in the following scenarios -

• Intermittent CPU spikes on the SQL Server box
• Performance degradation during specific hours (say between 2:00 PM and 4:00 PM everyday)
• Troubleshooting Blocking chains in SQL Server
• Analyzing overall performance bottlenecks in SQL Server using wait statistics
• Identifying top queries by CPU, Duration, Reads and Writes during a specific timeframe (say between 9:00 AM and 5:00 PM)
• Identifying specific databases and logins (sessions) that are consuming the maximum CPU and IO on a SQL sever instance during a specific timeframe (say between 9:00 AM and 5:00 PM)
• Troubleshooting performance issues with a batch job that runs every night during a specific timeframe (say between 2:00 AM and 4:00 AM)
• Comparing SQL server performance for various tests in a lab testing scenario – say you are seeing performance and scalability issues with a 150 user test, while the 50 user test runs fine.

Of course, this is not an exhaustive list  that describes the application of these tools and you will find them useful in troubleshooting other issues in SQL Server as well. In my future posts, I will attempt to describe how I used these tools to troubleshoot some of the above scenarios.

How are these tools different from  the inbuilt features such as Standard Reports and Activity Monitor in Management Studio ?

That’s a great question! Why even bother to learn these additional tools when we have introduced so many great features in SQL Server 2005 such as Standard Reports, and Activity Monitor in Management Studio reports and Dynamic Management Views (DMVs). The answer is – it depends! Each of these tools have their own strengths and can be useful in a specific scenario depending on the problem you are trying to troubleshoot. SQLDiag, SQLNexus and PAL are very helpful when troubleshooting the point-in-time performance issues – such as issues occurring during a specific timeframe,  say between 2:00 PM and 4:00 PM. Standard Reports and Activity Monitor in Management Studio are based on DMVs (Dynamic Management Views), which stores cumulative statistics since the last start of SQL Server. So if your SQL Server instance has been running since last two months and say you have been experiencing performance issues between 2:00 PM and 4:00 PM since only a last few days, then you may not get a lot of useful information from the Standard reports or Activity Monitor. Standard reports will show you the cumulative information since the last restart of SQL Server instance and not for the specific timeframe that your are interested in - such as between 2:00 PM and 4:00 PM. Similarly, Activity Monitor will display the live performance statistics from a SQL server instance and not for a specific time frame in the past. But if you want to be able to collect the diagnostic data for a specific time frame so that you can analyze it later, or so that you can share it with someone else for later troubleshooting, then SQLDiag/SQLNexus/PAL are your friends.

How are these tools different from the new Management Data Warehouse (MDW) feature introduced in SQL Server 2008?

The Management Data Warehouse (MDW) is a new feature that was introduced in SQL Server 2008. MDW implements the best of SQLDiag/SQLNexus and “generally” replaces the need for each of them. It’s a framework that allows for collecting performance data from one or more SQL Server instances on an ongoing basis and storing that data permanently in a SQL Server database. MDW provides a set of inbuilt reports that enables you to go back in time, select a particular time frame and analyze performance data only for the selected timeframe. It provides similar reports as the SQLNexus tool. Customers must have enabled this feature on their SQL Server instances to be able to use it. Many customers are not fully aware of this great feature and many have not implemented it in their environment yet, so I still rely on the SQLDiag, SQLNexus and the PAL tools to assist my customers when troubleshooting performance issue. In addition, when there is a need to do in-depth analysis, I like several specific aspects of the SQLNexus reports that are not available in the MDW reports . For example, the SQLNexus reports can tell me which specific databases and logins are responsible for most CPU usage on my server. I like the Interesting Events reports available in SQLNexus while we don’t have a similar report in MDW. Another great feature I like in the SQLNexus reports is the distribution of query statistics such as # of executions, Duration, I/O, and CPU across smaller intervals. Consider a scenario where you are troubleshooting CPU spikes that occur everyday on your server between 2:00 PM and 4:00 PM. Say you collected the SQLDiag data between 2:00 PM and 4:00 PM and analyzed it with SQLNexus. Assuming that a particular SQL query was responsible for these CPU spikes, you can drill down into the SQLNexus reports and find out which particular query execution relates to the CPU spikes on the server and if it is the number of executions of that query that is responsible for the CPU spikes.

Don’t get me wrong, I am not trying to devalue the significance of the MDW reports here when comparing them with the SQLNexus reports. There is so much information in the MDW reports which is absent in the SQLNexus reports and for all practical purposes, MDW is a great feature to analyze generic performance issues in SQL Server. I like to consider the SQLNexus reports as a complement to the MDW reports if I need to dive deeper into the performance issues. I would suggest that you explore the reports presented by SQLNexus and MDW and see for yourself which one of them suits your taste! I love both of them :)

So now that we have talked about the significance of these tools, lets talk about how to use them.

How to use SQLDiag?

On a SQL Server 2005 server, the SQLDiag utility is present under the following folder as SQLDiag.exe file – [systemroot]\Program Files\Microsoft SQL Server\90\Tools\Binn. On a SQL Server 2008 box, SQLDiag is present under [systemroot]\Program Files\Microsoft SQL Server\100\Tools\Binn. SQLDiag uses an XML configuration file which can be used to specify the data that you want to collect. By default, SQLDiag uses the SQLDiag.xml configuration file that gets automatically created in the \Binn directory when you launch SQLDiag for the very first time. This file is configured to collect only the following information -

• SQL Server error logs and configuration information in a file named [servername]_SQL200[x]_sp_sqldiag_Shutdown.out
• Default trace files named Log_[x].trc (Note that these are not regular profiler traces that you want to collect for performance troubleshooting)
• SQLDumper logs (if any)
• System information in a file named [servername]_MSINFO32.txt

You can create you own XML configuration file and configure it to specify the data that you want to collect. You can make a copy of the SQLDiag.xml file and modify it to collect Windows event logs, Perfmon counters, Profiler traces, SQL Blocking information etc. For example, to collect the event logs, enable the EventlogCollector by modifying the tag <EventlogCollector enabled="false" startup="false" shutdown="true" > as <EventlogCollector enabled="true" startup="false" shutdown="true" >. Similarly, you can enable the <PerfmonCollector> to collect specific performance counters and the <ProfilerCollector> to collect profiler traces. Please take a note of few important things when creating your own configuration file -

• Do not modify the default SQLDiag.xml file, because this file gets overwritten every time you run the SQLDiag utility. Instead, copy this file and give it a different name, something like MyCustom_SQLDiag_config.xml
• If you have multiple instances of SQL server running on the same box,  SQLDiag (by default) will collect data from all running SQL Server instances of the same version. So if you launch SQLDiag from the \90\Tools\Binn directory, it will capture data from all running SQL Server 2005 instances on that server. However, you can modify your custom configuration file to specify the name of a specific SQL Server instance for data collection. All you need to do is to replace the tag <Instance name="*" with <Instance name="[InstanceName]" in your custom configuration file. If you want to collect data from only the default instance of SQL Server, modify this tag as <Instance name="MSSQLSERVER".
• If you are running the utility on the active node of a cluster, replace the <Machine name="."> tag with <Machine name="[SQLServerVirtualServerName]">
• The <EventlogCollector> tag allows data collection at two intervals – at startup when you start the SQLDiag data collection or at shutdown when you stop the data collection. You may chose to enable either or both of them, but for most scenarios you may only want to enable the data collection at shutdown (which is the default)
• If you do not wish to collect default traces or SQLDumper logs or system information using Msinfo32 utility, you can disable the appropriate data collectors under the <CustomDiagnostics> tag. Additionally, you can add your own custom data collector under this section.
• The XML configuration file is case-sensitive, so be careful while enabling or disabling specific data collectors. Make sure that when you are modifying the “true” or “false” values, they are in lower case.
• And, don’t forget to save your configuration file after making changes to it!

Once your custom configuration file is ready, you can launch the SQLDiag tool from the command prompt and use the /I parameter to specify the name of your configuration file, something like this -

By default, SQLDiag creates a folder with the name SQLDIAG under the \Tools\binn directory and dumps all the output data files into this folder. However, you can change this default location by using the /O parameter

How to collect SQLDiag data needed to generate the SQLNexus reports

As you may have noticed, some initial efforts are required, such as configuring your own custom configuration file, to capture meaningful performance data with the SQLDiag utility.  If your goal of capturing the SQLDiag data is only to generate the SQLNexus reports, there is an easier way to do it. Follow these steps to collect the SQLDiag data, which is required to generate the SQLNexus reports -

• Click here to download the appropriate scripts files to run SQLDiag. There are two .zip files available for download on this page, PerfStatsSCript2005.zip for SQL Server 2005 and PerfStatsScript2008.zip for SQL Server 2008.
• Extract the contents of the ZIP file on a drive with plenty of disk space available on the concerned SQL Server box. I would suggest that the drive where you plan to collect the diagnostic data should have at least a couple of GBs of disk space available, since the SQL trace files may grow pretty huge depending on the load on your server. After you have started the data collection, you should monitor the size of the trace files to see how fast they are growing and if you have sufficient disk space available on the drive or not.
• Examine the extracted files. You will notice that there are three XML configuration files and the corresponding CMD batch files. For initial troubleshooting purposes, you should use the SQLDiagPerfStats_Trace200[x].xml file and the corresponding StartSQLDiagTrace2008.cmd file. As the names suggest, the other two configuration files are meant for trace replay purposes (if you want to simulate production workload in your test system) and if you are interested in capturing detailed traces (statement level events). Notice that capturing detailed traces can introduce significant overheads on the server.
• The XML configuration files are configured to collect the data from all running instances of SQL Server on the box. If you have multiple SQL Server instances running on the same computer and want to collect data from a specific instance, open the file SQLDiagPerfStats_Trace200[x].xml in Notepad and replace “*" in the tag <Instance name="*" with the actual name of the named instance (portion after the backslash) . If it’s a default instance of SQL server, replace “*” with the word “MSSQLSERVER”. Click on File -> Save to save the file.
• When you are ready to start the SQLDiag data collection, open command prompt (use Run as Administrator options if on Windows Server 2008) and navigate to the folder where you had extracted the perf stats script files. Run the file named StartSQLDiagTrace200[x].cmd. This will start the SQLDiag data collection processes and you should see a prompt saying ‘Data collection has started, press CTRL + C to stop’
• The SQLDiag tool will create a folder with the name SQLDiagOutput inside the same folder where you are running this script from. All output data files are captured inside this folder
• Allow the tool to run for the duration of the performance problems, while SQLDiag captures the diagnostic data. When you are ready to stop the SQLDiag data collection, press CTRL + C.  Do not close the command prompt window while SQLDiag is in the process of shutting down. Doing so can leave orphaned processes on your server. Close the command window only after it returns back the the prompt.
• Examine the files inside the SQLDiagOutput folder. You should see one (or more) Perfmon counter files with extension .BLG, one (or more ) trace files with extension .TRC, three .CSV files one for each Windows event logs and a bunch of files with extension .OUT that contains SQL server error logs, configuration, blocking and wait stats data.

Now that you have captured the SQLDiag data, you are ready to analyze it using the SQLNexus tool.

How to use the SQLNexus tool at analyze SQLDiag data?

The SQLNexus tool requires a SQL Server 2005 or 2008 databases for its analysis. Install a local SQL Server 2005/2008 instance on the machine where you plan to do the SQLNexus analysis, if you don’t have one already. A remote SQL server instance will work okay as well, but I would recommend using a local instance if possible. I would not recommend using the SQLNexus tool on a production server or using the production SQL Server instance for analysis, even if it is possible in your environment. Also note that prior to using SQLNexus, you would need to install the pre-requisite components as discussed in the steps below.

Follow these steps to use the SQLNexus tool for analyzing SQLDiag data -

• Download and install the Report Viewer 2008 Redistributable. This is a pre-requisite for SQLNexus
• Download and install the 32 bit version of RML utilities and here to install 64 bit version of RML utilities. This is a pre-requisite for SQLNexus
• Download the SQLNexus.zip file from here
• Extract the contents of the zipped file in a folder. SQLNexus doesn’t require any installation as such. All it needs to run are the files inside the extracted folder. Since SQLNexus does not get installed like a regular Windows application, you won’t see a shortcut for it under Start –> Programs or on the Desktop. I would suggest creating a shortcut for the SQLNexus.exe in the shortcut pane or on the desktop
• Launch the SQLNexus.exe from the location where you extracted the contents. You will see the following screen that prompts for the SQL server name and the instance name. Type in the instance name that you plan to use for the SQLNexus analysis.

• Once you click on Connect on the above screen, SQLNexus will create a database by the name SQLNexus on the instance that you have specified. If you want to use SQLNexus again to analyze a different set of SQLDiag data, either you would need to delete this database manually, or create a new database from within SQLNexus, which can be done by using the Current DB box on the toolbar and selecting the option <New Database>.
• Click on the Import option under Data menu and browse to the SQLDiagOutput folder that contains the SQLDiag data. Click on the Import button.

• After SQLNexus has completed importing data into the SQL database, you can analyze various reports by using various links under the Reports menu on the top left hand side. These reports are drill down reports and can be used to dive deeper into a particular area. For example to view the top queries by CPU, Duration, Reads and Writes, click on the ReadTrace_Main Report on the left hand side and then click on Unique_Batch_TopN report link. You can also export these reports to Excel, PDF or HTML format using the Export link under Tasks menu on the left hand side.

Note: On several occasions, I had noticed that SQLNexus would fail to analyze the trace files. I have been able to resolve that issue by moving the _Blk.trc file to separate location or deleting it altogether. This trace file captures the blocking chains from the data collection period. SQLNexus uses a different set of scripts based on DMVs for capturing blocking and wait stats information and so this file is not required by the SQLNexus tool.

How to use the PAL tool to analyze performance counters data (.BLG files) collected by SQLDiag?

The PAL (Performance Analysis of Logs) tool is a powerful tool that reads in a performance monitor counter log and analyzes it using known thresholds. Follow these steps to use the PAL tool to analyze performance counters collected by the SQLDiag tool in one or more .BLG files -

• If you are using older Windows versions such as Windows XP/Vista or Windows Server 2003, download and install the following pre-requisites - Microsoft .NET Framework 3.5 Service Pack 1 and PowerShell v2.0 or greater
• Download and install the Microsoft Chart Controls for Microsoft .NET Framework 3.5 from here, which is another pre-requisite for the PAL tool
• Download and install the PAL tool from here
• Launch PAL by clicking on Start –> Programs –> PAL. Click Next on the Welcome screen
• On the Counter Log page, browse and select the SQLDiag.blg file in the SQLDiagOutput folder, and click Next

• On the Threshold file page, select the option “Microsoft SQL Server 2005/2008” from the drop down box and click on Next

• Answer all questions on the Questions page and click on Next
• Select appropriate options on the Output Options, File Output, Queue and Execute pages. For most cases, you can leave these options at their default settings.
• After you click on Finish, a command prompt window will open automatically and close once the PAL the analysis is completed.

PAL analysis may take some time depending on the amount of Perfmon data that you have collected, so please be patient. Once the analysis is complete, PAL reports will open up automatically in a browser window.  By default, these reports are stored under My Documents\PAL Reports.

Here are some additional reference links if you are interested in learning more about the PAL tool -

• Training on PAL tool
• Clint Huffman's Windows Performance Analysis Blog

I hope you would enjoy working with these tools and the wonderful reports that they present! Please feel free to leave me any comments or feedback you may have and I will get back to you as soon as possible

Cheers!

So I decided to write my first notes post on an important and sometimes perplexing topic - Index Fragmentation in SQL Server. I have often found it challenging to memorize types of index fragmentation, their causes and various differences between Reorganizing vs. Rebuilding indexes. In this post I will attempt to provide notes on these aspects of index fragmentation

What is Index Fragmentation: Index fragmentation is an “expected” and “unavoidable” characteristic of any OLTP environment. Fragmentation is defined as any condition which causes more than the optimal amount of disk I/O to be performed in accessing a table, or causes the disk I/Os that are performed to take longer than they optimally would

Causes of Index Fragmentation:  Some of the most common causes of index fragmentation are -

1. Insert and Update operations causing Page splits - One of the most common causes of index fragmentation is the “page-split” operation. A page-split occurs when there is no free space on a data page to accommodate new inserts or updates. As a result, SQL Server will moves ~ 50% data from the current page to the newly allocated data page. This is done to keep the index tree balanced so that SQL Server has to traverse through a predictable number of levels to get to any data page in the tree. This is the reason why SQL server indexes are implemented as b-trees where “b” means “balanced”. Page splits are caused by -

• Non Sequential inserts – when doing a non-sequential insert, SQL Server moves ~50% of data from the old page to the newly allocated page. This would result in a page split, with each page having ~50% of data from the old page.
• Updates to an existing row value with a larger value, which doesn’t fit on the same page
• GUIDs if included in the clustered index key causes index fragmentation in both clustered and non-clustered indexes – see this article for more details

1. Delete operations - Random delete operations, which leave individual pages in use but not completely full of rows will result in Internal Fragmentation. Similarly heavy deletes will cause pages removed to be removed from the page chain resulting in Logical Fragmentation.
2. Initial allocation of pages from mixed extents – this is well demonstrated in this blog post
3. Large row size - for example, if rows are 3200 bytes each, then only two rows can fit on a 8K page, leaving approximately 1600 bytes unused on every page

Various Types of Index Fragmentation:

• Internal Fragmentation –When pages are less than fully used, the part of each page that is unused constitutes a form of fragmentation, since the table’s or index’s rows are no longer packed together as tightly as they could be. This is known as Internal Fragmentation and usually occurs due to -
  • Random deletes resulting in empty space on data pages
  • Page-splits due to inserts or updates
  • Shrinking the row such as when updating a large value to a smaller value
  • Using a fill factor of less than 100
  • Using large row sizes

• Logical Fragmentation  - SQL Server uses 8KB Pages to store data on disk. When a clustered index is created on a table, SQL Server creates a b-tree data structure for the index and links all data pages at the leaf level of the tree in a logical order by using a doubly linked list. Logical fragmentation occurs when the pages in this doubly linked list are not contiguous in the index, meaning that indexes have pages in which the logical ordering of pages, which is based on the key value, does not match the physical ordering inside the data file. This could happen due to -

  • Page-splits due to inserts or updates
  • Heavy deletes that can cause pages be removed from the page chain, resulting in dis-contiguous page chain

• Extent Fragmentation - Extent fragmentation occurs when the extents of a table or index are not contiguous with the database leaving extents from one or more indexes intermingled in the file. This can occur due to:

  • Random deletes, which could leave some of the pages in an extent unused while the extent itself is still reserved as part of the table’s space allocation. Think of it like Internal fragmentation, but in extents instead of pages
  • Deletes on ranges of contiguous rows within the table, causing one or more entire extents to become de-allocated, thus leaving a gap between the surrounding extents of the table or index
  • Interleaving of a table’s data extents with the extents of other objects

Why does Index Fragmentation matters: High fragmentation can affect performance of read-intensive disk-bound workloads. Optimal performance of SELECT queries is only possible with contiguous data pages within the database file, and if the data pages are packed as fully as possible. Internal fragmentation will cause queries to read more data pages in order to retrieve the  same number of records because the pages are not fully packed. Similarly Logical fragmentation and Extent fragmentation will cause the read performance to slow down because SQL Server can’t perform Read-Ahead reads. The SQL Server Read-Ahead manager dynamically adjusts the size of reads it performs based on the physical ordering of the underlying pages. With high Logical and Extent fragmentation, the read-ahead manager must read smaller blocks of data. A few important things to note about the effects of index fragmentation on SQL Server performance - 

• Fragmentation only affect performance if the pages have to be read from physical disk (aka physical read). However, if the pages being read are already in buffer pool cache (aka logical read), fragmentation does not affect query performance
• Fragmentation happens because of DML operations. So if there is very little DML activity on a table, it may not get fragmented at all. 
• Fragmentation only impacts range queries and not the singleton SELECT queries.
• Reducing or eliminating Internal fragmentation can help you save disk space.

How to detect Index Fragmentation: Use the sys.dm_db_index_physical_stats DMF, that replaces the deprecated DBCC SHOWCONTIG command. This DMF has three modes -

1. DETAILED - reads all data and index pages. Be careful with using this options since it causes the entire index be read into memory and may result in IO/Memory issues
2. SAMPLED: reads 1% of the pages if more than 10,000 pages
3. LIMITED: only reads the parent level of b-tree (same as DBCC SHOWCONTIG WITH FAST). Limited option doesn't report page density, since it does not read the leaf level pages

Pay attention to the following columns in the output of this DMF -

• Avg_fragmentation_in_percent – depicts logical fragmentation
• Avg_page_space_used_in_percent – depicts internal fragmentation

Solutions to address Index Fragmentation: Main choices are -

• Recreate - CREATE INDEX WITH DROP_EXISTING – Not a good choice because of possible issues with constraints (Foreign Keys etc.)
• Rebuild - ALTER INDEX ... REBUILD (replaces DBCC REINDEX)
• Reorganize - ALTER INDEX ... REORGANIZE (replaces DBCC INDEXDEFRAG)
• To reduce the extent fragmentation of a heap, create a clustered index on the table and then drop the index

Reorganize vs. Rebuilding indexes

# 1 - DBCC INDEXDEFRAG is an online operation; therefore, the table and indexes are available while the index is being defragmented. DBCC DBREINDEX is an offline operation. When DBCC INDEXDEFRAG is run on non-quiescent systems on which updates to the underlying data are taking place, DBCC INDEXDEFRAG skips locked pages as it encounters them. As a result, DBCC INDEXDEFRAG may not be able to completely eliminate fragmentation. SQL Server 2005 introduces the ability to rebuild your indexes in an online fashion so that other processes are able to access the table while the rebuild is occurring. Because you can access the indexes during the rebuild, you are not limited to only rebuilding indexes during off-peak hours. For more information on online index operations, please refer to this article

# 2 - REORGANIZE attempts to raise the page-density level of pages to the original fillfactor. DBCC INDEXDEFRAG does not, however, reduce page density levels on pages that currently have a higher page density than the original fillfactor.

# 3 - Rebuilding indexes is usually faster then reorganizing indexes if the logical fragmentation is high. A general rule of thumb is to rebuild indexes when the average logical fragmentation is more than 30%

# 4 - DBCC INDEXDEFRAG can be stopped and restarted without losing any work. The entire DBCC DBREINDEX operation runs as one atomic transaction. This means if you stop DBCC DBREINDEX the entire operation is rolled back, and you must start over. However, if you stop DBCC INDEXDEFRAG it stops instantly and no work is lost, because each unit of work performed by DBCC INDEXDEFRAG occurs as a separate transaction.

# 5 - Note that while Rebuilding indexes will automatically rebuild all index statistics (which is equivalent to a full scan), it does not update column statistics created by Query Optimizer (statistics names starting with _WA in sys.stats view) automatically.

# 7 - Interleaving occurs when index extents (a group of eight index pages) for an index are not completely contiguous within the data file, leaving extents from one or more indexes intermingled in the file. Interleaving can occur even when there is no logical fragmentation, because all index pages are not necessarily contiguous, even when logical ordering matches physical ordering.

# 8 - When in full recovery mode, Alter Index REBUILD logs images of each index page, which does not occur when in bulk logged mode. For this reason, in full recovery mode, the log space required by the REBUILD operation is roughly equivalent to the number of index pages multiplied by 8 KB. On the other hand  the amount of work that Alter Index REORGANIZE performs when defragmenting indexes is dependent on the number of page movements and the amount of page compaction necessary. You can reclaim the log space used by the REORGANIZE operation by backing up the log because the work performed is a series of small transactions

# 9 - Rebuilding indexes requires adequate free space in the data file(s). With not enough free space in the data file(s), re-index operation may be unable to rebuild the indexes, or the indexes may be rebuilt with logical fragmentation values above zero. Without large enough contiguous blocks of free space, rebuilding indexes may be forced to reuse other areas of free space within the data files, resulting in indexes being rebuilt with a small amount of logical fragmentation

References:

• Index Fragmentation – video from Paul Randal
• Microsoft SQL Server 2000 Index Defragmentation Best Practices
• Fragmentation and Database Performance
• Example: Index fragmentation with insert/updates, measuring it and fixing it
• Can GUID cluster keys cause non-clustered index fragmentation?
• Fragmentation (part 1): What are records?
• Fragmentation (part 2): What are pages?
• Fragmentation (part 3): What are extents?
• Fragmentation (part 4): what are heaps?
• SQL Server 2008 Fragmentation – Jose Barreto’s notes
• sys.dm_db_index_physical_stats
• Reorganizing and Rebuilding Indexes
• How Online Index Operations Work
• Reading Pages

First of all let’s talk about what Attentions events are and what may cause them. According to BOL –

"The Attention event class indicates that an attention event, such as cancel, client-interrupt requests, or broken client connections, has occurred. Cancel operations can also be seen as part of implementing data access driver time-outs"

There are three most common reasons of Attention events that I have seen in my experience so far. In each case, Attention events were causing open transactions and resulting in massive blocking issues. Here are those three scenarios -

1. Query Cancellation

User may cancel the currently executing query/batch request at any time - may be the user got tired of waiting on the screen! If the query or the batch was in an explicit user transaction (BEGIN TRAN …. END TRAN), these Attention events could result in open transactions and severe blocking problems. We can reproduce this scenario from within SQL Server Management Studio. Try the following -

• Run a SQL Server Profiler trace with the following events selected - Attention, SQL:BatchCompleted, SQL:BatchStarting, SQL:StmtCompleted, SQL:StmtStarting

• Open Management Studio and run the following batch –

 USE TEMPDB 
 GO 
 
 IF EXISTS (SELECT * FROM SYS.OBJECTS WHERE OBJECT_ID = OBJECT_ID (N'[DBO].[TEST]') AND TYPE in (N'U')) 
 DROP TABLE [DBO].[TEST] 
 GO 
 
 --Create a Test table and Insert few records in it 
 
 CREATE TABLE TEST (C1 INT, C2 VARCHAR (100)) 
 GO 
 
 INSERT INTO TEST VALUES (1, 'AAA') 
 GO 
 INSERT INTO TEST VALUES (2, 'BBB') 
 GO 
 
 -- Simulate a user transaction that will take a several seconds to execute 
 -- we will use the WAITFOR command to simulate such a transaction 
 
 BEGIN TRAN 
 UPDATE TEST SET C2 = 'MODIFY1' 
 WHERE C1 = 1 
 WAITFOR DELAY '00:00:45' 
 COMMIT TRAN

• While the transaction is in progress, click on the Cancel option (red square button on the toolbar) -

• You will notice an Attention event in Profiler trace.

• In the trace, you will also notice that the COMMIT TRAN statement never executed and so the transaction is still open. You can confirm that by running the following statement from the same query window in which you had cancelled the query

SELECT @@TRANCOUNT 
GO

• Now let’s see what kind of locks are being held by this open transaction. Run a select query on the sys.dm_tran_locks DMV (you may want to filter the result on your session to exclude extra noise). You should notice an exclusive (X) lock on the row (RID), and Intent exclusive (IX) locks on the Page and the table. So with those locks in place, if another user starts a query that needs a conflicting lock on any of these resources, you will see blocking. For example, if another user/transaction/query tries to SELECT/UPDATE/DELETE the same row, it will get blocked by the open transaction. Try running this SELECT query in a another query window

SELECT * FROM TEST WHERE C1=1 
GO 

• Open the “Activity - All Blocking Transactions” server report to analyze the blocking chain (right click on your SQL instance in Object Explorer -> Reports -> Standard Reports -> Activity – All Blocking Transactions)

So the question is how long will this blocking go on for? Actually this blocking chain will go indefinitely until one of the following events occurs –

• The open transaction is committed or rolled back manually, since we just saw that SQL server will not automatically commit or rollback the transaction on Attention events.
• The connection (or SQL Server session) that started the transaction is terminated. This will happen when the user disconnects from the application.
• The connection is recycled in a connection the pool, if the application is using connection pooling.

2. Query Timeouts:

This is the second most common reason I have seen in my experience, which causes Attention events, open transactions and blocking issues. The default query timeout in SQL Server is 0, which means that a query will run indefinitely until completion. However, application developers can change the default query timeout value in SQL Server connection string. Also, OLEDB and ODBC providers of SQL Server have non-zero query timeout values as discussed in this article – DBA’s Quick Guide to Timeouts. Query timeouts, if not handled properly, can cause blocking issues in SQL Server. Let’s see how that can happen -

• Open a new query editor window in Management Studio that will prompt you to specify the authentication details. You can do that by clicking on the File menu –> New –> Database Engine Query. On the Connect to Database Engine screen, click on the Option >> button and change the Execution time-out setting from default of 0 to 15. The default value of 0 means the query will run indefinitely. Here is a snapshot-

• Restart the Profiler trace and run the following batch from the query window that you just opened –

 USE TEMPDB 
 GO 
 
 -- Run a TSQL batch request that would take a few seconds to execute 
 -- We will use the WAITFOR command to simulate such a query 
 
 IF EXISTS (SELECT * FROM SYS.OBJECTS WHERE OBJECT_ID = OBJECT_ID(N'[DBO].[TEST2]') AND TYPE IN (N'U')) 
 DROP TABLE [DBO].[TEST2] 
 GO 
 
 --Create a Test table and Insert few records in it 
 
 CREATE TABLE TEST2 (C1 INT, C2 VARCHAR (100)) 
 GO 
 
 INSERT INTO TEST2 VALUES (1, 'AAA') 
 GO 
 INSERT INTO TEST2 VALUES (2, 'BBB') 
 GO 
 
 --Start a user transaction that would take about 10 seconds to complete
 
 BEGIN TRAN 
 UPDATE TEST2 SET C2 = 'MODIFY1' 
 WHERE C1 = 1 
 WAITFOR DELAY '00:00:45' 
 COMMIT TRAN

After 15 seconds, which is the query timeout value that we had set while connecting to the server, the batch will terminate with the following timeout error –

Msg -2, Level 11, State 0, Line 0 Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

• You will also notice an Attention event in the trace. If you check the open transaction count by running the SELECT @@TRANCOUNT query from the same query window in which you were running the batch, it will return 1, meaning that there is one open transaction. If you run a SELECT query on the sys.dm_tran_locks DMV, you will notice the same set of locks being held by the open transaction as you saw in the first scenario. If the application is not handling the query timeout errors and rolling back open transactions, it will result in blocking.

What about Lock timeout errors? Will they also result in Attention events?

The answer is NO! Lock Timeout errors do not cause Attention events and execution will proceed to the next statements in the transaction. See it for yourself!

• Run the following batch from a new query window (if you are using an existing query window that is already open, make sure to roll back all open transactions in that window) -

  
   

   USE TEMPDB 
   GO 
   
   -- Create a Test table and Insert some rows in it 
   IF EXISTS (SELECT * FROM SYS.OBJECTS WHERE OBJECT_ID = OBJECT_ID(N'[DBO].[TEST3]') AND TYPE IN (N'U')) 
   DROP TABLE [DBO].[TEST3] 
   GO 
   
   CREATE TABLE TEST3 (C1 INT, C2 VARCHAR (100)) 
   GO 
   
   INSERT INTO TEST3 VALUES (1, 'AAA') 
   GO 
   
   INSERT INTO TEST3 VALUES (2, 'BBB') 
   GO 
   
   -- Run a TSQL batch request that will start a 
   -- transaction but do not COMMIT or ROLLBACK the transaction 
   -- I am doing this to create a classic blocking 
   -- scenario to test lock timeout errors 
   
   BEGIN TRAN 
   UPDATE TEST3 SET C2 = 'MODIFY1' 
   WHERE C1 = 1 
   GO

   

• Restart the Profiler trace and run the following batch in a second query editor window –

USE Tempdb
 GO
 
 SET LOCK_TIMEOUT 5000 
 GO 
 
 BEGIN TRAN 
 UPDATE TEST3 SET C2 = 'MODIFYAGAIN' 
 WHERE C1 = 1 
 COMMIT TRAN

• Expectedly, after 5 seconds, the batch will terminate with a lock timeout error (Error 1222) –

Msg 1222, Level 16, State 45, Line 3 Lock request time out period exceeded. The statement has been terminated.

• However, you will not see an Attention event in the Profiler trace this time. You will also notice that after the lock timeout error occurred, SQL server continued with the execution of the batch and finally committed the transaction even though the UPDATE statement had failed. If you run a SELECT @@TRANCOUNT in the same query window, it will return 0, meaning there is no open transaction

3. Applications not processing the result set completely

I have seen this problem while working with a customer who had an ODBC application fetching data from SQL Server. The application was coded to ask for a result set in a cursor, fetch the first few records (and not all of the requested rows) and discard the rest. This problem has been described in detail in this blog. Unfortunately, I don’t have an application to demonstrate this behavior. But if you are noticing the following characteristics with your application and seeing a large number of Attention events and blocking chains in SQL Server, this could be the case.

• You notice a large number of Attention events in the a SQL Server profiler trace and you have confirmed that they are not caused by user cancellation or query timeouts
• You are seeing a pattern in the trace showing that the Attention event arrived right after the batch started, generally milliseconds afterwards and not something like a 30 second query timeout. Here is a snapshot of this pattern that I was seeing in SQL traces -

• You are seeing ASYNC_NETWORK_IO waits in SQL Server, which is an indication that the application is not processing the result set completely.

Solutions:

So now that we know how Attention events can result in open transactions and blocking problems, what can we do to resolve this problem? Here are two solutions that I have successfully tried while working with customers -

• Check after each transaction (or before starting a new transaction in the same session) to see if the transaction is complete by using the following statement:

IF @@TRANCOUNT > 0 ROLLBACK

• Use SET XACT_ABORT ON for the connection, or in any stored procedures which begin transactions and are not cleaning up following an error. In the event of a run-time error, this setting will abort any open transactions and return control to the client. Note that T-SQL statements following the statement which caused the error will not be executed. It’s a good practice to set XACT_ABORT to on before starting any user transactions so that any batch terminating error (such as Attention events) would roll back the entire transaction. If you can’t modify the application easily to specify XACT_ABORT, you can try the USER_OPTIONS configuration setting in SQL Server to turn on XACT_ABORT on an entire instance of SQL Server. The below snapshot shows how to turn on XACT_ABORT on an instance of SQL Server –

Important: This (or setting it at the connection level) does not guarantee desired behavior in case of an Attention event though, since application code can override the setting. The only completely reliable way would be to set XACT_ABORT to ON before every BEGIN TRAN

Before starting SQL Server installation

1. Configure MSDTC as per best practices

It is required to configure the MSDTC as a clustered resource on a Windows 2003 Failover cluster to be able to install SQL Server 2000/2005. If you do not cluster MS DTC prior to installing SQL Server 2000/2005 on Windows Server 2003, you may notice errors when using SQL Server features that depend on MSDTC (such as some Replication functionality). SQL Server 2008 and R2 do not have any dependency on the MSDTC service anymore and so you don’t need to configure MSDTC as a clustered resource before installing SQL Server 2008 and R2. However, if your applications require MSDTC (which you can confirm with your application vendor or your in-house development team) or if you are using SQL Server features that always require MSDTC such as Linked Servers, then MSDTC service must be configured as a clustered resource on Windows Server 2003. There are many enhancements to the MSDTC service in Windows Server 2008/R2 that you can take advantage of. To list a few of them –

• In Windows Server 2003, you could only have one MSDTC clustered resource, which could cause Isolation and Performance issues. In Windows Server 2008/R2, you can have multiple MSDTC clustered resources on a multi-instance (Active-Active) cluster.
• In Windows Server 2003, we didn’t have the ability to run MSDTC as a standalone service on a Windows Failover cluster. In Windows Server 2008/R2 you can do that.

If your application uses distributed transactions or Linked servers, you might encounter various error messages if the MSDTC is not configured correctly. Sometimes application dependency on the MSDTC is not very clear, and so I usually suggest my customers to configure the MSDTC service on a cluster as per best practices. The recommended configuration parameters for MSDTC service are discussed in these articles –

• Enable Network Access Securely for MS DTC
• Enable Network DTC Access

2. Pre-stage MSDTC and SQL Server account objects in Active Directory prior to installing a MSDTC or SQL server cluster

You might run into issues while configuring a cluster resource or bringing it online, if the computer object of the cluster service does not have permissions to create computer objects in the Active Directory. If you are experiencing this problem, you will also notice the event ID 1194 in the event logs. Here is the explanation -

When you create a new clustered service or application, a computer object (computer account) for that clustered service or application must be created in the Active Directory domain. This computer object is created by the computer object of the cluster service itself. If the computer object of the cluster service does not have the appropriate permissions, it cannot create or update the computer object for other clustered service or application such as SQL Server, MSDTC etc. This can result in an error with event ID 1194, while configuring the cluster resource or bringing it online.

Therefore it’s important to pre-stage a computer object for the new clustered service or application in Active Directory before configuring that application (such as SQL Server and MSTDC).

3. Benchmark Disk Performance

It’s no secret that the I/O system is very important to ensure good SQL Server performance. When configuring a new server for SQL Server or when adding or modifying new disks to an existing system, it is good practice to determine the capacity of the I/O subsystem prior to deploying SQL Server. Tools such as SQLIOSIM and SQLIO can be used for storage benchmark testing and capacity planning. Here are some additional references on these tools –

• How to use the SQLIOSim utility to simulate SQL Server activity on a disk subsystem
• SQL Server Storage Engine team blog on SQLIOSIM
• SQLIOSim - Outstanding I/O Warnings
• How It Works: SQLIOSim - Running Average, Target Duration, Discarded Buffers
• SQLIOSIMParser
• SQLIOSim is "NOT" an I/O Performance Tuning Tool
• Tools used to test I/O subsystems

4. Use Slip Streaming or install the latest support files prior to starting the SQL Server installation

Slipstreaming is a process that incorporates all necessary Service Packs and hotfixes into a folder that contains the complete, original installation files. It is recommended to use the slipstreaming to install SQL Server to ensure that the new SQL Server instance is fully updated and ready to go, saving significant time. Slip streaming also installs the latest setup support files on the box, which helps avoid any bugs/known issues with the SQL Server installation engine. For more information on how to slip stream SQL Server binaries, please refer to these article -

• How to update or slipstream an installation of SQL Server 2008
• Creating a merged (slipstreamed) drop containing SQL Server 2008 RTM + Service Pack 1

If using Slip streaming is not an immediate option (such as due to time constraints to create the merged folder), it is recommended to install the latest setup support files in order to fix the SQL Server 2008 Setup even before you run it. For example, use the SQL Server 2008 SP2 to install the SP2 setup support files and then start the SQL Server 2008 installation wizard. This will help you avoid bugs and known issues with the SQL Server installation engine. More information on this can be found in this article - How to fix your SQL Server 2008 Setup before you run setup

5. Ensure that the account used to install SQL Server cluster has the appropriate permissions

I was working with a customer recently, helping them in installing a SQL Server 2008 R2 cluster on Windows Server 2008 R2 and we were getting a lot of permission errors during the installation. The setup account was a member of Local Administrators group on all nodes in the cluster. After troubleshooting the errors, we discovered that in addition to local administrative permissions, the setup account must have these additional permissions on each node in the cluster, to successfully install SQL Server -

• Backup files and directories (SeBackupPrivilege)
• Debug Programs (SeDebugPrivilege)
• Manage auditing and security log (SeSecurityPrivilege)

This issue is described in detail in this KB article - SQL Server 2008 installation will fail if the setup account does not have certain user rights. We granted the above permissions to the logged on user account manually through the Local Security Policies administrative tool and were able to complete the installation successfully. I suggest that you verify the setup account permissions as described in the KB article prior to starting the SQL Server cluster installation, to avoid any setup failures.

6. Ensure that the SQL Server account should not be a member of Local Administrators group

It is not recommended to assign the local Administrative permissions to the SQL Server and/or SQL Server Agent service account. SQL Server and Agent services needs specific permissions on the box, which are granted to the service accounts automatically during installation or when the service account is changed through SQL Server Configuration Manager tool. For more information on specific permissions required the by SQL Server services, please refer to the following article - Setting Up Windows Service Accounts

During SQL Server Installation

1. Use Service Identifiers (SIDs) instead of Domain groups

SQL Server enables per-service SID for each of its services on Windows Server 2008 and R2 systems in SQL Server 2008 and SQL Server 2008 R2 to provide service isolation and defense in depth. The per-service SID is derived from the service name and is unique to that service. Service SID name for a default instance of SQL Server is NT Service\MSSQLSERVER and SQL Server Agent Service is NT Service\SQLSERVERAGENT. Service SID name for a named instance of SQL Server is NT Service\MSSQL$<InstanceName>. Service isolation enables access to specific objects without the need to run a high-privilege account or weaken the security protection of the object. By using an access control entry that contains a service SID, a SQL Server service can restrict access to its resources. For more information, please refer to this article - Setting Up Windows Service Accounts

After SQL Server Installation

1. Implement antivirus exclusion best practices:

Besides contending for precious system resources such as CPU and memory, antivirus programs can cause issues during SQL Server startup and in some cases may also cause database corruption. This could happen during system startup, if an antivirus program gets a lock on a file (such as MDF, LDF etc.) before SQL Server does. It’s true that antivirus programs have evolved quite a bit in the last few years and this is less of an issue these days. Nonetheless it’s a best practice to exclude SQL Server related directories and files from Antivirus scanning, wherever possible. In addition to excluding the MDF, LDF and NDF database files, antivirus exclusion list should be configured to exclude these additional directories on a Windows Failover cluster –

· Quorum drive

· <system root>:\Windows\Cluster

Please refer to this KB article for more information on how to configure an antivirus program that is running on a SQL Server box - Guidelines for choosing antivirus software to run on the computers that are running SQL Server

2. Run SQL Server 2008 BPA tool to check for other best practices

SQL Server Best Practices Analyzer (BPA) is a free downloadable tool that can be used to scan a SQL Server instance for common best practices and other stability issues that could result in server-down scenarios. This tool encapsulates a variety of checks to prevent most common SQL server issues, which are received by Microsoft  support. The best part is that new test cases are added to the tool on a frequent basis to help our customers prevent most common SQL Server issues, which other customers are experiencing. The BPA tools is available for all versions of SQL Server. Here are the download links for each version -

• BPA for SQL Server 2000
• BPA for SQL Server 2005
• BPA for SQL Server 2008 and R2

3. Configure SQL Server memory settings appropriately on a multi instance cluster

By default, every SQL Server installation has “Max Server Memory” parameter set to use all available physical memory. This could raise problems when multiple instances are present on the cluster. It is important to cap SQL server memory by configuring the “max server memory” configuration option for each instance on a multi instance cluster. This would ensure that each SQL Server instance will get sufficient amount of memory if both instances failover to the same node in the cluster during the planned or unplanned failover. Note that “max server memory” setting only specifies the Buffer Pool memory. SQL Server as a process acquires more memory than specified by “max server memory” option. Both internal and external components can allocate memory outside of the Buffer Pool and this additional memory consumed by SQL Server is known as Multi Page Allocator (MPA) or Mem to Leave (MTL). The Performance Monitor counter SQL Server:Memory Manager\Total Server Memory (MB) can be used to find the Buffer Pool memory consumption of a SQL Server process. Another Performance Monitor counter, Process(sqlservr)\Private Bytes can be used to know the total memory consumption by a SQL Server process. The difference of these two counters can be used to calculate the MPA/MTL memory consumption by a SQL Server instance. MPA/MTL memory is used by components as such threads (usually the biggest consumer on a busy production server), linked servers, COM components, extended stored procedures etc. Following formula can be used to calculate the “max server memory” for each instance in a multi instance cluster –

SUM (max server memory for “resident” instance/s) + MPA for all instances <= RAM –(OS memory + Applications)

where “resident” instance/s refers to the total number of instances that could be running on a single node of the cluster keeping node failure into consideration.

For example on a 2 node Active-Active cluster, “resident” instances would be 2 because if a single node fails, then both instances would failover to the same node. Let’s assume that each node has 24 GB of RAM, OS needs 2 GB of dedicated memory, other applications needs 2 GB , MPA for each SQL Server is 1 GB and both instances need equal amount of memory. So how much “max server memory‟ should you give to each instance in case of a node failure? Let’s say max server memory for each instance is X GB. Using the formula above

SUM (max server memory for “resident” instance/s) + MPA for all instances <= Total Available Memory –(OS memory + Applications)

(X+ X) + 1GB * 2 <= 24  – (2GB + 2GB)

X <= 9 GB

i.e. “max server memory” for each SQL Server instance should be configured as 9GB in this scenario. For more information on SQL Server memory configuration, please refer to this article – Server Memory Options

4. Consider adding a third passive node to the 2-node cluster

As discussed above, when using the Active-Active configuration on a 2 node cluster, each instance is configured to use far lesser amount of memory than what is available on each node. For example on a 2 node cluster with 24GB of physical memory on each node, assuming that the OS and other application need 4 GB and the MPA for each SQL instance is 1 GB, only 9 GB of memory can be allocated to each instance. However, if we add a third passive node to this cluster, the number of “resident” instances becomes 1 instead of 2. In that case, we can allocate as much as 19 GB of memory to each instance by using the above formulae. Adding a third node would also provide an added advantage of having additional resources such as CPU, network and I/O on the third passive node to address planned/unplanned failover without causing performance degradation.

 If you are using Mount Points

1. Mount Point Best Practices

SQL Server 2005 and later versions failover clustered instances fully support mounted drives if the mounted drive is hosted by a cluster drive with a drive letter assigned. When mount points are being used together with the physical disks, each mount point must appear as a cluster resource. Additionally, each mount point must also be added as a SQL Server dependency. If only the root physical disks dependency is added and the mount points are not added, database corruption will occur on failover. Database corruption may also occur when SQL Server is restarted without failing over. For more information please refer to these articles -

• SQL Server support for mounted volumes
• How to configure volume mount points on a Microsoft Cluster Server

To avoid any issues during SQL Server installation, it is recommended to verify that Mount Points are configured and working properly. To verify that mount points are working correctly, try copying/creating files on mounted volumes. Also make sure that the root volumes and the mounted volumes appear as “Available Storage” in the Failover Cluster Manager and rename the mounted volumes to display their logical function.

2. While configuring the data directories in the SQL Server setup, do not use the root mount drive (Master drive) to store any data/log/backup files

When using mount points, it is not recommended to use the root (host) volume to store files and folders. The root volume is the volume that is hosting the mount points. Try to use the root (host) volume exclusively for mount points. This greatly reduces the time that it takes to restore access to the mounted volumes if you have to run a chkdsk. This also reduces the time that it takes to restore from backup on the host volume. If you use the root (host) volume exclusively for mount points, the size of the host volume only has to be several MB. This reduces the probability that the volume is used for anything other than the mount points

Below are more details on this scenario and a few workable solutions –

 
Scenario:

You have a table in your database that has the following attributes –

1. Few records in the table (anywhere from 1 record to a few thousand records)
2. You are using UPDLOCK hints with SELECT queries on this table in explicit user transactions (BEGIN TRAN… COMMIT TRAN)
3. The SELECT queries have a WHERE clause on a column in the table and this columns either does not have any index defined on it or has a non-unique, non-clustered index on it
4. You see blocking chains where multiple transactions on this table with SELECT and UPDLOCK are fighting for the same KEY resource (the same key hash value)

Steps to reproduce this scenario:

-- Create a Test database
Create Database Test
Go

-- Create a table that has the attributes as discussed above
Use Test
Go

Create Table Mytable (Col1 int not null, Col2 varchar(100) null, Col3 bigint null)
Go

-- Create Indexes 
Create Unique Clustered Index IdxPrimary on Mytable (Col1)
Go

Create Index #IdxNC on Mytable(Col3)
Go

-- Insert a few records into this table
Insert into Mytable values (1, 'AAA', 50)
Insert into Mytable values (2, 'BBB', 60)
Insert into Mytable values (3, 'CCC', 70)
Insert into Mytable values (4, 'DDD', 80)
Insert into Mytable values (5, 'EEE', 90)
Go

-- Start a transaction in a separate query window, say this comes as SPID X in SQL Server
Begin Tran Select * from dbo.Mytable with (UPDLOCK) where Col3 = 50


-- Start another transaction in a second query window, say this comes as SPID Y in SQL Server
Begin Tran Select * from dbo.Mytable with (UPDLOCK) where Col3 = 60

What you will notice is that SPID X is blocking SPID Y. If you look at the query plan for SPID X in SQL Server Profiler, you will notice that SQL Server has chosen to do a clustered index scan, although there is a non-clustered index on the filter column Progress –

Clustered Index Scan(OBJECT:([TEST].[dbo].[Mytable].[IdxPrimary]), WHERE:([Test].[dbo].[Mytable_repro].[Col3]=(50))

This clustered index scan is causing the second transaction (SPID Y) to get blocked by the first transaction (SPID X). You may be wondering that even though SQL Server has chosen the clustered index scan, The locked resource is still a KEY resource (and not the entire table), so why are we seeing blocking. Here’s what’s happening -

SPID X start scanning the table and it checks for every row in the table to see if it matches the criteria specified in the WHERE clause. Remember that it’s doing that because the SQL Server has chosen not to use the non-clustered index on our filter column, so it doesn’t know which rows would qualify for our filter criteria (Col3 = 50). Once SPID X finds the row (or all the rows) matching the filter criteria, it locks that row (or rows) with an Update (U) Lock. So now, when SPID Y starts scanning the clustered index, it has to wait on the key that is locked by SPID X because, it has to verify whether or not the locked row matches the criteria specified in the WHERE clause. The result is that SPID X continues to hold the update lock while the transaction is in progress, while SPID Y waits on SPID X to release the Update (U) lock on that key, so that it can continue with the scan.

So the question is why SQL Server is doing a Clustered Index Scan instead of using the non-clustered index that exists on filter column (Col3 in our scenario)? SQL Server has a cost-based query optimization engine. The goal of SQL Server Query Optimizer is this – out of many (and not all!) possible query plans, discover a plan that has the least amount of cost associated with it, with cost being a function of CPU, Memory and IO.  When Query Optimizer evaluated the query plans for our SELECT query, it found that the plan with clustered index scan has a lower cost when compared to the plan with a seek on the non-clustered index followed by the lookup on the clustered index.

Workarounds:

Here are a couple of workarounds to fix this issue by ensuring that the Query Optimizer will select the expected non-clustered index on the filter column. You can choose any one of these depending on your application requirements.

• Make the Non Clustered Index a Covering Index and include all columns referenced in the query in the index as INLCUDED columns:
• Include the filter column in the Clustered index key
• Populate the table with dummy records so that QO can pass the threshold, where it can use the Non Clustered Index on the filter column
• Use the FORCESEEK query hint in your blocking query to force the index seek.
• Use Plan Guides with the Index hint specifying the name of the NC index or with the FORCESEEK hint
• Make the Non Clustered Index a Unique index

In our case, we worked with the ISV vendor and their partner to enforce the FORCESEEK query hint with all of their SELECT, UPDATE and DELETE queries in their application to fix this issue.