Memory Manager

We have a new memory manager in SQL 2012. The new memory manager is responsible for almost all the memory management related activities, especially memory allocation. It includes 2 allocators, the Page Allocator, and the Virtual Address Space (VAS) allocator.

Memory consumers request memory from the memory manager through these allocators. The new memory manager supports allocations of all sizes, we no longer have the concept of single page and multipage allocators. For the sake of simplicity, I will refer to the allocator as "Any Size Page allocator" in this post. Consumers like memory clerks and the Buffer Pool are clients of the memory manager. This is a major change from previous versions, where the Buffer pool was a consumer as well as a provider(allocator) of memory (single pages).

Another major change that you will experience is that with the introduction of the Any Size Page allocator, the SQL Server "Max Server Memory" setting now controls the memory allocated for the Buffer Pool, CLR, as well as larger than 8 kb (single page) allocations. In prior versions, max server memory only controlled the size of the buffer pool, which made it difficult for DBA's/architects to apportion memory between applications/instances on shared servers. SQL Server 2012 gives us the ability to establish much tighter control on memory allocation using tools such as resource governor, because the resource governor is now able to control all page allocations.

The buffer pool in SQL 2012 does not contain any memory management functionality. It just manages the caching of database pages, and is now treated like a regular, external cache. Considering that the Buffer Pool is still likely to be the largest consumer of memory though, it is allowed to use almost all free memory from the SQL memory manager.

There are several visual representations of the changes to memory manager in SQL 2012 floating around on the internet, which I've included for your benefit:

As you might have noticed, memory allocation for CLR forms a bit of a "special case", in that even though it uses VAS Allocator(Virtual Allocator in the diagram), it still is governed by the limits set using the "max server memory" setting in SQL. An auxiliary effect of this is that CLR is now initialized at SQL Server DB Engine startup, as opposed to being initialized when it was called in earlier versions of SQL Server.

The memory manager architecture in SQL Server 2012 can be broken down into the following components:

1. Fragments/Fragment Manager: A fragment is a large region of VAS. The fragment manager deals with fragments, which the memory manager will commit dynamically.
2. Top level block allocator: The Top level block allocator takes these large fragments and splits them up into top level blocks. These top level block sizes are fixed at 16 MB in 64-bit installations and 4 MB in 32-bit installations.
3. Workspace: A Workspace is a set of top level blocks. A workspace contains allocations of similar lifetime. Examples of workspaces include buffer pool, memory objects, etc.
4. Fixed Size Allocator: The most crucial component of the new memory manager hierarchy is the Fixed Size Allocator. This allocator is part of the workspace, and it gets memory from the top level block allocator, and breaks it up into smaller sized blocks. These are called parent blocks, and each parent block has its own descriptor, which has a free list as well as the current state of the block. The parent block can be in one of four states:

• Active - The block from which allocations happen. New allocations requests are serviced from the Active block of the local CPU or the NUMA node.
• Full - The block is full of allocations. There is no more free space in that block. This was previously an Active block.
• Partial - The block is neither Full nor Active. There are multiple lists of Partial blocks. These lists are per NUMA node. A block is made part of a specific list according to the fill factor of allocations inside the block. This supports the mechanism to try to Free blocks which have less allocations sooner or try to allocate from blocks which have more allocation already to make it full. Thus, the overall algorithm pushes against having a large list of Partial blocks to manage.
• Empty - When the block is allocated for the first time, it is in empty state. It is does not have any allocations. 

Memory Allocation

Let's talk about how memory is allocated using the new memory manager. The first step is, of course, to determine which allocator to use, based on the amount of memory requested. This involves traversing the workspace hierarchy to find the appropriate allocator. If we're unable to find an allocator of the exact same size, the next highest size allocator is used.

Once we have the allocator, the fastest way to allocate is to look at the local partition, find the Active block and then pop an entry from the Free list. If the Active block does not have enough free space to allocate from, then we need to replace this Active block with one that has enough free space.

Next, we start traversing the partial buckets, looking for a partial block that is almost full. If a partial block is found, we first re-evaluate the fill factor, since many free operations could have happened after this partial block was inserted into this fill factor bucket.  If the fill factor has changed, then this partial block is inserted to the appropriate fill factor bucket. Repeat this process to find a partial block with the most allocations in it. Once found this partial block is installed as an Active Block. After this the allocation is performed from this Active block.

During the above process, if another thread installs another active block, then that Active block should be used for allocation instead. If allocation is successful from that active block, then this block should be returned to the partial bucket where it came from.

If no partial block was found, then go up to the parent block allocator and request a new block [empty] to be created. This corresponds to the process of growing against the target memory configured [for conventional and locked page memory].  

If during the above process we were not able to find a suitable block, then we switch to the next NUMA node and repeat the same process. Switching to next node happens once the target for current node is reached. If all nodes reach target memory then an OOM (Out Of Memory) error is returned.

In order to free a block, we first find the parent block descriptor for the current block that is being freed. Next, we return the block to the free list and update the state in the descriptor. It can go to two states, Partial or Empty.

When we configure SQL to use large pages, all the top level blocks are committed at startup. For conventional and locked page memory model, the blocks are committed as needed.

The rest of the components work in pretty much the same way as earlier versions of SQL Server. For more details on the memory architecture up to SQL Server 2008 R2, please refer the following posts:

An in-depth look at SQL Server Memory–Part 1

An in-depth look at SQL Server Memory–Part 2

An in-depth look at SQL Server Memory–Part 3

As always, comments and feedback are appreciated.

Disclaimer: The information in this weblog is provided “AS IS” with no warranties, and confers no rights. This weblog does not represent the thoughts, intentions, plans or strategies of my employer. It is solely my opinion. Feel free to challenge me, disagree with me, or tell me I’m completely nuts in the comments section of each blog entry, but I reserve the right to delete any comment for any reason whatsoever (abusive, profane, rude, or anonymous comments) – so keep it polite, please.

The primary instance was SM-UTSQL, where we configured a Peer-to-Peer publication named "PUBLISH1" on the database "DDS-TRANS". Next, we proceeded to configure Peer-to-Peer topology (right-click on publication, click on "Configure Peer-To-Peer topology"), and added the other 2 instances to the topology. After this, we clicked on the primary node and selected "Connect to all displayed nodes" :

We then went ahead through the UI and configured the replication. However, when we checked in object explorer, we saw that on the Primary instance (SM-UTSQL), under replication->Publish1, we could see both the Peer nodes SO-UTSQL and ST-UTSQL as subscribers, but on SO-UTSQL and ST-UTSQL, we could see on SM-UTSQL as the subscriber for the publication i.e. SO-UTSQL and ST-UTSQL did not recognize each other as subscribers.

We tried to add the missing subscriber through the new subscriptions wizard, but got the following error:

TITLE: New Subscription Wizard

------------------------------

SQL Server could not create a subscription for Subscriber 'ST-UTSQL'.

------------------------------

ADDITIONAL INFORMATION:

An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)

------------------------------

Peer-to-peer publications only support a '@sync_type' parameter value of 'replication support only', 'initialize with backup' or 'initialize from lsn'.

The subscription could not be found.

Changed database context to 'DDS_TRANS'. (Microsoft SQL Server, Error: 21679)

For help, click: https://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=10.50.4000&EvtSrc=MSSQLServer&EvtID=21679&LinkId=20476

------------------------------

BUTTONS:

OK

------------------------------

The resolution? Here are the steps:

1. Navigate to the replication tab in object explorer on the Primary instance (where you can see both subscriptions under the publication, SM-UTSQL in our case).
2. Right click on the publication and select generate scripts, and select the "To Create or enable the components" radio button.
3. In the resulting script, navigate to the bottom. Here, you will see 2 sets of "sp_addsubscription" and "sp_addpushsubscription_agent" calls:
   

-- Adding the transactional subscriptions

use [DDS_TRANS]

exec sp_addsubscription @publication = N'PUBLISH1', @subscriber = N'SO-UTSQL', @destination_db = N'DDS_TRANS', @subscription_type = N'Push', @sync_type = N'replication support only', @article = N'all', @update_mode = N'read only', @subscriber_type = 0

exec sp_addpushsubscription_agent @publication = N'PUBLISH1', @subscriber = N'SO-UTSQL', @subscriber_db = N'DDS_TRANS', @job_login = N'dds\dtsql.admin', @job_password = null, @subscriber_security_mode = 1, @frequency_type = 64, @frequency_interval = 1, @frequency_relative_interval = 1, @frequency_recurrence_factor = 0, @frequency_subday = 4, @frequency_subday_interval = 5, @active_start_time_of_day = 0, @active_end_time_of_day = 235959, @active_start_date = 0, @active_end_date = 0, @dts_package_location = N'Distributor'

GO

 

use [DDS_TRANS]

exec sp_addsubscription @publication = N'PUBLISH1', @subscriber = N'ST-UTSQL', @destination_db = N'DDS_TRANS', @subscription_type = N'Push', @sync_type = N'replication support only', @article = N'all', @update_mode = N'read only', @subscriber_type = 0

exec sp_addpushsubscription_agent @publication = N'PUBLISH1', @subscriber = N'ST-UTSQL', @subscriber_db = N'DDS_TRANS', @job_login = N'dds\dtsql.admin', @job_password = null, @subscriber_security_mode = 1, @frequency_type = 64, @frequency_interval = 1, @frequency_relative_interval = 1, @frequency_recurrence_factor = 0, @frequency_subday = 4, @frequency_subday_interval = 5, @active_start_time_of_day = 0, @active_end_time_of_day = 235959, @active_start_date = 0, @active_end_date = 0, @dts_package_location = N'Distributor'

GO


 

4. Copy these commands over, provide the value for the @job_password parameter (password for the login used to configure replication, reflected in the @job_login parameter), and run the appropriate set on the 2 subscribers. For example, we ran the first set of commands (@subscriber=N'SO-UTSQL') on ST-UTSQL instance, and the second set(@subscriber=N'ST-UTSQL') on the SO-UTSQL instance.
   

And voila, the subscriptions were created and syncing.

Hope this helped you. Comments and feedback are welcome.

I recently worked on an interesting issue with my good friend Igor (@sqlsantos), where we were facing performance issues with a piece of code that used the IS_MEMBER () function. Basically, the IS_MEMBER function is used to find out whether the current user (windows/sql login used for the current session) is a member of the specified Windows group or SQL server database role.

In the specified code, the IS_MEMBER function was being used to determine the windows group membership of the windows login. The windows groups were segregated according to geographical areas, and based on the user's group membership, the result set was filtered to show rows for only those geographical areas for which the user was a member of the corresponding groups in Active Directory.

Here's an example of a piece of code where we perform this check:

With SalesOrgCTE AS(

SELECT

  MinSalesOrg, MaxSalesOrg

FROM

  RAuthorized WITH (NOLOCK)

WHERE

  IS_MEMBER([Group]) = 1

)

The problem was that the complete procedure where we were using IS_MEMBER was taking several minutes to complete, for a table where the max result set cardinality was in the range of 18000-20000. We noticed the following wait types while the procedure was executing:

PREEMPTIVE_OS_AUTHORIZATIONOPS

PREEMPTIVE_OS_LOOKUPACCOUNTSID

I did some research on these waits, and found that since both of these are related to the communication/validation from Active Directory, they lie outside of SQL server, and there's no changes we can do from a configuration standpoint to help reduce/eliminate these waits.

Next, we studied the code, broke it down and tested the performance of the various sections that used the IS_MEMBER function, and found that the main section responsible for the execution time was the "WHERE" condition where we were using the result set of the code mentioned above. This is what the "WHERE" clause looked like:

(SELECT

  COUNT(*)

FROM

  SalesOrgCTE

WHERE

  SORGNBR BETWEEN MinSalesOrg AND MaxSalesOrg) > 0

Notice that in this code, we've asked SQL to check the value of SORGNBR for each row, and if it's between MinSalesORG and MaxSalesOrg, then add it to the rowcount. We observed that due to this design, it had to make a trip to AD for validating each row, which meant quite a long time for a 18000-20000 row result set, which was responsible for the slow performance of the procedure.

We did some more research with different approaches for the where clause, and the combined efforts of myself, Igor and his team resulted in the following where clause whose performance was acceptable:

WHERE

SORGNBR IN

(

       SELECT MinSalesOrg FROM SalesOrgCTE

)

AND SORGNBR IN

(

       SELECT MaxSalesOrg FROM SalesOrgCTE

)

If you look carefully, you'll notice that in this code snippet, we'll need to communicate with AD only twice, thereby improving the performance of the procedure as a whole.

Summing up: The importance of writing good code cannot be over-emphasized. It's good coding practices like this that lead to performance gains most of the time.

Hope this helps.

I recently worked on an interesting issue with a SQL Server Failover Cluster Instance (FCI). We were trying to use an SSL certificate on the instance, and we followed these steps:

1. Made sure the certificate was requested according to the requirements defined here.
2. Loaded the certificate into the Personal store of the computer account across all the nodes
3. Copied the thumbprint of the certificate, eliminated the spaces, and pasted it into the value field HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10.CLUSTEST\MSSQLServer\Certificate key. Please note that this was a SQL 2008 instance named "CLUSTEST"

However, when we restarted SQL Server after performing these changes, it failed. In the errorlog, we saw these messages:

2013-07-21 14:06:11.54 spid19s     Error: 17182, Severity: 16, State: 1.

2013-07-21 14:06:11.54 spid19s     TDSSNIClient initialization failed with error 0xd, status code 0x38. Reason: An error occurred while obtaining or using the certificate for SSL. Check settings in Configuration Manager. The data is invalid.

2013-07-21 14:06:11.54 spid19s     Error: 17182, Severity: 16, State: 1.

2013-07-21 14:06:11.54 spid19s     TDSSNIClient initialization failed with error 0xd, status code 0x1. Reason: Initialization failed with an infrastructure error. Check for previous errors. The data is invalid.

2013-07-21 14:06:11.54 spid19s     Error: 17826, Severity: 18, State: 3.

2013-07-21 14:06:11.54 spid19s     Could not start the network library because of an internal error in the network library. To determine the cause, review the errors immediately preceding this one in the error log.

2013-07-21 14:06:11.54 spid19s     Error: 17120, Severity: 16, State: 1.

2013-07-21 14:06:11.54 spid19s     SQL Server could not spawn FRunCommunicationsManager thread. Check the SQL Server error log and the Windows event logs for information about possible related problems.

I checked and made sure the certificate was okay, and that it was loaded properly. Then, I noticed something interesting. After copying the thumbprint to a text file, I got a Unicode to ANSI conversion warning when I tried to save the file in txt format:

1. Eliminated the Unicode character from the thumbprint
2. Converted all the alphabetical characters in the thumbprint to Caps.
3. Eliminated the spaces from the thumbprint
4. Saved this thumbprint to the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10.CLUSTEST\MSSQLServer\Certificate key.

The instance came online just fine this time.

Hope this helps.

We checked the errorlogs on the server, found the following successful stacks for the failovers that had been attempted for the successful instance:

And

We then checked the critical events for the AG in failover cluster manager, this was all i could find:

Cluster resource 'SQL2012CLUS02' in clustered service or application 'SQL2012CLUS02' failed.

I then collected the errorlog for the CDGI-SQLPROD-02 instance, and found this:

We can clearly see that the Login failure seems to be responsible for the failed failover of the AG. I tried adding the login manually, and restarted the instance, but the failover still failed. I then checked the event logs on CDGI-SQLPROD-02, found just this:

And then this:

The interesting thing here is that the connection attempt seems to be coming from the OS on the same box. I then captured a Profiler trace to confirm, and saw this:

Looked at the profiler trace, found the login error event:

The profiler trace confirms our hunch. We then proceeded to run the following commands to add local system as a sysadmin on the problem instance:

USE [master]

GO

/****** Object:  Login [NT AUTHORITYSYSTEM]    Script Date: 01-02-2013 03:31:56 ******/

CREATE LOGIN [NT AUTHORITYSYSTEM] FROM WINDOWS WITH DEFAULT_DATABASE=[master],

DEFAULT_LANGUAGE=[us_english]

GO

GO

ALTER SERVER ROLE [sysadmin] ADD MEMBER [NT AUTHORITYsystem]

GO

After this, the failover worked perfectly fine.

Hope this helps.

I recently worked on a number of issues where SQL Server Service Pack/patch installation would fail, and we would see this error in the relevant Detail.txt (located in C:Program FilesMicrosoft SQL Server100Setup BootstrapLog<Date time of the installation attempt> for SQL 2008/2008 R2):
 

2013-04-07 20:14:07 Slp: Package sql_bids_Cpu64: - The path of cached MSI package is: C:WindowsInstaller5c23b5e.msi . The RTM product version is: 10.50.1600.1

2013-04-07 20:14:07 Slp: Error: Action "Microsoft.SqlServer.Configuration.SetupExtension.InitializeUIDataAction" threw an exception during execution.

2013-04-07 20:14:13 Slp: Received request to add the following file to Watson reporting: C:UserskalerahulAppDataLocalTemp2tmpCC09.tmp

2013-04-07 20:14:13 Slp: The following is an exception stack listing the exceptions in outermost to innermost order

2013-04-07 20:14:13 Slp: Inner exceptions are being indented

2013-04-07 20:14:13 Slp:

2013-04-07 20:14:13 Slp: Exception type: System.Resources.MissingManifestResourceException

2013-04-07 20:14:13 Slp:     Message:

2013-04-07 20:14:13 Slp:         Could not find any resources appropriate for the specified culture or the neutral culture.  Make sure "Errors.resources" was correctly embedded or linked into assembly "Microsoft.SqlServer.Discovery" at compile time, or that all the satellite assemblies required are loadable and fully signed.

2013-04-07 20:14:13 Slp:     Stack:

2013-04-07 20:14:13 Slp:         at System.Resources.ResourceManager.InternalGetResourceSet(CultureInfo culture, Boolean createIfNotExists, Boolean tryParents)

2013-04-07 20:14:13 Slp:         at System.Resources.ResourceManager.GetObject(String name, CultureInfo culture, Boolean wrapUnmanagedMemStream)

2013-04-07 20:14:13 Slp:         at Microsoft.SqlServer.Discovery.MsiException.GetErrorMessage(Int32 errorNumber, CultureInfo culture)

2013-04-07 20:14:13 Slp:         at Microsoft.SqlServer.Discovery.MsiException.GetErrorMessage(MsiRecord errorRecord, CultureInfo culture)

2013-04-07 20:14:13 Slp:         at Microsoft.SqlServer.Discovery.MsiException.get_Message()

2013-04-07 20:14:13 Slp:         at System.Exception.ToString()

2013-04-07 20:14:13 Slp:         at System.Exception.ToString()

2013-04-07 20:14:13 Slp:         at System.Exception.ToString()

2013-04-07 20:14:13 Slp:         at Microsoft.SqlServer.Setup.Chainer.Workflow.ActionEngine.RunActionQueue()

2013-04-07 20:14:13 Slp:         at Microsoft.SqlServer.Setup.Chainer.Workflow.Workflow.RunWorkflow(WorkflowObject workflowObject, HandleInternalException exceptionHandler)

2013-04-07 20:14:13 Slp:         at Microsoft.SqlServer.Chainer.Setup.Setup.RunRequestedWorkflow()

2013-04-07 20:14:13 Slp:         at Microsoft.SqlServer.Chainer.Setup.Setup.Run()

2013-04-07 20:14:13 Slp:         at Microsoft.SqlServer.Chainer.Setup.Setup.Start()

2013-04-07 20:14:13 Slp:         at Microsoft.SqlServer.Chainer.Setup.Setup.Main()

Now that's a weird and hard to understand error, isn't it? However, look closely at what the setup is trying to do, and you will see that it's trying to access the following file from the installer cache:
 C:WindowsInstaller5c23b5e.msi
 

Open the installer cache and try to install the msi manually. If it succeeds, try running the patch setup again and it should proceed beyond the error this time. If the msi setup fails, then you will need to troubleshoot that first, before the patch setup proceeds further. This behaviour is expected, in that the service pack setup will try to access the msi's (Microsoft Installer files, installed with the base installation of SQL) and msp's (Microsoft Patch files, installed by Service packs, CU's and hotfixes) of each of the installed components of SQL Server. If it's unable to access/run any of these, the Service pack setup will fail.

Hope this helps.

The Symptoms

The following are the most common symptoms of an I/O bottleneck on the SQL Server machine:

• You see a lot of threads waiting on one or more of the following waits:
• PAGEIOLATCH_*
• WRITELOG
• TRACEWRITE
• SQLTRACE_FILE_WRITE_IO_COMPLETION
• ASYNC_IO_COMPLETION
• IO_COMPLETION
• LOGBUFFER
   
• You see the famous "I/O taking longer than 15 seconds" messages in the SQL Server errorlogs: 
  2012-11-11 00:21:25.26 spid1 SQL Server has encountered 192 occurrence(s) of IO requests taking longer than 15 seconds to complete on file [E:SEDATAstressdb5.ndf] in database [stressdb] (7). The OS file handle is 0x00000000000074D4. The offset of the latest long IO is:0x00000000022000”.

Troubleshooting

Data Collection:

If you see the symptoms outlined above quite frequently on your SQL Server installation, then it will be safe to draw the conclusion that your instance is suffering from a disk subsystem or I/O bottleneck. Let's look at the data collection and troubleshooting approach pertaining to the same:

1. Enable a custom Performance Monitor collector to capture all disk related counters. Just go to start->run, type perfmon, and hit ok. Next, go to Data Collector sets->User Defined, right click on User Defined, and click New-> Data Collector set.
   Note: The best thing about perfmon(apart from the fact that it is built into windows) is that it's a very lightweight diagnostic, and has negligible performance overhead/impact.
   
   
2. Give the data collector set a name, and select Create manually. Under type of data, select the "Create data logs" option, and check the Performance Counter checkbox under it.
   
   
3. Next, click on add performance counters, and select the "LogicalDisk", "Process" and "PhysicalDisk" groups, and select "All instances" for both before adding them.
   
   
4. After you have added the counters, you can also modify the sample interval. You might want to do this if you see spikes lasting less than 15 seconds, which is the default sample interval. I sometimes use an interval of 5 seconds when I want to closely monitor an environment .
   
   
5. Click on Finish and you will now see the new Data Collector set created under User Defined.
   
   
6. Next, right click on the Data Collector set you just created, and click start.

I normally recommend my clients to run the perfmon collector set for at least one business day, so that it has captured data for the load exerted by at least one standard business cycle.

Analysis:

Now that we have the data, we can start the analysis. After stopping the collector set, you can open the blg file generated (the path is displayed under the output column, on the right hand side in perfmon) using perfmon (a simple double click works, as the file type is associated with perfmon by default). Once open, is should have automatically loaded all the counters. Analysing with all the counters can be a bit cumbersome, so I would suggest that you first delete all the counters and then add specific counters one by one.

I will list out the important counters here, along with their expected values: 

1. Process->IO Data Bytes/sec: This counter represents the average amount of IO Data bytes/sec spawned by each process. In conjunction with IO Other Bytes/sec, this counter can be used to determine the average IO per second as well as the total amount of IO spawned by each process during the capture. Check for the largest I/O consumers, and see if SQL is being starved of I/O due to some other process spawning a large amount of I/O on the system.
    
2. Process-> IO Other Bytes/sec: This counter represents the non-data IO spawned by each process during the capture. Usually, the amount of non-data IO is very low as compared to data IO. Use the total of both IO Data Bytes and IO other bytes to determine the total amount of IO spawned by each process during the capture. Check for the largest I/O consumers, and see if SQL is being starved of I/O due to some other process spawning a large amount of I/O on the system.
    
3. Physical Disk/Logical Disk->Avg. Disk Sec/Read: This counter signifies the average amount of time, in ms, that it takes for a read I/O request to be serviced for each physical/logical disk. An average of less than 10 ms (0.010) is good, and between 10-15 ms (0.010-0.015) is acceptable, but anything beyond 15 ms (0.015) is a cause for concern.
    
4. Physical Disk/Logical Disk->Avg. Disk Sec/Write: This counter signifies the average amount of time, in ms, that it takes for a write I/O request to be serviced for each physical/logical disk. An average of less than 10 ms (0.010) is good, and between 10-15 ms (0.010-0.015) is acceptable, but anything beyond 15 ms (0.015) is a cause for concern.  
   
   
5. Physical Disk/Logical Disk->Disk Bytes/Sec: This counter represents, in bytes, the throughput of your I/O subsystem for each physical/logical disk. Look for the max value for each disk, and divide it by 1024 twice to get the max throughput in MB for the disk. SAN's generally start from 200-250 MB/s these days. If you see that the throughput is lower than the specifications for the disk, it's not necessarily a cause for concern. Check this counter in conjunction with the Avg Disk Sec/Read or Avg Disk Sec/Write counters (depending on the wait/symptom you see in SQL), and see the latency at the time of the maximum throughput. If the latency is green, then it just means that SQL spawned I/O that was less the disk throughput capacity, and was easily handled by the disk.
   
   
6. Physical Disk/Logical Disk->Avg. Disk Queue Length: This counter represents the average number of I/O's pending in the I/O queue for each physical/logical disk. Generally, if the average is greater than 2 per spindle, it's a cause for concern. Please note that I mentioned the acceptable threshold as 2 per spindle. Most SAN’s these days have multiple spindles. So, for example, if your SAN has 4 spindles, the acceptable threshold for Avg Disk Queue Length would be 8.
   Check the other counters to confirm.
   
   
7. Physical Disk/Logical Disk->Split IO/Sec: This counter indicates the I/O's for which the Operating System had to make more than one command call, grouped by physical/logical disk. This happens if the IO request touches data on non-contiguous file segments. It's a good indicator of file/volume fragmentation.
   
   
8. Physical Disk/Logical Disk->%Disk Time: This counter is a general mark of how busy the physical/logical disk is. Actually, it is nothing more than the “Avg. Disk Queue Length” counter multiplied by 100. It is the same value displayed in a different scale. This is the reason you can see the %Disk Time going greater than 100, as explained in the KB https://support.microsoft.com/kb/310067. It basically means that the Avg. Disk Queue Length was greater than 1 during that time. If you've captured the perfmon for a long period (a few hours or a complete business day), and you see the %Disk Time to be greater than 80%, it's generally indicative of a disk bottleneck, and you should take a closer look at the other counters to arrive at a logical conclusion.
    

It's important to keep 2 things in mind. One, make sure your data capture is not skewed or biased in any way (for example, do not run a capture at the time of a monthly data load or something). Second, make sure you correlate the numbers reflected across the various counters to arrive at the overall picture of how your disks are doing.

Most of the time, I see that people are surprised when they are told that there are I/O issues on the system. Their typical response is "But, it's been working just fine for x years, how can it create a bottleneck now?". The answer lies within the question itself. When the server was initially configured, the disk resources were sufficient for the load on the server. However, with time, it's inevitable that the business grows as a whole, and so do the number of transactions, as well as the overall load. As a result, there comes a day when the load breaches that threshold, and the disk resources on the server are no longer sufficient to handle it. If you come to office one fine day, see high latency on the disks during normal working hours, and are sure that

• No special/additional workloads are running on SQL
• No other process on the server is spawning excessive I/O,
• Nothing changed on the server in the past 24 hours (like a software installation, patching, reboot, etc.)
• All the BIOS and disk drivers on the server are up to date,

Then it's highly likely that the load on your server has breached this threshold, and you should think about asking your disk vendor(s) for a disk upgrade (after having them check the existing system once for latency and throughput, of course). Another potential root cause that can cause high latency is that your disk drivers and/or BIOS are out of date. I would strongly recommend checking periodically for updates to all the drivers on the machine, as well as the BIOS.

Hope this helps. As always, comments, feedbacks and suggestions are welcome.

Here's another common issue that I've seen quite a few people run into of late.

When you run a patch against SQL Server, the patch installs successfully, but on restart, SQL goes into "script upgrade mode" and you're unable to connect to it. Upon looking at the errorlog, you see something like this:

2012-08-23 03:43:38.29 spid7s      Error: 5133, Severity: 16, State: 1.

2012-08-23 03:43:38.29 spid7s      Directory lookup for the file "D:SQLDatatemp_MS_AgentSigningCertificate_database.mdf" failed with the operating system error 2(The system cannot find the file specified.).

2012-08-23 03:43:38.29 spid7s      Error: 1802, Severity: 16, State: 1.

2012-08-23 03:43:38.29 spid7s      CREATE DATABASE failed. Some file names listed could not be created. Check related errors.

2012-08-23 03:43:38.31 spid7s      Error: 912, Severity: 21, State: 2.

2012-08-23 03:43:38.31 spid7s      Script level upgrade for database 'master' failed because upgrade step 'sqlagent100_msdb_upgrade.sql' encountered error 598, state 1, severity 25. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the 'master' database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.

2012-08-23 03:43:38.31 spid7s      Error: 3417, Severity: 21, State: 3.

2012-08-23 03:43:38.31 spid7s      Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online.

Script upgrade means that when SQL is restarted for the first time after the application of the patch, the upgrade scripts are run against each system db (to upgrade the system tables, views, etc. ). During this process, SQL Server attempts to create this mdf file in the default data location, and if the path is not available, then we get this error. Most of the time, it's a result of the data having been moved to a different folder, and the original Default Data path being no longer available.

The default data path can be checked from the following registry key (for a default SQL 2008 instance):

HKEY_LOCAL_MACHINESoftwareMicrosoftMicrosoft SQL ServerMSSQL10.MSSQLSERVERMSSQLServer

The Mssqlserver key will have a string entry named "DefaultData". If you see a location here that's no longer available, please change it to the current data location (alternatively, you can also "recreate" the default data path mentioned in the string value).

If you do not see the key, check for the please check the HKEY_LOCAL_MACHINESOFTWAREMicrosoftMicrosoft SQL ServerMSSQL10.<instance name>Setup key, and see if you can spot the SQLDataRoot key there. Check to see if this key has the path mentioned above, and if so, update it to the current path.

If the path is correct, then one of the following conditions holds true:

1. The relevant drive is not added as a resource to the SQL Server group in Failover cluster manager.

2. The SQL Server resource does not have a dependency on the specified drive.

After this, restart SQL Server and the script upgrade should complete successfully this time. Hope this helps.

I've seen this issue a few times in the past few months, so decided to blog about this. When upgrading from SQL 2005 to SQL 2008/SQL 2008 R2 (or even from SQL 2008 to SQL 2008 R2), you might face an error with the in-place upgrade.

Open the setup logs folder (located in C:Program filesMicrosoft SQL Server<Version -100 for 2008 and 2008 r2>Setup BootstrapLog folder by default), and look for a folder with the datetime of the upgrade attempt. Inside this folder, look for a file named "Detail.txt".

Looking inside the detail.txt file, check for the following stack:

2013-01-21 11:16:42 Slp: Sco: Attempting to check if container 'WinNT://Harsh2k8,computer' of user account exists

2013-01-21 11:16:42 Slp: Sco: User srv_sql@contoso.test wasn't located

2013-01-21 11:16:42 Slp: Sco: User srv_sql@contoso.test doesn't exist

2013-01-21 11:16:42 SQLBrowser: SQL Server Browser Install for feature 'SQL_Browser_Redist_SqlBrowser_Cpu32' generated exception, and will invoke retry option.  The exception: Microsoft.SqlServer.Configuration.Sco.ScoException: The specified user 'srv_sql@contoso.test' does not exist.

   at Microsoft.SqlServer.Configuration.Sco.UserGroup.AddUser(String userName)

   at Microsoft.SqlServer.Configuration.SqlBrowser.SqlBrowserPrivateConfig.AddAccountToGroup(SqlBrowserPublicConfig publicConfigSqlBrowser)

   at Microsoft.SqlServer.Configuration.SqlBrowser.SqlBrowserPrivateConfig.UpdateAccountIfNeeded(SqlBrowserPublicConfig publicConfigSqlBrowser)

   at Microsoft.SqlServer.Configuration.SqlBrowser.SqlBrowserPrivateConfig.ConfigUserProperties(SqlBrowserPublicConfig publicConfigSqlBrowser)

   at Microsoft.SqlServer.Configuration.SqlBrowser.SqlBrowserPrivateConfig.ExecConfigNonRC(SqlBrowserPublicConfig publicConfigSqlBrowser)

   at Microsoft.SqlServer.Configuration.SqlBrowser.SqlBrowserPrivateConfig.SelectAndExecTiming(ConfigActionTiming timing, Dictionary`2 actionData, PublicConfigurationBase spcbPublicConfig)

   at Microsoft.SqlServer.Configuration.SqlBrowser.SqlBrowserPrivateConfigBase.ExecWithRetry(ConfigActionTiming timing, Dictionary`2 actionData, PublicConfigurationBase spcbPublicConfig).

2013-01-21 11:16:42 SQLBrowser: The last attempted operation: Adding account 'srv_sql@contoso.test' to the SQL Server Browser service group 'SQLServer2005SQLBrowserUser$Harsh2k8'..

The key thing here is the message "Attempting to check if container WinNT://Harsh2k8, computer of user account exists". If you see this message, go to the SQL Server configuration manager, right click on the offending service mentioned in the detail.txt, open the properties window and navigate to the "Log On" tab. Check the format of the service account here. It should be in the format domainusername. Change this to username@domain, and type in the password. After this, restart the SQL Service to make sure the changes have taken effect.

Try the setup again, and it should work this time.

Hope this helps.

Before we start on the troubleshooting part though, we need to determine the type of memory pressure that we're seeing here. I've tried to list those down here:

1.     External Physical Memory pressure - Overall RAM pressure on the server. We need to find the largest consumers of memory (might be SQL), and try to reduce their consumption. It might also be that the system is provided with RAM inadequate for the workload it's running.

2.     Internal Physical Memory pressure - Memory Pressure on specific components of SQL Server. Can be a result of External Physical Memory pressure, or of one of the components hogging too much memory.

3.     Internal Virtual Memory pressure - VAS pressure on SQL server. Mostly seen only on 32 bit (X86) systems these days (X64 has 8 TB of VAS, whereas X86 only had 4 GB. Refer to Part 1 for details).

4.     External Virtual Memory pressure - Page file pressure on the OS. SQL Server does not recognize or respond to this kind of pressure.

Troubleshooting

Now for getting our hands dirty. When you suspect memory pressure on a server, I would recommend checking the following things, in order:

1.     Log in to the server, and take a look at the performance tab of the Task Manager. Do you see the overall memory usage on the server getting perilously close to the total RAM installed on the box? If so, it's probable that we're seeing External Physical Memory pressure.

2.     Next, look at the Processes tab, and see which of the processes is using the maximum amount of RAM. Again, for SQL, the true usage might not reflect in the Working set if LPIM is enabled (i.e. SQL is using AWE API's to allocate memory). To check SQL's total memory consumption, you can run the following query from inside SQL (valid from SQL 2008 onwards):

select physical_memory_in_use_kb/(1024) as sql_physical_mem_in_use_mb,

locked_page_allocations_kb/(1024) as awe_memory_mb,

total_virtual_address_space_kb/(1024) as max_vas_mb,

virtual_address_space_committed_kb/(1024) as sql_committed_mb,

memory_utilization_percentage as working_set_percentage,

virtual_address_space_available_kb/(1024) as vas_available_mb,

process_physical_memory_low as is_there_external_pressure,

process_virtual_memory_low as is_there_vas_pressure

from sys.dm_os_process_memory

Go

For SQL installations prior to 2008 (valid for 2008 and 2008 R2 as well), you can run DBCC Memorystatus, and take the total of VM Committed and AWE Allocated from the memory manager section to get a rough idea of the amount of memory being used by SQL Server.
 

3.     Next, compare this with the total amount of RAM installed on the server. If SQL seems to be taking most of the memory, or at least, much more than it should, then we need to focus our attentions on SQL Server. The exact specifics will vary according to the environment, and factors such as whether it is a dedicated SQL server box, number of instances of SQL Server running on the server, etc. In case you have multiple instances of SQL Server, it will be best to start with the instance consuming the maximum amount of memory (or the maximum deviation from "what it should be consuming"), tune it and then move on to the next one. 
 

4.     One of the first things to check should be the value of the "max server memory" setting for SQL Server. You can check this by turning on the 'show advanced options' setting of sp_configure, or by right clicking on the instance in Object Explorer in SSMS, selecting properties, and navigating to the "memory" tab. If the value is "2147483647", this means that the setting has been left to default, and has not been set since the instance was installed. It's absolutely vital to set the max server memory setting to an optimal value. A general rule of thumb that you can use to set a starting value is as follows:
Total server memory - (Memory for other applications/instances+ OS memory)
The recommendation for the OS memory value is around 3-4 GB on 64 bit systems, and 1-2 GB on 32 bit systems. Please note that this is only a recommendation for the starting  value. You need to fine tune it based on observations w.r.t performance of both SQL and other applications (if any) on the server.
 

5.     Once you've determined that the max server memory is set properly, the next step is to find out which component within SQL is consuming the most memory. The best place to start is, quite obviously, the good old "DBCC Memorystatus" command, unless you're using NUMA, in which case, it will be best to use perfmon counters to track page allocations across NUMA nodes, as outlined here.
I will try to break down most of the major components in the DBCC Memorystatus output here (I would recommend reading KB 907877 as a primer before this):

                   I.     First up is the memory manager section. As discussed earlier, this section contains details about the overall memory comsumption of SQL Server. An example:

Memory Manager                           KB

---------------------------------------- -----------

VM Reserved                                 4059416

VM Committed                                 43040

Locked Pages Allocated                   41600

Reserved Memory                              1024

Reserved Memory In Use                        0

                  II. Next, we have the memory nodes, starting with 0. As I mentioned, because there is a known issue with the way dbcc memorystatus displays the distribution of allocations across memory nodes, it is best to study the distribution through the SQL Server performance counters. Here's a sample query:
 

select * from sys.dm_os_performance_counters

      where object_name like '%Buffer Node%'
 

                III. Next, we have the clerks. I've tried to outline the not so obvious ones in this table, along with their uses:        

Clerk Name	Used for
MEMORYCLERK_SQLUTILITIES	Database mirroring, backups, etc.
MEMORYCLERK_SQLXP	Extended Stored Procedures (loaded into SQL Server)
MEMORYCLERK_XE, MEMORYCLERK_XE_BUFFER	Extended Events

If you see any of the clerks hogging memory, then you need to focus on that, and try and narrow down the possible causes.

 

Another thing to watch out for is high values for the multipage allocator. If you see any clerk with extremely high values for multipage allocator, it means that the non-Bpool area is growing due to one of the following:

                                       i.            CLR Code: Check the errorlog for appdomain messages

                                     ii.            COM Objects : Check the errorlog for sp_oacreate

                                    iii.            Linked servers: Can be checked using Object Explorer in SSMS

                                   iv.             Extended stored procedures :  Check the errorlog for loading extended stored procedure messages.

                                    Alternatively, you can query the sys.extended_procedures view as well.

                                     v.            Third party DLL's : Third party DLL's loaded into the SQL server process space. Run the following query to check:
        select * from sys.dm_os_loaded_modules where company <> 'Microsoft Corporation'
 

Here's a query to check for the biggest multipage consumers:

select type, name, sum(multi_pages_kb)/1024 as multi_pages_mb

from sys.dm_os_memory_clerks

where multi_pages_kb > 0

group by type, name

order by multi_pages_mb desc

Yet another symptom to watch out for is a high ratio of stolen pages from the Buffer Pool. You can check this in the 'Buffer Pool' section of the MEMORYSTATUS output. A sample:

Buffer Pool                                      Value

---------------------------------------- -----------

Committed                                          4448

Target                                                25600

Database                                             2075

Dirty                                                        50

In IO                                                          0

Latched                                                     0

Free                                                       791

Stolen                                                  1582

Reserved                                                   0

Visible                                                25600

Stolen Potential                                 22738

Limiting Factor                                        17

Last OOM Factor                                       0

Last OS Error                                             0

Page Life Expectancy                         87529

What this means is that Buffer Pool pages are being utilized for "other" uses, and not for holding data and index pages in the BPool. This can lead to performance issues and a crunch on the Bpool, thereby slowing down overall query performance (please refer to part 1 for consumers that "Steal" pages from the BPool). You can use the following query to check for the highest "Steal" consumers:

select type, name, sum((single_pages_kb*1024)/8192) as stolen_pages

from sys.dm_os_memory_clerks

where single_pages_kb > 0

group by type, name

order by stolen_pages desc

               IV.   Next, we have the stores namely, Cachestore, Userstore and Objectstore. Please refer to part 1 for how and by which component these clerks are used. You can use the following queries to check for the biggest Cachestores, Userstores and Objectstores respectively:
 

select name, type, (SUM(single_pages_kb)+SUM(multi_pages_kb))/1024

as store_size_mb

from sys.dm_os_memory_cache_counters

where type like 'CACHESTORE%'

group by name, type

order by store_size_mb desc

go

 

select name, type, (SUM(single_pages_kb)+SUM(multi_pages_kb))/1024

as store_size_mb

from sys.dm_os_memory_cache_counters

where type like 'USERSTORE%'

group by name, type

order by store_size_mb desc

go

 

select name, type, (SUM(single_pages_kb)+SUM(multi_pages_kb))/1024

as store_size_mb

from sys.dm_os_memory_clerks

where type like 'OBJECTSTORE%'

group by name, type

order by store_size_mb desc

go

                 V.   Next, we have the gateways. The concept of gateways was introduced to throttle the use of query compilation memory. In plain english, this means that we did not want to allow too many queries with a high requirement for compilation memory to be running at the same time, as this would lead to consequences like internal memory pressure (i.e. one of the components of the buffer pool growing and creating pressure on other components).

The concept basically works like this: When a query starts execution, it will start with a small amount of memory. As its consumption grows, it will cross the threshold for the small gateway, and must wait to acquire it. The gateway is basically implemented through a semaphore, which means that it will allow upto a certain number of threads to acquire it, and make threads beyond the limit wait. As the memory consumption for the query grows, it must acquire the medium and big gateways before being allowed to continue execution. The exact thresholds depend on factors like total memory on the server, SQL Max server memory sitting, memory architecture (x86 or x64), load on the server, etc.

The number of queries allowed at each of the gateways described in the following table:             

So if you see a large number of queries waiting on the large gateway, it means that you need to see why there are so many queries requiring large amounts of memory, and try to tune those queries. Such queries will show up with  RESOURCE_SEMAPHORE_QUERY_COMPILE or RESOURCE_SEMAPHORE wait types in sysprocesses, sys.dm_exec_requests, etc.

I am listing down some DMV's that might come in handy for SQL Server Memory Troubleshooting:

Sysprocesses

Sys.dm_exec_requests

Sys.dm_os_process_memory: Usage above.

Sys.dm_os_sys_memory: Will give you the overall memory picture for the server

Sys.dm_os_sys_info: Can be used to check OS level information like hyperthread ratio, CPU Ticks, OS Quantum, etc.

Sys.dm_os_virtual_address_dump: Used to check for VAS usage (reservations). The following query will give you VAS usage in descending order of reservations:

with vasummary(Size,reserved,free) as (select size = vadump.size,

reserved = SUM(case(convert(int, vadump.base) ^ 0)  when 0 then 0 else 1 end),

free = SUM(case(convert(int, vadump.base) ^ 0x0) when 0 then 1 else 0 end)

from

(select CONVERT(varbinary, sum(region_size_in_bytes)) as size,

region_allocation_base_address as base

from sys.dm_os_virtual_address_dump

where region_allocation_base_address<> 0x0

group by region_allocation_base_address

UNION(

select CONVERT(varbinary, region_size_in_bytes),

region_allocation_base_address

from sys.dm_os_virtual_address_dump

where region_allocation_base_address = 0x0)

)

as vadump

group by size)

select * from vasummary order by reserved desc

go

Sys.dm_os_memory_clerks (Usage above)

Sys.dm_os_memory_nodes: Just a select * would suffice. This DMV has one row for each memory node.

Sys.dm_os_memory_cache_counters: Used above to find the size of the cachestores. Another sample query would be

select (single_pages_kb+multi_pages_kb) as memusage,* from Sys.dm_os_memory_cache_counters order by memusage desc

 

Once you have narrowed down the primary consumer and the specific component which is causing a memory bottleneck, the resolution steps should be fairly simple. For example, if you see some poorly written code, you can hound the developers to tune it. For other processes hogging memory at the OS Level, you will need to investigate them. For high consumption by a particular clerk, check the corresponding components. An example would be, say,  in case of high usage by the SQLUtilities clerk, one of the first things you need to check if there is any Mirroring set up on the instance, and if it’s working properly.

Another thing I would strongly recommend would be to watch out for memory related KB articles, and make sure you have the relevant fixes applied.

Hope this helps. Any feedback, questions or comments are welcome.