Don Castelino

Troubleshooting Query timeouts in SQL Server

Don Castelino[MSFT] — Fri, 12 Oct 2018 13:45:17 GMT

Hello all, While working with SQL Server, one of the common issue application teams encounter is timeouts. In this article, we are covering approach to troubleshoot SQL Command/Query timeouts, data collection and data analysis steps to isolate further. There are 2 types of timeouts.

Command Timeout
Connection Timeout.

For more information on connection timeout issues, refer here: Simulating Command timeout: For demonstration purpose, I am using a sample web application which connections to the SQL database Adventureworks2014. The application connects to the database to retrieve customers/Products details. To simulate the timeout error, click on Customers menu: From the exception stack trace, its evident that application is trying to retrieve some data and System.Data.SqlClient.SqlCommand.ExecuteReader method is called, but since the ExecuteReader method execution has not completed in the Command Timeout value (30 seconds), the application is reporting Command Timeout. Exception Details: System.ComponentModel.Win32Exception: The wait operation timed out Stack Trace:

[Win32Exception (0x80004005): The wait operation timed out]
[SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.]
System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) +3305692
System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) +736
System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) +4061
System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() +90
System.Data.SqlClient.SqlDataReader.get_MetaData() +99
System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted) +604
System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) +3303
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) +667
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) +83
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method) +301
System.Data.Entity.Infrastructure.Interception.InternalDispatcher`1.Dispatch(TTarget target, Func`3 operation, TInterceptionContext interceptionContext, Action`3 executing, Action`3 executed) +104
System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.Reader(DbCommand command, DbCommandInterceptionContext interceptionContext) +499
System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior) +36

Identifying SQL Queries which are Timing Out: Next step is to identify the SQL queries timing out. Easier approach would be check with the application team on the queries executed. Using SQL tools like Extended Events and SQL Profiler, we can also identify the queries timing outs using Attention event. If you are using SQL Extended Events, collect the events like SQL_Batch_Starting, SQL_batch_completed along with attention event to identify the queries timing out. DATA COLLECTION FOR QUERY TIMEOUTS: Another easier approach to collect the data would be to use PSSDIAG Configuration manager to collect the relavant data like Performance Statistics dmv data, Profiler/Extended event data, blocking data, etc. PSSDIAG data collection utility can be downloaded from: https://github.com/Microsoft/DiagManager Once you launch PSSDIAG configuration utility, fill in the necessary details like: Instance name, SQL Build, Scenario and Profiler/Xevent. While capturing the data to track query timeouts, ensure that Attention events are selected in Errors section. Once the events are selected, save the pssd.zip file and extract the contents of the zip file. To initiate the data collection, navigate to the pssdiag location using command prompt(Run as Administrator) and launch pssdiag.cmd Once "SQLDIAG collection started" message appears, reproduce the timeout issue. Once the issue is reproduced, stop the data collection using CTRL + C. DATA ANALYSIS: To analyze the PSSDIAG data collected, you can use SQLNexus tool which can be downloaded from: https://github.com/Microsoft/SqlNexus Installation of SQL nexus tool details are at: https://github.com/Microsoft/SqlNexus/wiki/Installation Once the installation is complete, launch the tool to import the PSSDIAG data collected earlier. Select the database to host the Nexus data. Click on Import and point to the PSSDIAG output folder: An interesting feature of SQL nexus is that it can import the SQL Profiler trace data collected and output the trace files by SPID, which will help when you are interested in tracing the Profiler data at SPID level. Once the data import is successful: Click on Read Trace reports which is a starting point to review all batch/Statement level queries. To review the timeouts: From the above screenshot, its evident that the query executed by session ID 53 has timed out. Launching the SQL profiler trace captured, verify that SPID 53 has timed out after 30 seconds as highlighted below. Review the CPU time as well, in our scenario, CPU time is in milli seconds, which explains that the query is not a CPU bound query. Hence you need to identify the waits associated with session ID 53. Reviewing the Blocking details, we can see that session ID 53 is waiting throughout on OBJECT:5:997578592:0 wait resource with wait type: LCK_M_IS. Since its execution didn’t complete in 30 seconds, it has timed out and application raises ABORT with SQL Server. Next step would be to identify the headblocker query and identify the blocking query and check if it’s a RUNNER/Waiter query. SUMMARY: We can use PSSDIAG and SQL Nexus tools to identify the queries timing out and root cause of timeout issue. Capture the SQL profiler trace/Xevent with Attention event which will help in narrowing down the SPID’s timing out. Also, not that the PSSDIAG and SQL nexus tool can be used to collect data for Slow query performance, General bottleneck analysis and reviews waits and address other SQL performance issues as well. Hope this blog helps in isolating query timeout issues. Please share your feedback, questions and/or suggestions. Thanks, Don Castelino | Premier Field Engineer | Microsoft Disclaimer: All posts are provided AS IS with no warranties and confer no rights. Additionally, views expressed here are my own and not those of my employer, Microsoft.

Connection timeout and Command timeout in SQL Server

Don Castelino[MSFT] — Thu, 11 Oct 2018 12:54:46 GMT

Hello all, While working with SQL Server, one of the common issue application teams report is timeouts. In this article, we are covering connection and Command timeouts and ways to isolate them. There are 2 types of timeouts.

Connection timeout
Command Timeout

CONNECTION TIMEOUT:

It is the time in seconds application waits while trying to create a connection with SQL Server before terminating the attempt. Default value of connection timeout is 15 seconds. When you encounter Connection timeout issues, you should review:

Check if you are able to telnet SQL Server on SQL port
Check if the 3 way TCP handshake is working
The troubleshooting approach should be on fixing SQL Connectivity with application.

Additional information on troubleshooting SQL Connectivity issues is documented HERE. Reviewing the exception and the stack trace is a starting point to isolate connection timeout issues. Reviewing the thread stack trace, you would observe that application tries to create a connection and times out post encountering Connection timeout value (by default 15 seconds).

Exception Details: System.Data.SqlClient.SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)

Stack Trace:

[SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)]
System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling, SqlAuthenticationProviderManager sqlAuthProviderManager) +1431
System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions) +1085
System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions) +70
System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) +964
System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) +109
System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection) +1529
System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection) +156
System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) +258
System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions) +312
System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry) +202
System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry) +413
System.Data.SqlClient.SqlConnection.Open() +128
System.Data.Entity.Infrastructure.Interception.InternalDispatcher`1.Dispatch(TTarget target, Action`2 operation, TInterceptionContext interceptionContext, Action`3 executing, Action`3 executed) +104
System.Data.Entity.Infrastructure.Interception.DbConnectionDispatcher.Open(DbConnection connection, DbInterceptionContext interceptionContext) +503
System.Data.Entity.SqlServer.<>c__DisplayClass1.b__0() +18
System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute(Func`1 operation) +234

COMMAND TIMEOUT:

This is the time in seconds to wait for the command to execute. This setting allows the cancellation of an ExecuteReader method call, due to delays from network traffic or heavy server use. The default value is 30 seconds. When you encounter Command timeout issues, you should review:

Check if there are any performance issues with SQL Server like blocking
Check in SQL Server why the query execution is taking more than Command Timeout value.
Collect Attention events in SQL profiler/Extended Events and track all queries timing out and review the waits.

Additional details on troubleshooting SQL query/command timeout is documented HERE: Reviewing the stack trace closely, you would observe that application connects to SQL Server and it calls ExecuteReader method for retrieving data from database. Exception Details: System.ComponentModel.Win32Exception: The wait operation timed out Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:

[Win32Exception (0x80004005): The wait operation timed out]
[SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.]
System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) +3305692
System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) +736
System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) +4061
System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() +90
System.Data.SqlClient.SqlDataReader.get_MetaData() +99
System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted) +604
System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) +3303
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) +667
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) +83
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method) +301
System.Data.Entity.Infrastructure.Interception.InternalDispatcher`1.Dispatch(TTarget target, Func`3 operation, TInterceptionContext interceptionContext, Action`3 executing, Action`3 executed) +104
System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.Reader(DbCommand command, DbCommandInterceptionContext interceptionContext) +499
System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior) +36

SUMMARY:

To summarize, when an application reports timeout error, its essential to identify the type of timeout, if it is Connection/Command timeout. Exception reported and the stack trace is a starting point to isolate the type of timeout. Accordingly, further data collection steps and actions can be performed to isolate the exact issue. Hope this blog helps in identifying the timeouts and helps in isolating the timeout issues. Please share your feedback, questions and/or suggestions. Thanks, Don Castelino | Premier Field Engineer | Microsoft Disclaimer: All posts are provided AS IS with no warranties and confer no rights. Additionally, views expressed here are my own and not those of my employer, Microsoft.

Rapid growth of Tempdb on AlwaysOn secondary replica due to version store

Don Castelino[MSFT] — Sat, 04 Aug 2018 10:31:09 GMT

Hello all, Recently I had worked on an interesting issue with rapid growth of Tempdb on AlwaysOn secondary replica. In this article, we will cover the approach and the queries used to isolate the growth issue and the workaround/fix used to control the growth of Tempdb on secondary replica. The following types of objects can occupy tempdb space:

Internal objects
Version stores
User objects

In the case of disk-based tables, readable secondary replicas can require space in tempdb for two reasons:

Snapshot isolation level copies row versions into tempdb.
Temporary statistics for secondary databases are created and maintained in tempdb. The temporary statistics can cause a slight increase in the size of tempdb.

When the Secondary replica is setup with Readable secondary setting to TRUE at AG level, there is 14 byte overhead on every row inserted/updated on the primary database. Along with that SQL Server maintains versions of the rows in tempdb database and maps the isolation level to SNAPSHOT ISOLATION level on secondary replica. The row version data is not generated by the primary databases. Instead, the secondary databases generate the row versions. However, row versioning increases data storage in both the primary and secondary databases. The scenario we had worked on, Tempdb data file had grown to 500GB and version store space was around 495GB. We used the below query to calculate the version store space usage.

select getdate() AS runtime, SUM (user_object_reserved_page_count)*8 as usr_obj_kb,
SUM (internal_object_reserved_page_count)*8 as internal_obj_kb,
SUM (version_store_reserved_page_count)*8  as version_store_kb,
SUM (unallocated_extent_page_count)*8 as freespace_kb,
SUM (mixed_extent_page_count)*8 as mixedextent_kb
FROM sys.dm_db_file_space_usage

There was no reporting workload running on the secondary replica. To identify the transactions currently maintaining an active version store, we used the below query:

select GETDATE() AS runtime,a.*,b.kpid,b.blocked,b.lastwaittype,b.waitresource,db_name(b.dbid) asdatabase_name,
b.cpu,b.physical_io,b.memusage,b.login_time,b.last_batch,b.open_tran,b.status,b.hostname,
b.program_name,b.cmd,b.loginame,request_id
from sys.dm_tran_active_snapshot_database_transactions a
inner join sys.sysprocesses b
on a.session_id = b.spid

From the above query we retrieved the below information: commit_sequence_num: Null session ID: 40 is_snapshot: 0 From sysprocesses, we identified that session ID (40) belonged to system thread which was performing REDO work on the secondary replica. We isolated the issue to be because of high number of updates happening on the primary database. On secondary replica, the redo thread was trying to catch up with the primary and was maintaining multiple versions of the rows updated to achieve read consistency. Application team confirmed that there was no reporting workload configured on secondary replica database. So we disabled readable secondary option as a workaround for the secondary replica. Post implementing the workaround SQL stopped maintaining versions of the rows on secondary replica and the space growth was controlled. Hope the information shared above helps in case you encounter Tempdb growth issues in AlwaysOn setup. Please share your feedback, questions and/or suggestions. Thanks, Don Castelino | Premier Field Engineer | Microsoft Disclaimer: All posts are provided AS IS with no warranties and confer no rights. Additionally, views expressed here are my own and not those of my employer, Microsoft.

SQL Performance troubleshooting guide: A walk through

Don Castelino[MSFT] — Sat, 04 Aug 2018 08:44:27 GMT

Hi all, , This article gives a high level view of isolating SQL Performance issues and logical approach to troubleshoot them. SQL Server performance issues can be broadly categorized as:

Overall Server slowness
Specific query slowness

If SQL Server overall performance is slow, then we can categorize the issues as below: If SQL Server overall performance is slow, then we can categorize the issues as below:

CPU/High Memory bottleneck
IO Bottleneck
Blocking
Network bottleneck
Latch Waits
SQL Configuration issues
Capacity limitation
SQL/Windows known issues

If a specific query is slow, then then the threads executing the query is either in one of the following states:

Executing/Running
Waiting

If the query is in executing state and the query execution is slow, the issues can be because of:

Stale statistics
High Index/Heap fragmentation
Bad SQL Execution Plan
Optimizer timeout
Long compilations

If the query is in waiting state and the query execution is slow, the issues can be because of:

IO Bottleneck (PAGE IO Latch/Writelog)
Blocking
Network IO/Client Fetch
Latch Waits
Resource Semaphore waits

The above walk through is at very high level and refer it as a guide map while troubleshooting performance issues. Based on the bottleneck observed, issue can be drilled further by using dmv's or other troubleshooting tools. Please share your feedback, questions and/or suggestions. Thanks, Don Castelino | Premier Field Engineer | Microsoft Disclaimer: All posts are provided AS IS with no warranties and confer no rights. Additionally, views expressed here are my own and not those of my employer, Microsoft.

Setting up ASP.NET session state with SQL In memory OLTP and AlwaysOn database

Don Castelino[MSFT] — Tue, 06 Mar 2018 10:30:57 GMT

Hello all, Recently I had worked on a requirement of achieving high availability for asp.net sessions. This blog post focuses on using SQL Server as the session state mode and use In-memory OLTP and AlwaysOn technology as the storage option for session data and achieve high availability. In memory OLTP ensures increased performance and scalability whereas AlwaysOn ensures that the database hosting the sessions is highly available. ASP.NET session state enables you to store and retrieve values for a user as the user navigates the different ASP.NET pages that make up a Web application. ASP.NET session modes available are:

InProc mode, which stores session state in memory on the Web server. This is the default.
StateServer mode, which stores session state in a separate process called the ASP.NET state service. This ensures that session state is preserved if the Web application is restarted and makes session state available to multiple Web servers in a Web farm.
SQLServer mode stores session state in a SQL Server database. This ensures that session state is preserved if the Web application is restarted and also makes session state available to multiple Web servers in a Web farm.

In a web farm scenario, the application which is running behind the load balancer, should maintain the same session state, and hence they need to point to same state server. If they point to different state server, then the state will be different for both application requests which results in erratic behavior. Now, because all the web servers point to the same state server, the solution is not highly available. If the state server hosting the asp session state crashes, then it’s very difficult to get the sessions back. Lets dive into the steps used and the errors reported during configuration of SQL Server as session mode and use In memory/AlwaysOn to achieve highly available solution to host sessions.

Use the script in the github solution repository to setup the SQL server database. Execution of the script creates the below tables and stored procedures needed and it also enables In memory OLTP on dbo.ASPStaTetempSessions table which is used to store the sessions. Please note that the script uses DURABILITY = SCHEMA_ONLY on In memory OLTP table, hence when SQL Server is restarted/AlwaysOn database is failed over, only the table schema persists, but data in the table is lost. Hence to preserve the sessions, use DURABILITY=SCHEMA_AND_DATA.

https://github.com/Microsoft/sql-server-samples/blob/master/samples/applications/aspnet-session-state/aspstate_sql2016_no_retry.sql Tables created: [dbo].[ASPStateTempSessions] and [dbo].[ASPStateTempApplications] To incorporate retry on the transactions, use the below script: https://github.com/Microsoft/sql-server-samples/blob/master/samples/applications/aspnet-session-state/aspstate_sql2016_with_retry.sql Next step is to configure AlwaysOn on ASPState database to achieve high availability of the database. For demonstration purpose, lets use buggybits site. To change the session state from In process (Default session mode) to SQL Server, connect to Internet Information Services manager (inetmgr) and locate the site and session state module. Select SQL Server state mode and click on create: Provide the AlwaysOn listener name in the Server list and specify ASPSTATE database. Also, specify the credentials used to connect to the SQL Server. Make sure that "Enable Custom database" setting is enabled. otheriwise, below error is reported:

Server Error in '/' Application.
Configuration Error
Description: An error occurred during the processing of a configuration file required to service this request. Please review the specific error details below and modify your configuration file appropriately. 
Parser Error Message: The sqlConnectionString attribute or the connection string it refers to cannot contain the connection options 'Database', 'Initial Catalog' or 'AttachDbFileName'. In order to allow this, allowCustomSqlDatabase attribute must be set to true and the application needs to be granted unrestricted SqlClientPermission. Please check with your administrator if the application does not have this permission.

Now access the application. When SQL Server is used as session state mode, dbo.ASPStatetempApplications table creates an entry, with the application ID details. Review, the ASPStateTempsessions table, and the table will reflect all the sessions currently active. Let us simulate a scenario of Availability Group fail over. When the database is in middle of the failover, because the session states are not retrievable, below error is reported:

Cannot open database "ASPSTATE" requested by the login. The login failed.
 Unable to access availability database 'ASPSTATE' because the database replica is not in the PRIMARY or SECONDARY role. Connections to an availability database is permitted only when the database replica is in the PRIMARY or SECONDARY role. Try the operation again later.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 
 Exception Details: System.Data.SqlClient.SqlException: Cannot open database "ASPSTATE" requested by the login. The login failed.
 Unable to access availability database 'ASPSTATE' because the database replica is not in the PRIMARY or SECONDARY role. Connections to an availability database is permitted only when the database replica is in the PRIMARY or SECONDARY role. Try the operation again later.

Once the Availability Group fail over is done, you would be able to access the application again. Hope the steps mentioned above will help you in setting up SQL Server as session state mode to host ASP.NET sessions. Please share your feedback, questions and/or suggestions. Thanks, Don Castelino | Premier Field Engineer | Microsoft Disclaimer: All posts are provided AS IS with no warranties and confer no rights. Additionally, views expressed here are my own and not those of my employer, Microsoft.

Login failed in SQL Server due to SSMS Intellisense

Don Castelino[MSFT] — Tue, 27 Feb 2018 14:54:45 GMT

Hello all, Recently, I had worked on an interesting Login failed error constantly getting reported in SQL Error log. In this blog, we are covering the steps taken to fix the issue. Below error was reported in Error log: 2018-02-22 09:48:19.76 Logon Error: 18456, Severity: 14, State: 38. 2018-02-22 09:48:19.76 Logon Login failed for user 'test'. Reason: Failed to open the explicitly specified database 'MODEL'. [CLIENT: >] Looking at the error message, the Login failed error message was reported from the Local machine itself. Connected to SQL instance using the account for which login failed error message was reported. Expanding the model database gave the same error. But the user was not trying to access Model database, when connected to SQL Server instance. To isolate the issue further, we collected SQL Profiler trace when the used logged in to SQL Server: From the trace collected, we identified that the Login failed for model database was reported from "Microsoft SQL Server management studio-Transact-SQL Intellisense" application. The login failed error was getting reported in Error log, as soon as the user opens a new query editor session in SSMS. Tried refreshing the Intellisense Local cache as per below, but user was still getting the login failed error. Logging onto the same SQL instance using a sysadmin account in SQL, no errors were getting reported. Management studio version used : 13.0.15000 (SSMS June 2016 version). Please note that the issue was reported only on instances with SQL AlwaysOn setup. Resolution: The issue was turned out to be a known issue with SSMS June 2016 version. We performed in-place upgrade of SSMS to 16.5.3 version using the below link which fixed the login failed error due to Intellisense. https://docs.microsoft.com/en-us/sql/ssms/sql-server-management-studio-changelog-ssms#previous-ssms-releases Hope the above steps shared help you troubleshoot SQL service startup issues. Please share your feedback, questions and/or suggestions. Thanks, Don Castelino | Premier Field Engineer | Microsoft Disclaimer: All posts are provided AS IS with no warranties and confer no rights. Additionally, views expressed here are my own and not those of my employer, Microsoft.

SQL database engine service startup failure: Troubleshooting Checklist

Don Castelino[MSFT] — Tue, 27 Feb 2018 09:29:30 GMT
Hi all, In this article, we are covering the common scenarios of SQL Service startup failures and troubleshooting approach to resolve these issues: 1. For standalone instance of SQL, always use SQL Server configuration manager to start the SQL database Engine service. For Clustered instance of SQL, always use Fail over Cluster manager tool to start the SQL Database engine service. Error while starting SQL service using Configuration manager: Error While starting service from services.msc: 2. Start SQL Server from the command prompt to rule out issues such as permissions or WMI issues or service account related issues. When you start SQL Server from the command prompt, it starts as an application, not a service, with the credentials of the currently logged-in Windows user. To start SQL Server from command prompt: Locate the SQL Database engine service and copy “Path to executable” section. Open command prompt in run as administrator mode and paste the above path and use   -c parameter as mentioned below: If you are successfully able to start the SQL Service, then it’s an indication that the SQL Service account used to start the SQL Service (permission, password, locked out) need to be reviewed. If you are not able to start the SQL Service using command prompt as well, review the Error log. 3. Check if an Errorlog entry is getting created when the SQL Service is started: If it is not generated,

Check startup parameters & ensure the path and files exists (master.mdf, Error log, mastlog.ldf).

Check registry for startup parameters.

4. Check the permissions on startup parameters to check if SQL Service account has read permissions on master database files and write permission to create Error log file. For example, permission error on master database file:
2018-02-27 12:10:22.65 Server      Registry startup parameters: -d C:\Program Files\Microsoft SQL Server\MSSQL14.SQL2017\MSSQL\DATA\master.mdf -e C:\Program Files\Microsoft SQL Server\MSSQL14.SQL2017\MSSQL\Log\ERRORLOG -l C:\Program Files\Microsoft SQL Server\MSSQL14.SQL2017\MSSQL\DATA\mastlog.ldf 2018-02-27 12:10:22.65 Server      Command Line Startup Parameters: -s "SQL2017" 2018-02-27 12:10:22.65 Server      Error: 17113, Severity: 16, State: 1. 2018-02-27 12:10:22.65 Server      Error 5(Access is denied.) occurred while opening file 'C:\Program Files\Microsoft SQL Server\MSSQL14.SQL2017\MSSQL\DATA\master.mdf' to obtain configuration information at startup. An invalid startup option might have caused the error. Verify your startup options and correct or remove them if necessary. 2018-02-27 12:10:22.65 Server      SQL Server shutdown has been initiated
To troubleshoot permission issues, take process monitor while reproducing the issue and review ACCESS DENIED reported by sqlservr.exe Reviewing the event properties will show the user and the set of permissions needed. Accordingly setup the required permissions. 5. From SQL 2012, Configuration manager has a new tab for Startup parameters. This will assist in scenarios where SQL service startup issues are reported because of parameters not being added correctly (examples include missing semi-colon, white-spaces etc). 6. While reviewing the Error log, check for “Recovery is complete” message like below:
2018-02-27 12:20:42.55 spid8s      Recovery is complete. This is an informational message only. No user action is required.
Even though startup parameter values are intact, if model and tempdb database files are not accessible, SQL Service will not start. Similarly, MSDB database files are needed for SQL Agent service startup. Review the SQL Error log to check if there are any issues in accessing the system database files. For example, in the below error log, model database is not accessible:
2018-02-27 12:40:43.04 spid11s     Error: 17204, Severity: 16, State: 1. 2018-02-27 12:40:43.04 spid11s     FCB::Open failed: Could not open file C:\Program Files\Microsoft SQL Server\MSSQL14.SQL2017\MSSQL\DATA\model.mdf for file number 1. OS error: 2(The system cannot find the file specified.). 2018-02-27 12:40:43.04 spid11s     Error: 5120, Severity: 16, State: 101. 2018-02-27 12:40:43.04 spid11s     Unable to open the physical file "C:\Program Files\Microsoft SQL Server\MSSQL14.SQL2017\MSSQL\DATA\model.mdf". Operating system error 2: "2(The system cannot find the file specified.)". 2018-02-27 12:40:43.07 spid11s     Error: 945, Severity: 14, State: 2. 2018-02-27 12:40:43.07 spid11s     Database 'model' cannot be opened due to inaccessible files or insufficient memory or disk space. See the SQL Server errorlog for details. 2018-02-27 12:40:43.07 spid11s     SQL Server shutdown has been initiated
7. If the SQL Service startup fails with the error “logon failure”, review the Windows event viewer system log: User name and password is incorrect:
The MSSQL$SQL2017 service was unable to log on as machinename\sqluser with the currently configured password due to the following error: The user name or password is incorrect. To ensure that the service is configured properly, use the Services snap-in in Microsoft Management Console (MMC).
User Account is locked out:
The MSSQL$SQL2017 service was unable to log on as machinename\SQLUSER with the currently configured password due to the following error: This user can't sign in because this account is currently disabled. To ensure that the service is configured properly, use the Services snap-in in Microsoft Management Console (MMC).
Based on the above error, accordingly take an appropriate action to reset the password/unlock the account. 8. SQL Service may not start if the System databases are corrupt.
Starting up database 'master'. Starting up database 'model'. Starting up database 'msdb'. SQL Server is now ready for client connections. This is an informational message; no user action is required. Error: 5172, Severity: 16, State: 15. The header for file 'C:\Databases\MSSQL14.SQL2017\MSSQL\DATA\model.mdf' is not a valid database file header. The PageAudit property is incorrect. Error: 945, Severity: 14, State: 2. Database 'model' cannot be opened due to inaccessible files or insufficient memory or disk space.  See the SQL Server errorlog for details. Could not create tempdb. You may not have enough disk space available. Free additional disk space by deleting other files on the tempdb drive and then restart SQL Server. Check for additional errors in the event log that may indicate why the tempdb files could not be initialized.
If the system databases are corrupt, supported options to recover the system databases are either restore the system databases from last known good backup or rebuild the master database. 9. Check if SQL Service startup issue is due to OS 32 error (The process cannot access the file because it is being used by another process.): For example:
2011-04-16 07:52:49.32 spid5s      Error: 17204, Severity: 16, State: 1. 2011-04-16 07:52:49.32 spid5s      FCB::Open failed: Could not open file C:\Program Files\Microsoft SQL Server\MSSQL14.SQL2017\MSSQL\DATA\mastlog.ldf for file number 2.  OS error: 32(error not found).
In the above scenario, some other process/application has a lock on master database ldf file and this isn’t allowing SQL to take access, hence the service fails to start. To find out which process it is you can use either process explorer or handle.exe tool. The usual culprits are Anti-Virus or Anti-Malware services. 10. Check if SQL Server is configured for SSL encryption. If there are any SSL certificates loaded in SQL Server and certificate doesn’t meet the requirements of SQL Server, SQL service wont start. For example:
2018-02-27 12:36:49.39 spid16s     The server could not load the certificate it needs to initiate an SSL connection. It returned the following error: 0x8009030d. Check certificates to make sure they are valid. 2018-02-27 12:36:49.39 spid16s     Error: 26014, Severity: 16, State: 1. 2018-02-27 12:36:49.39 spid16s     Unable to load user-specified certificate [Cert Hash(sha1) "7031C0BF54FEFFD42BB7B7589BAF9A92E57A4A21"]. The server will not accept a connection. You should verify that the certificate is correctly installed. See "Configuring Certificate for Use by SSL" in Books Online. 2018-02-27 12:36:49.39 spid16s     Error: 17182, Severity: 16, State: 1. 2018-02-27 12:36:49.39 spid16s     TDSSNIClient initialization failed with error 0x80092004, status code 0x80. Reason: Unable to initialize SSL support. Cannot find object or property. 2018-02-27 12:36:49.39 spid12s     Starting up database 'mssqlsystemresource'. 2018-02-27 12:36:49.39 spid16s     Error: 17182, Severity: 16, State: 1. 2018-02-27 12:36:49.39 spid16s     TDSSNIClient initialization failed with error 0x80092004, status code 0x1. Reason: Initialization failed with an infrastructure error. Check for previous errors. Cannot find object or property.
To check if certificate is loaded in SQL, refer the below registry key: Check in SQL Configuration manager, if the certificate used is listed in drop down list. If SSL encryption for SQL Server is not needed, then remove the thumbprint entry from the SQL registry and restart the SQL Service. If SSL encryption to SQL is needed, then SSL certificate used for encryption need to evaluated to check if it meets the pre-requisites. 11. You can also use the below parameters as startup parameter to troubleshoot further: Hope the above steps shared help you troubleshoot SQL service startup issues. Please share your feedback, questions and/or suggestions. Thanks, Don Castelino | Premier Field Engineer | Microsoft Disclaimer: All posts are provided AS IS with no warranties and confer no rights. Additionally, views expressed here are my own and not those of my employer, Microsoft.

Operating System Best Practice Configurations for SQL Server

Don Castelino[MSFT] — Thu, 01 Feb 2018 13:18:24 GMT
Hi all, In this article, we are covering few important Operating System (Windows) best practice configurations for SQL Server. SQL Server is an user mode application runs on Windows Operating system and hence these configuration settings are important for SQL Server performance. 1. Partition the Disk volumes hosting SQL databases (Data and log) with 64 KB allocation unit size: A drive can be formatted with different sizes ranging from 512 bytes to 64K sizes with the default being 4KB (4096 bytes). This setting is also called as “Bytes Per Cluster”. The atomic unit of storage in SQL Server is a page which is 8KB in size. Extents are groups of eight 8 KB pages that are physically contiguous to each other for a total of 64 KB. SQL Server uses extents to store data. Hence, on a SQL Server machine the NTFS Allocation unit size hosting SQL database files (Including tempdb files) should be 64K. 2. Instant file initialization:
SQL database Data and log files are initialized to overwrite any existing data left on the disk from previously deleted files. Data and log files are first initialized by zeroing the files (filling with zeros) when one of the following operations are performed:

Create a database.

Add data or log files, to an existing database.

Increase the size of an existing file (including auto grow operations).

Restore a database or file group.

File initialization causes these operations to take longer.
In SQL Server, data files can be initialized instantaneously to avoid zeroing operations. Instant file initialization allows for fast execution of the previously mentioned file operations. Instant file initialization reclaims used disk space without filling that space with zeros. Instead, disk content is overwritten as new data is written to the files. Log files cannot be initialized instantaneously. Instant file initialization is only available if the SQL Server service startup account has been granted SE_MANAGE_VOLUME_NAME. Add SQL service startup account/Account used for backup operation to the "Perform Volume Maintenance Tasks" windows security policy.
To grant an account the Perform Volume Maintenance Tasks permission:

On the computer where the backup file will be created, open the Local Security Policy application (secpol.msc).

In the left pane, expand Local Policies, and then click User Rights Assignment.

In the right pane, double-click Perform volume maintenance tasks.

Click Add User or Group and add any user accounts that are used for backups.

Click Apply, and then close all Local Security Policy dialog boxes.

Starting with SQL Server 2016, instant file Initialization permission can be granted to the SQL service account at install time, during SQL setup. 3. OS Power Saving setting to High Performance: In Windows Server 2008 and later OS, the default power saving setting is set to Balanced, which means that components such as CPU and storage will be scaled back if the system is not busy. In some cases, this may result in performance degradation for SQL Server. If the instance of SQL Server is under heavy load and is hosted by Windows Server 2008 or later OS, set the operating system power saving plan to High Performance. 4. Antivirus exclusion on SQL files: When you configure antivirus software settings, make sure that you exclude the following files or directories on SQL Server machine from virus scanning. Doing this improves the performance of the files and helps make sure that the files are not locked when the SQL Server service must use them. However, if these files become infected, your antivirus software cannot detect the infection.

SQL Server data files (.mdf, .ndf, .ldf files)

SQL Server backup files (.bak, .trn files)

Full-Text catalog files

Trace files (.trc files )

SQL audit files for SQL Server 2008 or later versions (.sqlaudit files)

SQL query files (.sql files)

The directory that holds Analysis Services data

The directory that holds Analysis Services temporary files that are used during Analysis Services processing

Analysis Services backup files

The directory that holds Analysis Services log files

Directories for any Analysis Services 2005 and later-version partitions that are not stored in the default data directory

Filestream data files (SQL 2008 and later versions)

Remote Blob Storage files (SQL 2008 and later versions)

The directory that holds Reporting Services temporary files and Logs (RSTempFiles and LogFiles)

Processes to exclude from virus scanning
SQL Server 2016

%ProgramFiles%\Microsoft SQL Server\MSSQL13.\MSSQL\Binn\SQLServr.exe

%ProgramFiles%\Microsoft SQL Server\MSRS13.\Reporting Services\ReportServer\Bin\ReportingServicesService.exe

%ProgramFiles%\Microsoft SQL Server\MSAS13.\OLAP\Bin\MSMDSrv.exe

SQL Server 2014

%ProgramFiles%\Microsoft SQL Server\MSSQL12.\MSSQL\Binn\SQLServr.exe

%ProgramFiles%\Microsoft SQL Server\MSRS12.\Reporting Services\ReportServer\Bin\ReportingServicesService.exe

%ProgramFiles%\Microsoft SQL Server\MSAS12.\OLAP\Bin\MSMDSrv.exe

SQL Server 2012

%ProgramFiles%\Microsoft SQL Server\MSSQL11.\MSSQL\Binn\SQLServr.exe

%ProgramFiles%\Microsoft SQL Server\MSRS11.\Reporting Services\ReportServer\Bin\ReportingServicesService.exe

%ProgramFiles%\Microsoft SQL Server\MSAS11.\OLAP\Bin\MSMDSrv.exe

SQL Server 2008 R2

%ProgramFiles%\Microsoft SQL Server\MSSQL10_50.\MSSQL\Binn\SQLServr.exe

%ProgramFiles%\Microsoft SQL Server\MSSQL10_50.\Reporting Services\ReportServer\Bin\ReportingServicesService.exe

%ProgramFiles%\Microsoft SQL Server\MSSQL10_50.\OLAP\Bin\MSMDSrv.exe

SQL Server 2008

%ProgramFiles%\Microsoft SQL Server\MSSQL10.\MSSQL\Binn\SQLServr.exe

%ProgramFiles%\Microsoft SQL Server\MSSQL10.\Reporting Services\ReportServer\Bin\ReportingServicesService.exe

%ProgramFiles%\Microsoft SQL Server\MSSQL10.\OLAP\Bin\MSMDSrv.exe

If you back up the database to a disk or if you back up the transaction log to a disk, you can exclude the backup files from the virus scanning.

You can run antivirus software on a SQL Server cluster. However, you must make sure that the antivirus software is a cluster-aware version. If you are running antivirus software on a cluster, make sure that you also exclude these locations from virus scanning:

Q:\ (Quorum drive)

C:\Windows\Cluster

Please refer the article for more information: https://support.microsoft.com/en-us/help/309422/how-to-choose-antivirus-software-to-run-on-computers-that-are-running

5. Lock Pages In memory: Lock pages in memory privilege to SQL is recommended in Windows Server 2008 or above only when there are signs of paging. Locking pages in memory may boost performance when paging memory to disk is expected.
To enable the lock pages in memory option, add the SQL Service account to "Lock pages in memory" security policy and restart the SQL database engine service.
Before enabling LPIM for SQL server, review the Performance monitor data for other applications running on the server and check if check working set memory of SQL is trimmed and accordingly setup LPIM. 6. Setting appropriate Page file size: Windows uses the paging file (pagefile.sys) as secondary random-access memory (RAM). The paging file and physical memory make up virtual memory. By default, Windows stores the paging file on the boot partition. When Windows 2008 and later page file is configured to "Automatically manage paging file for all drives", page file is managed by the system according to how much virtual memory and disk space you have. By default, "Automatically manage paging file size for all drives" setting is selected so that Windows 2008 or later system can manage the paging file without users interruption and configures roughly to two times to the size of physical RAM. If you want to change the paging file size, move the pagefile.sys to another drive, or disable virtual memory paging, uncheck the check box of Automatically manage paging file size for all drives. To set Paging file to Custom Size: 1. Click Start, right-click Computer and select Properties 2. Select the Advanced System Settings 3. Under Performance, Click Settings button. 4. Click the Advanced tab 5. Under Virtual Memory, click Change button 6. If "Automatically manage paging file size for all drives" is selected, uncheck the option 7. Select the appropriate drives 8. Choose Custom size 9. Set Initial Size (MB) and Maximum Size (MB) as appropriate. Follow the guidelines as discussed in the KB below to determine the appropriate page file size for 64-bit versions of Windows 2008 and above: https://support.microsoft.com/en-us/help/2860880/how-to-determine-the-appropriate-page-file-size-for-64-bit-versions-of 7. Windows security policy and permissions The account assigned to start SQL service needs the Start, stop and pause permission for the service. The SQL Server Setup program automatically assigns this. SQL Setup account adds the SQL service account as part of below security policies:

Log on as a service

Replace a process-level token

Bypass traverse checking

Adjust memory quotas for a process

Permission to start SQL Writer

Permission to read the Event Log service

Permission to read the Remote Procedure Call service

If SQL Service account need to be changed, always use SQL Server tools such as SQL Server Configuration Manager to change the account used by the SQL Server Database Engine or SQL Server Agent services, or to change the password for the account. In addition to changing the account name, SQL Server Configuration Manager performs additional configuration such as updating the Windows local security store which protects the service master key for the Database Engine. Other tools such as the Windows Services Control Manager can change the account name but do not change all the required settings. To change Reporting Services options, use the Reporting Services Configuration Tool. SQL Server Setup does not open ports in the Windows firewall. Connections from other computers may not be possible until the Database Engine is configured to listen on a TCP port, and the appropriate port is opened for connections in the Windows firewall. To access an instance of the SQL Server through a firewall, configure appropriate inbound and outbound firewall rules on the SQL Server machine.
File System Permissions Granted to SQL Server service SID's or Local Windows Groups and registry permissions are documented here: https://docs.microsoft.com/en-us/sql/database-engine/configure-windows/configure-windows-service-accounts-and-permissions
8. Dynamic Quorum Configuration in Windows 2012 and 2012R2 cluster: In Windows Server 2012 clusters and later versions, the Dynamic Quorum gives the administrator the ability to automatically manage the quorum vote assignment for a node, based on the state of the node. When a node shuts down or crashes, the node loses its quorum vote. When a node successfully rejoins the cluster, it regains its quorum vote. By dynamically adjusting the assignment of quorum votes, the cluster can increase or decrease the number of quorum votes that are required to keep running. This enables the cluster to maintain availability during sequential node failures or shutdowns. In Windows Server 2012 R2 clusters, if the cluster is configured to use dynamic quorum (the default), the witness vote is also dynamically adjusted based on the number of voting nodes in current cluster membership. If there are an odd number of votes, the quorum witness does not have a vote. If there is an even number of votes, the quorum witness has a vote. The quorum witness vote is also dynamically adjusted based on the state of the witness resource. If the witness resource is offline or failed, the cluster sets the witness vote to 0. Ensure Dynamic Quorum is enabled. In addition, on Windows Server 2012 R2 cluster, use Node and Disk Majority or Node and File Share Majority as the quorum model. Please share your feedback, questions and/or suggestions. Thanks, Don Castelino | Premier Field Engineer | Microsoft Disclaimer: All posts are provided AS IS with no warranties and confer no rights. Additionally, views expressed here are my own and not those of my employer, Microsoft.

SSIS Package Data flow Execution Slowness Troubleshooting

Don Castelino[MSFT] — Tue, 30 Jan 2018 06:53:50 GMT
Hi all, In this blog, we are covering "SQL SSIS package data flow execution slowness" troubleshooting guidelines. The approach mentioned in the blog can be used for reference while troubleshooting SSIS package data load performance issues. Consider a simple SSIS data flow task, which is selecting the data from the OLE DB source and inserts the rows to the destination table using OLE DB destination task. For data flow execution slowness issues, its essential to narrow down which task in data flow is slow. The issue could be at: OLE DB source Transformation task OLE DB destination To identify if the Source is causing slowness: To check if the issue is with the source or not, replace OLE DB destination task with Multicast task and check the execution time. Multicast task Distributes every input row to every row in one or more outputs. For example, branch your data flow to make a copy of data so that some values can be masked before sharing with external partners. Record the execution time using SSIS execution logs. Another way to check the slowness is at Source or not, use Row count task. Create a user defined variable with Int data type and execute the package. Record the execution time and compare with the execution time with OLE DB destination task. If The package execution with Row Count task takes more time to execute, then review the source database for performance bottlenecks. Review the execution time of the select when the package is executing on the source server. The data flow engine uses a buffer-oriented architecture to efficiently load and to manipulate datasets in memory. Reviewing the properties of the Data flow task has the below options: DefaultBufferMaxRows, default value is 10000. DefaultBufferSize, default value is 10 MB. The data flow engine begins the task of sizing its buffers by calculating the estimated size of a single row of data. Then it multiplies the estimated size of a row by the value of DefaultBufferMaxRows to obtain a preliminary working value for the buffer size. Hence reduce your Estimated Row Size as much as possible by removing any unnecessary columns and configuring data types correctly.

If the result is more than the value of DefaultBufferSize, the engine reduces the number of rows.

If the result is less than the internally-calculated minimum buffer size, the engine increases the number of rows.

If the result falls between the minimum buffer size and the value of DefaultBufferSize, the engine sizes the buffer as close as possible to the estimated row size times the value of DefaultBufferMaxRows.

Before adjusting the sizing of the buffers, improvement can be achieved to great extent by reducing the size of each row of data by removing unneeded columns and by configuring data types appropriately. When sufficient memory is available, smaller number of large buffers provide better performance by reducing the total number of buffers required to hold the data, and by fitting as many rows of data into a buffer as possible. To determine the optimum number of buffers and their size, experiment with the values of DefaultBufferSize and DefaultBufferMaxRows while monitoring performance. Do not increase buffer size to the point where paging to disk starts to occur. Paging to disk hinders performance more than a buffer size that has not been optimized. To identify if the Destination is causing slowness: With the steps mentioned in above section, if execution of the data flow execution takes much less time in fetching the data from the source, then the focus should be to tune the destination. Tuning OLE DB destination involves, reviewing the Performance on destination server for any bottlenecks during data load, check for lock blocking, review wait types etc. In the OLE DB destination task, review if changing the data access to mode to “Table or View- fast load” improves the performance. When Table or view data access mode is used, SQL uses sp_cursor to insert the rows on destination table. As per the screenshot, 1 row is inserted on destination at a time. When "Table or view - fast load" data access method is used, SQL uses insert bulk command to insert the data on destination table. Review the RowCounts column. Modifying DefaultBufferMaxRows and DefaultBufferSize has an impact on the rows transferred in insert bulk command. As per below, when DefaultBufferMaxRows value is changed to 100000 and DefaultBufferSize to 100MB, SQL inserted 99879 rows in one batch. SSIS has a transformation task called the Balanced Data Distributor (BDD) which provides an easy way to make use of multi-processor and multi-core servers by introducing parallelism in the data flow of an SSIS package. Balanced Data distributor task can be used which evaluates and directs rows in a dataset to multiple destinations. Evaluate the SSIS package execution with BDD task and check the execution time for improvement. BDD can be downloaded from https://www.microsoft.com/en-us/download/details.aspx?id=4123 Here the number of rows is split across 4 OLE DB destination tasks to achieve parallelism. Review the logic of the SSIS data load, if delta inserts are performed or complete reload of the table is done. In complete reload scenarios, we have seen performance gain by dropping Indexes before data load process and by creating them post data load. This will improve in scenarios where Index insert is causing an overhead impacting the performance. Post implementing the above steps mentioned, post eliminating source and destination bottlenecks, evaluate the network from source to the destination. Check if the SSIS package is being called from an application server. Review if the Network packets are getting dropped/re transmit of Network packets which can cause major performance issues. 32-bit execution mode of SSIS package has process virtual memory address limitation. With adequate memory, it’s possible to get as many records into a buffer with fewer but larger buffers, by tweaking DefaultMaxBufferRows and DefaultBufferSize appropriately. Please note that changing package execution mode to 64-bit would require updating the data providers users to connect to the source and destination. Please review if the source/destination is compatible with 64-bit providers and check if the package can be migrated to 64-bit. Check if changing the provider at the source/destination has any impact on the data flow task. We have seen performance improvements in data flow, when Microsoft Attunity provider for Oracle is used for Oracle data source. So, check with different providers available and check if they improve the performance. Hope the above steps mentioned help you in troubleshooting SSIS package execution slowness issues. Please share your feedback, questions and/or suggestions. Thanks, Don Rohan Castelino | Premier Field Engineer | Microsoft Disclaimer: All posts are provided AS IS with no warranties and confer no rights. Additionally, views expressed here are my own and not those of my employer, Microsoft.

Isolating SQL Performance Issues due to slow Disk IO subsystem

Don Castelino[MSFT] — Mon, 29 Jan 2018 05:27:56 GMT
Hello all, In this blog, we are covering few checklists to identify and isolate disk IO subsystem bottlenecks causing SQL performance issues. The checklist can be used as reference while troubleshooting SQL Performance issues because of slow Disk IO subsystem. Below screenshot summarizes the approach to troubleshooting Slow disk IO issues: Checklist for troubleshooting: 1.Check in SQL Server Error logs for "I/O requests taking longer than 15 seconds" warning message:  SQL Server performance highly relies on the Disk performance. If the IO taking more than 15 seconds are reported frequently, its an indication of slow Disk IO subsystem. SQL Server has encountered x occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [Drive:\MSSQL\MSSQL.1\MSSQL\Data\xyz.mdf] in database [database]. The OS file handle is 0x00000000. The offset of the latest long I/O is: 0x00000000000000 2. Check through dynamic management views ( sys.dm_os_wait_stats, sys.dm_exec_requests) if the queries are having the below wait types. Below wait types indicate that the threads are waiting on IO completion. PAGEIOLATCH_* IO_COMPLETION ASYNC_IO_COMPLETION WRITELOG BACKUPIO 3. Review the Windows Event viewer system logs for NTFS related errors. Some common errors are listed below: 4. Windows Performance monitor tool (perfmon) is a great tool to review the Disk counters and isolate the slowness. Add the below counters for the drives hosting SQL database files. LogicalDisk (Drive:)\Avg. Disk Queue Length >>>> should not be above 5. LogicalDisk (Drive:)\Avg. Disk sec/Read LogicalDisk (Drive:)\Avg. Disk sec/Transfer LogicalDisk (Drive:)\Avg. Disk sec/Write Benchmark: < 0.005 excellent; 0.005 – 0.010 Good; 0.010 – 0.025 Fair; > 0.025 Disk bottleneck LogicalDisk (Drive:)\Disk Bytes/sec >>>>Above 300 MB/sec, need to be investigated. If LogicalDisk (Drive:)\Disk Bytes/sec is 300 MB/sec, then load the below counters to check which process is throttling the IO subsystem. Process(_Total) \IO Data Bytes/sec Process(_Total) \IO Read Bytes/sec Process(_Total) \IO Write Bytes/sec Process(sqlservr)\IO Data Bytes/sec Process(sqlservr)\IO Read Bytes/sec Process(sqlservr)\IO Write Bytes/sec If SQL Server is posting more IO compared to the total IO posted, then we need to focus on SQL queries which are posting high IO. If SQL Server IO data bytes/sec is much less compared to total IO data bytes/sec, then load the counters for other processes and check which process is posting high IO. To confirm SQL Server is doing a lot of reads/writes, look at below counters: SQL Server: Buffer Manager: Page Read/sec SQL Server: Buffer Manager: Page Writes/sec If you see the values are more than 20,000 consistently, then clearly queries are driving the high volumes of data. Please note that the above counters posted are for reference purpose only. 6. Check for any Flushcache messages in SQL Error log like below: FlushCache: cleaned up xxxxxx bufs with xxxxx writes in xxxxx ms (avoided xxxxx new dirty bufs) for db x:0 average throughput: xx.xxMB/sec, I/O saturation: xxxxx, context switches xxxxx last target outstanding: xx, avgWriteLatency xx FlushCache is the SQL Server routine that performs the checkpoint operation. Post SQL Server 2012, if the above messages are posted in SQL Error log, then it’s in indication that long checkpoint operation on a database has exceeded the configured recovery interval. Hope the above steps mentioned help you in troubleshooting Disk bottlenecks in SQL environment. Please share your feedback, questions and/or suggestions. Thanks, Don Castelino | Premier Field Engineer | Microsoft Disclaimer: All posts are provided AS IS with no warranties and confer no rights. Additionally, views expressed here are my own and not those of my employer, Microsoft.