Poojakamath

Find the Session ID of an executing SQL Agent job

Pooja Kamath — Thu, 19 Jul 2018 07:50:26 GMT

On a busy production server I wanted to check when a job started running , I tried to look at the sysjob history table in the MSDB However there are no entries for inflight jobs (meaning no entries for job that haven't failed or pass) The sysjobschedules table from msdb will only show you failed or successful runs While there are scripts available online to do this , you can use the below DMV's as well select * from sys.jobs -->Copy the job ID of your job name select PROGRAM_NAME , * from sysprocesses where spid> 50 The result will be something like: SQLAgent - TSQL JobStep (Job xx23453AB7EC97864084xxxxxF72B8CC9E : Step 1) -->Make sure the program_name matches the Job ID you are looking for Get the spid from here --> You can now use it in below DMV's to check for blockings or waittypes select * from sys.dm_exec_requests where session_id = 102 select last_batch, * from sysprocesses where spid = 102

Where is my SQL BPA(Best Practice Analyzer) for SQL 2014/2016/2017?

Pooja Kamath — Thu, 03 May 2018 09:24:47 GMT

We recently built a SQL 2016 Server for a client and wanted to check if the best practices have been followed or not. Most of us have used the tool BPA (Best Practice Analyzer) to evaluate an instance. The Microsoft SQL Server 2012 BPA is a diagnostic tool that performs the following functions:

Gathers information about a Server and a Microsoft SQL Server 2012 instance installed on that Server
Determines if the configurations are set according to the recommended best practices
Reports on all configurations, indicating settings that differ from recommendations
Indicates potential problems in the installed instance of SQL Server
Recommends solutions to potential problems

The latest version I worked with was the BPA for SQL 2012 found here Now we didn’t have this tool available for download for SQL 2014 or SQL 2016. It appears the BPA has been replaced by Policy Evaluation using built in Best Practice Templates. Here is how you can perform one. These are the policies that will be evaluated as part of the template : https://docs.microsoft.com/en-us/sql/relational-databases/policy-based-management/monitor-and-enforce-best-practices-by-using-policy-based-management?view=sql-server-2017 To perform an on-demand evaluation by using Object Explorer

Start Management Studio, and then connect to the Database Engine.
In Object Explorer, expand Management, expand Policy Management, right-click Policies, and then click Evaluate.

2. In the Evaluate Policies dialog box, next to the Source box, click the ellipsis (…) button 3. In the Select Source dialog box, you can select either Files or Server as the source of the policy files to evaluate. Click Files, and then select the individual policy files that you want to evaluate. To do this, follow these steps:

Click Files.
Next to Files, click the ellipsis (…) button.
In the Select Policydialog box, browse to the following folder, which contains the best practices

C:\Program Files (x86)\Microsoft SQL Server\110\Tools\Policies\DatabaseEngine\1033

Select one or more .xml policy files to evaluate, and then click Open.

The list of selected files appears in the Files box.

In the Select Sourcedialog box, click OK.
If the Loading Policiesdialog box appears, click Close.
The policies that you selected are listed on the Policy Selection page. Be aware that a warning icon next to a policy indicates that the policy contains scripts.
1. Click Evaluate to evaluate the policies.

In the Results table, the results for each policy are listed. A red "x" icon indicates that policy compliance failed. You can click on the Details to see what policy has failed Make sure that you fully understand the policy setting before automatically updating a target instance.

Configure Alerts for a Hung/Stuck SQL Agent Job step

Pooja Kamath — Fri, 20 Apr 2018 13:59:43 GMT

SQL Server does a decent job alerting us of SQL Agent Job failures/ Success But, what if we want to know if a job is hung, been running for a long time and has been stuck? When my customer approached me with the below problem I realized that the Job history table in the MSDB does not have entries for Job or Job Steps that are hung or in Transit. It only adds an entry to the Job History table after the Job has completed or Failed. We had the following SQL Agent Job in our environment 1. We were using an SSIS package with an WMI Event Watcher Task to continuously watch for arrival of new files. Each day the new file would arrive at 3AM 2. After this there were more steps in the Job execution which would process the file..etc The Job had steps 1 to 8 The problem was that it was getting hung on the 2nd step which was the SSIS Package execution. This was to process the file after the WMI Event Watcher Task determined that the file had arrived. It was getting hung “intermittently” with no specific pattern , even after the file has arrived. Each time the application user would raise a complaint that they did not recieve a file for processing their Daily report which would then alert the SQL team. They would find the Job stuck executing the Step 2. To fix it they would check if the file had arrived and then jump to step3. We needed a proactive way to alert us if the Job step was stuck. Hence we used the below approach Find your Step_id and Job_Id from Use msdb go select * from dbo.sysjobsteps We created a new SQL Agent Job for msdb database to alert us: declare @run_date int set @run_date = (select TOP 1 run_date from dbo.sysjobhistory where step_id = 2 and job_id='XXXXXXX-XXX-XXXX-XXXX-XXXXXXX' ORDER BY run_date DESC) --Getting the latest run_date of the Job step declare @return_value int = 0; declare @d1 int = (SELECT YEAR(GETDATE()) * 10000 + MONTH(GETDATE()) * 100 + DAY(GETDATE())) -- Getting todays date in the same format as the last run_date declare @run_time int = (select TOP 1 run_time from dbo.sysjobhistory where step_id =2 and job_id='XXXXXX-XXX-XXXX-XXXX-XXXXXXX' ORDER BY run_date DESC) --Getting the latest run_time of the Job Step declare @threshold_time int = 50000 --Declare a threshold time at which you want to be alerted. Example Job step was to complete at 3am ,but even at 5AM it hasnt posted a success message --Here 5am is your threshold_time if @d1 > @run_date and @run_time < @threshold_time SET @return_value = 1 ; -- If the Job History doesnt have run_date record for Today and if Job History shows that the latest run_time is less than the threshold_time --it means the Job Step hasn't run or is probably stuck , which means we should be alerted IF (@return_value) > 0 begin exec msdb.dbo.sp_send_dbmail @profile_name = 'Default', @recipients = 'me@example.com', @subject = 'warning step 2 is stuck' end To setup Database Mail you can refer the following steps We now get alerted when our job step is stuck! Hope this helps if someone has the same issue. If you have any more ideas on how to approach this please do share in the comment section below:)

SQL 2012 BPA error- 'Login does not exist on SQL Server OR Login is not a member of the Systems Administrator role'

Pooja Kamath — Mon, 08 May 2017 11:51:15 GMT

I was recently trying to use the diagnostic tool Microsoft SQL Server 2012 Best Practice Analyzer on my customer's machine that was getting ready for a migration. · Using this tool a DBA can determine whether the SQL Server is configured as per the best practices recommended by the SQL Server Products Team. · This tool validates your instance of SQL Server with certain built-in rules that can help you rectify common installation and configuration issues. While this is a great tool that generates a ready report for us, we were not able to get it running on a default cluster instance. We were able to successfully generate reports on a Standalone instance and a named cluster instance without issues but the default cluster instance was throwing an error " Login does not exist on SQL Server OR Login is not a member of the Systems Administrator role" We made sure we belonged to the local windows administrator's group on both nodes of the (active/passive) cluster – both as an individual login and as a member of a domain group. The AD account was in the SQL sysadmin group too. We have couple of online forums and blogs suggesting the modification of Engine.ps1 file but they didn't work for us and we ended up with the same error. Sharing what finally worked for us and allowed us to generate a report Workaround: ===========

Open up the file Engine.ps1 to modify

Should be found in %Programdata%\Microsoft\Microsoft Baseline Configuration Analyzer 2\Modules\SQL2012BPA\Engine 2. We added this $SQLInstanceID =”MSSQL11.MSSQLServer” just below the line $SQLInstanceID = $SQLInstallPath[3] to make this work. 3. Run the Best Practice Analyzer GUI without providing the name of the Instance (leave it blank) Hope this simple solution saves some valuable time for someone attempting to run the BPA on a Default Clustered Instance of SQL 2012 Solution Credits to John Hatricks for parsing through the Powershell script to find this solution!

Using REST API to pause Azure SQL Data Warehouse - “Bearer authorization error='invalid_token', error_description='The authentication scheme of not supported”

Pooja Kamath — Sat, 24 Dec 2016 11:24:58 GMT

A customer of mine recently approached me for assistance to help resolve an Authorization error he was receiving , when he was running a custom prepared automation script in C# that would pause and restart the SQL data warehouse.

To save costs, you can pause and resume compute resources on-demand. He had tried the instructions provided here https://azure.microsoft.com/en-in/documentation/articles/sql-data-warehouse-manage-compute-rest-api/

The URL created looked like this:
https://management.azure.com/subscriptions/ ID>/resourceGroups//providers/Microsoft.Sql/servers/.database.windows.net/databases//pause?api-version=2014-04-01-preview+HTTP%2F1.1

It asked for an Authorization option and he provided the Authorization by following the link.
https://msdn.microsoft.com/en-IN/library/azure/dn798668.aspx

After getting the Authorization key, when he executed the REST API to pause the SQL database from REST client It gave the following error

“Bearer authorization_uri='https://login.windows.net/xxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx', error='invalid_token', error_description='The authentication scheme of eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Ik1uQ19WWmNBVGZNNXBPWWlKSE1iYTlnb0VLWSIsImtpZCI6Ik1uQ19WWmNBVGZNNXBPWWlKSE1iYTlnb0VLWSJ9.eyJhdWQiOiJodHRwczovL21hbmFnZW1lbnQuYXp1cmUuY29tLyIsImlzcyI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0LzY4MjE5NTc0LWZjN2EtNGY0Yy05ZjY5LWYyZDcwMjZkMmY0MC8iLCJpYXQiOjE0Njk3NjU2OTgsIm5iZiI6MTQ2OTc2NTY5OCwiZXhwIjoxNDY5NzY5NTk4LCJhcHBpZCI6IjEyZmQ0MTU1LWU4YTEtNGEyMi1iOTE3LWRiMWE3YzNmYjcyZSIsImFwcGlkYWNyIjoiMSIsImlkcCI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0LzY4MjE5NTc0LWZjN2EtNGY0Yy05ZjY5LWYyZDcwMjZkMmY0MC8iLCJvaWQiOiI2YWU3ZjkyMy0xMTBjLTQ3ZDItODg0Yi05M2I5ZmM4OThiMGMiLCJzdWIiOiI2YWU3ZjkyMy0xMTBjLTQ3ZDItODg0Yi05M2I5ZmM4OThiMGMiLCJ0aWQiOiI2ODIxOTU3NC1mYzdhLTRmNGMtOWY2OS1mMmQ3MDI2ZDJmNDAiLCJ2ZXIiOiIxLjAifQ.BfIsC85k4HZTIOnSaM_MqnvJHUJG8eBUhSPxKTm4qgkPQ366DN4chQeNoLIz51Gmd4cxZqcUR7zf8ZCA8iJMe9Per-tTUoyHoYHUqVt07d0HKmsHELEsFuEAveAFZM6jL_3oYHPBOQvmvzrsHnjMxVU2JL7e1_V8j9qADpMi3G1GuimsyQU2ObojhV1FkVOoFJD7p2BaBDOfHRuyJ_r32SIdYc35D7yyrT6lJ7Kdb6ZZrtPFjMaxiecSTe-_nvMD2O6rJBoA_bo2gTaf8nlW6PJIfovGPyvCceqKo1wAMbrPbEpHKJYryfP42v-iz81yr_Y51Ye0teumsmwJdPJKvQ is not supported”

Steps to resolution:

+++++++++++++++++++++++++++++++++

A simple way of testing whether the REST API calls are working is by using POSTMAN client which is an add-on to Google chrome browser. It is an intuitive user interface that would make testing APIs effortless

My colleague Suhas recently blogged about using Testing Azure REST API using POSTMAN Client - The steps and the screenshots to setup the environment to test the Azure REST API using POSTMAN Client is described very well in his blog https://blogs.msdn.microsoft.com/suhaspv/2016/11/22/geo-replication-fail-over-testing-using-rest-api/

I decided to give this a shot to save time for testing!

I was able to test the scenario out using the client (Postman app) and was able to pause and unpause my Azure SQL DW successfully by altering the REST API to our scenario

I used the REST API from the following: https://azure.microsoft.com/en-in/documentation/articles/sql-data-warehouse-manage-compute-rest-api/

Rest API calls used:

POST https://management.azure.com/subscriptions//resourceGroups//providers/Microsoft.Sql/servers//databases//resume?api-version=2014-04-01-preview

POST https://management.azure.com/subscriptions//resourceGroups//providers/Microsoft.Sql/servers//databases//pause?api-version=2014-04-01-preview

Note: I did not use the api-version=2014-04-01-preview HTTP/1.1 as mentioned in the above article as I was getting an error with it. So you may want to try with both to be sure.

However my customer was still receiving an error like the below when he tried attempt to generate a Request Token in Postman even though he provided a Manage Service API Access

Additional technical information:

Correlation ID: 6e4eda78-e520-496e-b897-0a397d46b325

Timestamp: 2016-08-01 14:03:44Z

TESTAPP90093: This application requires application permissions to another application. Consent for application permissions can only be performed by an administrator. Sign out and sign in as an administrator or contact one of your organization's administrators.

This error message here was more descriptive than the earlier authorization error and gave us a clear idea that we were missing some admin access.

++ We found that the customer was a Co-Admin on the subscription but was still giving us this error

We figured that - A subscription co-administrator does not automatically make you an Azure AD administrator and You'd need explicit role grant on the target Azure AD.

Once the customer was granted Global Administrator in the Azure AD domain he was able to generate the token without the error

After this we were able to successfully Pause/Resume the Azure SQL DW using the calls above.

More info on different AD roles: https://docs.microsoft.com/en-us/azure/active-directory/active-directory-assign-admin-roles

Now that we had it working with the Postman Client the customer was able to use the REST API calls above in his automation script in C#:

OAuth 2.0 is what we need which is explained in below article https://azure.microsoft.com/en-us/documentation/articles/resource-manager-api-authentication/#connect-subscription-to-application

Hope this helps! J

Automatically Monitor and Change the service tier and performance level (pricing tier) of an Azure SQL database based on the DTU alert using Webhooks, Alerts, Runbooks

Pooja Kamath — Sat, 24 Dec 2016 09:06:47 GMT

We all know how easy it is to change the service tier of an Azure SQL database as the needs of your application change using the Azure Portal/PowerShell.

But what if you want to automate the upgrade and the downgrade of your database?

And, what if you have many databases and don’t want to be individually monitoring the DTU usage and then have to manually fire an upgrade?

You can achieve this using Azure Automation! Recently introduced as a public preview service, Azure Automation brings a powerful, much needed PowerShell Workflow execution service to the Azure platform.

In this blog, I’ll show you how to leverage the Azure Automation service combined with Alerts and Webhooks to accomplish a customer requested scenario- (To automatically scale up the DTU’s of the database only if there is a resource crunch)

Step 1: Create your Automation Account:

To do the Automation you have to create an automation account

Add the details for your Automation account and proceed

Step 2: Create a Runbook under the Automation account you created

Click on the Add a runbook button and then Create a new runbook with the Type “Powershell Workflow”

Copy the UpdateSLO script from the below link (taken from the Powershell script gallery) to the run book.
Save the run book after this.

https://gallery.technet.microsoft.com/scriptcenter/Azure-SQL-Database-e957354f

Create a credential with the server admin user and password like the one mentioned in the below screenshot.

Add a credential

Fill in the details/ description for the New Credential

Publish the Runbook once you have created the credential

You may receive an error message like the below if you haven’t change the workflow name

Make sure the Workflow name in the script reflects the name of the runbook you created earlier to avoid the above error

Step 3: Create a new Webhook

& copy the Webhook URL after the creation (Save it on a notepad for later use)

Webhooks allow you to route an Azure alert notification to other systems for post-processing or custom actions.

Create the new Webhook

Edit the parameters of the Webhook to reflect the SQL Server name and the Database name we are configuring this for.

Use the Credential we configured earlier in the Credential tab.

I have a database that is usually in the Basic Edition and I would like to scale it to S1 when the DTU goes above 70% so I fill in the Edition parameter to Standard and PerfLevel to S0
You can choose from the following based on your requirement

Edition: Basic, Standard, Premium
PerfLevel: Basic, S0, S1, S2, P1, P2, P3

· You can run a test (Which will not actually scale up the database, but will check if the workflow has any errors)

Step 3: Configuring the Alert to call the Webhook we created:

Now that we have our Webhook configured with the Runbook and we have tested it, we will need to configure an Alert to be able to call the webhook when it triggers due a condition being met.

You can configure an alert to do the following when it triggers:

· send email notifications to the service administrator and co-administrators

· send email to additional emails that you specify.

· call a webhook

· start execution of an Azure runbook (only from the Azure portal)

The steps to Configure an alert from the Portal are documented here : https://docs.microsoft.com/en-in/azure/monitoring-and-diagnostics/insights-alerts-portal

The only difference would be to include the Webhook URL in this alert that we have configured.

You can also edit an already existing alert and add the Web hook URL to it

You can check the status of the Runbook Job and will be able to see that it is Queued as we haven’t specified a schedule:

Step 4: Testing the configured Alert, Webhook and Runbook

Now let’s test a scenario to check if this works as expected
I run a test workload to increase the resource utilization of my database for which we have configured the DTU alert on

Monitoring the alert I see that the DTU usage is steadily increasing, until it triggers the alert once the condition is met

(DTU threshold greater than 60% for the last 5 minutes)

Now that the alert has triggered, it should have called the Webhook to start the execution of the Runbook.
Let’s check the Database blade in the Azure Portal to see if upgrade has indeed started

You can see that the SLO upgrade has automatically kicked in and it has already begun to upgrade the database from Basic to the S0 we had configured

You can also check the Job status from the Runbook blade

In the Azure portal, select Automation and then then click the name of your automation account.
Select the Runbooks tab.
Click your runbook to open its Runbook blade to check if it is running.

Monitor it to completion

You can see that the database is now in S0 Standard

If you want to keep a track of when you database was upgraded based on the Alert trigger you can check the Activity log from the Database blade in the Azure Portal.

In this example I have filtered this for activity from the last one hour only.

Now you do not have to worry about monitoring your Alert emails and then taking a reactive action of upgrading your database.

The Webhooks allows you to route the High DTU utilization Azure alert notification to use the Runbook with the configured Powershell script to take action when the condition is triggered even when you are fast asleep or vacationing away!
Hope this helps!

Could not obtain exclusive lock on database 'model'. Retry the operation later. CREATE DATABASE failed

Pooja Kamath — Tue, 23 Dec 2014 23:02:18 GMT

After installing a SQL server 2008 R2 instance SQL_SERVER\MSSQLSERVER2008R2 side by side on the Box which already has SQL 2005 production instance, we are unable to create a database from the management studio

Receiving the below error

Create failed for Database 'x'. (Microsoft.SqlServer.Smo)

------------------------------

ADDITIONAL INFORMATION:

An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)

------------------------------

Could not obtain exclusive lock on database 'model'. Retry the operation later.

CREATE DATABASE failed. Some file names listed could not be created. Check related errors. (Microsoft SQL Server, Error: 1807)

Cause:

By design when we create a database we need exclusive lock on model. When SQL Server creates a new database, "The SQL Server uses a copy of the model database to initialize the database and its metadata". Also, users could create, modify, and drop objects in the Model database. As such, it is important to exclusively lock the model database to prevent copying the data in change from the Model database. Otherwise, there is no guarantee that the content copied from the Model database are consistent and valid.

We need to check what is taking the lock on model database
We ran sp_who2 and found a couple of spids using model with .Net SQLClient Data Provider" as the application
According to http://blog.sqlauthority.com/2010/01/01/sql-server-fix-error-1807-could-not-obtain-exclusive-lock-on-database-model-retry-the-operation-later/ - we tried to Disconnect and Reconnect your SQL Server Management Studio's session. Your error will go away.
But this wasn't working for us.
Checked the other basic things
-SQL startup account was Local System
-We had permissions on the folder
-Same file names were not being used
--They have 6.36TB free in the drive where we create the database.
We also found that in order to create a database on the SQL 2005 prod instance we need to turn off SQL 2008 instance (Strange!)

#---How to identity:

Use master

IF EXISTS(SELECT request_session_id FROM sys.dm_tran_locks

WHERE resource_database_id = DB_ID('Model'))

PRINT 'Model Database being used by some other session'

ELSE

PRINT 'Model Database not used by other session'

-----You can identify what kind of statement being used by the session...using the script given below

SELECT request_session_id FROM sys.dm_tran_locks

WHERE resource_database_id = DB_ID('Model')

-------We have to identity what kind of query/activity being performed by this session(52).

DBCC InputBuffer(52)

----EventInfo column returns the query performed, Based on that, you have to decide whether the Session ID(52) can be Aborted/ Not. If you want to abort the session (52),
run the command
Kill 52

--Now immediately create a database (Before another application takes a lock)

Resolution

From the sp_who2 output we see that the login XLOGIN\Administrator is continually querying the model database which in turn is not allowing us to get an exclusive lock on the database

SPID Status                         Login                                HostName   BlkBy DBName Command          CPUTime DiskIO LastBatch      ProgramName                                    SPID REQUESTID
----- ------------------------------ ------------------------------------ ---------- ----- ------ ---------------- ------- ------ -------------- ---------------------------------------------- ----- ---------

51    sleeping                       XLOGIN\Administrator                   SQL_SERVER   .   master AWAITING COMMAND 0       0      08/06 11:48:37 .Net SqlClient Data Provider                   51    0
52    sleeping                       XLOGIN\Administrator                   SQL_SERVER   .   tempdb AWAITING COMMAND 0       0      08/06 11:48:37 .Net SqlClient Data Provider                   52    0
53    sleeping                       XLOGIN\Administrator                   SQL_SERVER   .   msdb   AWAITING COMMAND 0       0      08/06 11:48:37 .Net SqlClient Data Provider                   53    0
54    sleeping                       XLOGIN\Administrator                   SQL_SERVER   .   tempdb AWAITING COMMAND 16      0      08/06 11:46:25 .Net SqlClient Data Provider                   54    0
55    sleeping                       XLOGIN\Administrator                   SQL_SERVER   .   model AWAITING COMMAND 16      0      08/06 11:47:31 .Net SqlClient Data Provider                   55    0
56    sleeping                       XLOGIN\Administrator                   SQL_SERVER   .   model AWAITING COMMAND 0       0      08/06 11:46:25 .Net SqlClient Data Provider                   56    0
57    sleeping                       XLOGIN\Administrator                   SQL_SERVER   .   model AWAITING COMMAND 0       0      08/06 11:45:19 .Net SqlClient Data Provider

From the input buffer we see the following every time we checked a spid
Select cat.name as [Name] FROM sys.fulltext_catalogs AS cat ORDER BY [Name] Asc
What was strange was that we did not have Full text Search Service running at all. In fact it isn't installed for SQL 2008R2

There are numerous articles one can find online about this issue, that suggest killing the spid that is taking the lock on the model database. This works for us too, however the solution is temporary.

If I kill the process, in about a minute, it automatically starts up the very same Select statement again ,which was strange!

So we dropped and recreated the login XLOGIN\Administrator and the issue hasn't occurred after this. Querying the sp_who2 output doesn't show us the Select statement that was earlier taking the lock.

Hope this simple resolution helps someone who is facing the same issue! J

There is insufficient system memory in resource pool 'internal' to run this query. Error: 701, Severity: 17, State: 123" on SSB Target Server - CACHESTORE_BROKERTO is consuming memory

Pooja Kamath — Tue, 23 Dec 2014 21:58:39 GMT

I recently worked with a customer where the Target server for his Service broker application encounters "There is insufficient system memory in resource pool 'internal' to run this query. Error: 701, Severity: 17, State: 123"

CACHESTORE_BROKERTO was consuming about 18 GB of RAM and SQL server is running out of memory.

Environment:

==================

SQL: Microsoft SQL Server 2012 - 11.0.5556.0 (X64)

OS: Microsoft Corporation Standard Edition (64-bit) on Windows NT 6.2 (Build 9200 )

Max server memory is capped to 26GB out of 32 GB

SQL server is the Target server for a Service broker application.

This is also the Principal server for mirroring configured on the same database.

From the DBCC memory status we see

MEMORYCLERK_SQLBUFFERPOOL (Total) KB

---------------------------------------- -----------

VM Reserved 841608

VM Committed 32768

Locked Pages Allocated 172488

SM Reserved 0

SM Committed 0

Pages Allocated 1722088

CACHESTORE_BROKERTO (node 0) KB

---------------------------------------- -----------

VM Reserved 0

VM Committed 0

Locked Pages Allocated 0

SM Reserved 0

SM Committed 0

Pages Allocated 17685088 ===> Around 17.6 GB

We checked the count of sys.conversation_endpoints

5895928

Five million eight hundred ninety-five thousand nine hundred twenty-eight (almost 6 million)

We also checked the state_desc and found that we have all the conversations in the closed state.

The blog post by Rusanu shows how sys.conversations_endpoints grows out of control with CLOSED conversations that are never cleaned up & how to fix it

http://rusanu.com/2014/03/31/how-to-prevent-conversation-endpoint-leaks/

++Look at the following in our case as well

select lifetime, state_desc, security_timestamp

from [target].sys.conversation_endpoints;

lifetime state_desc security_timestamp

----------------------- ------------------------------------------------------------ -----------------------

2080-10-16 13:25:49.790 CLOSED 1900-01-01 00:00:00.000

2081-06-29 07:58:46.440 CLOSED 1900-01-01 00:00:00.000

2081-09-18 10:08:05.960 CLOSED 1900-01-01 00:00:00.000

2081-08-26 07:26:49.037 CLOSED 1900-01-01 00:00:00.000

2082-04-23 10:22:49.263 CLOSED 1900-01-01 00:00:00.000

2081-11-20 09:13:52.453 CLOSED 1900-01-01 00:00:00.000

2082-12-03 15:23:07.873 CLOSED 1900-01-01 00:00:00.000

2081-11-06 08:26:34.610 CLOSED 1900-01-01 00:00:00.000

2081-08-05 07:52:59.677 CLOSED 1900-01-01 00:00:00.000

2082-05-23 08:25:33.107 CLOSED 1900-01-01 00:00:00.000

The target conversation endpoint is in CLOSED state, but the lifetime field indicates that the

This endpoint will be reclaimed on May 23^rd 2082, because that is the conversation lifetime.

In case you wonder that date comes from adding MAX_INT32 (ie. 2147483647) seconds to the current date.

From an operational point of view this target endpoint is 'leaked'. It will consume DB space and the system will refrain from deleting it for quite some time. Repeat this vicious exchange pattern several thousand times per hour and in a few days your target database will simply run out of space

In our case we do not see the Target database running out of space, however we are seeing memory pressure.

Service Broker Transmission Object Cache is taking up to 18GB of RAM even though the conversations are in the CLOSED state.

Why?

This could be because the Broker uses transmission objects (TO) to record the state of a

Conversation endpoint.

These are in-memory objects about 1KB in size, stored in the DB Engine address space
Once a TO is assigned to a conversation, it remains assigned to the conversation until the conversation ends.
So long as there is enough memory to hold all the TOs for all existing concurrent conversations, the TOs are all in memory and nothing is cached to tempdb.

So if a system has an application that is never ending conversations, eventually there will be more TOs than can be stored in memory and Broker will start caching them in tempdb

The Resolution is as per Rusanu's blog http://rusanu.com/2014/03/31/how-to-prevent-conversation-endpoint-leaks/ . Even though the blog speaks target endpoint 'leak' resulting in the DB space consumption, the same solution can be used in our situation as well.

Note: The blog gives 2 detailed long term solutions for controlling the issue of closed conversations not being cleaned up

If you are already facing a situation in which the conversation endpoints have leaked then you first have to fix the application using one of the proposed solutions above (or both!). Then you can go ahead and cleanup the leaked conversations with the cleanup.sql script presented in Rusanu's blog.

References/ Credits: http://rusanu.com/2014/03/31/how-to-prevent-conversation-endpoint-leaks/

http://blogs.msdn.com/b/sqlserverfaq/archive/2010/03/25/sql-server-encounters-memory-pressure-and-generate-701-error-due-to-incorrect-configuration-of-service-broker.aspx

Service Pack Setup completes successfully on clustered instance but SQL Server Version is not updated on one node

Pooja Kamath — Mon, 22 Dec 2014 03:13:44 GMT

I recently worked on an issue where one of my customers had successfully finished patching their SQL 2005 instances on a 3 node cluster. However the Build number was not getting updated on one of the nodes (NODE3) even after successfully completing the SP4 installation

SQL version On problem node: Database Services (INSTANCEX) SP3 2005.090.4340.00 x86 ENTERPRISE

Expected version: SQL Server Database Service 2005(INSTANCEX) ENU SP4 2005.090.5000.00 x86 ENTERPRISE

OS version: Microsoft Windows Server 2003 family, Enterprise Edition Service Pack 2 (Build 3790) x86

Number of Nodes : 3 (NODE1, NODE2, NODE3)

Note: With SQL Server 2005 unlike the higher versions, the patch is applied to the active node of the cluster and it is the patch that will go off to all passive nodes and will apply the patch files remotely

Patching of the passive node is achieved by remotely creating a scheduled task

With SQL 2005 the patch does everything so once it's finished either all nodes are patched, or none are patched. If there is an issue during the patching phase, then the entire patch needs to be rolled back on all nodes. Therefore if you have 3 node cluster and the last node fails to patch, the patch will be rolled back on all nodes.

However in our case the patch was not rolled back on the nodes even though NODE3 wasn't patched correctly because the summary log still reports a success!
We collected the setup logs on the NODE3 and also the successfully patched nodes to make detailed analysis to check if features to be updated have been skipped for any reason
.We paused the NODE2 and NODE3 from the Cluster Administrator and ran the SP4 setup on the NODE3

We see that the setup succeeded (but completes in less than 30 seconds) showing us it may not be actually patching in the background

If you look at the summary file on the NODE2 AND NODE4 where the build is updated successfully:

Product Installation Status

Product : SQL Server Database Services 2005 (INSTANCEX)

Product Version (Previous): 5000

Product Version (Final) : 5324

Status : Reboot Required

Log File : C:\Program Files\Microsoft SQL Server\90\Setup Bootstrap\LOG\Hotfix\SQL9_Hotfix_KB2716427_sqlrun_sql.msp.log

Error Number : 3010

Error Description :

Comparing this to NODE3 our problem node we do not see the

----------------------------------------------------------------------------------

Product : Database Services (INSTANCEX)

Product Version (Previous): 4340

Product Version (Final) : 5000

Status : Success

Log File :

Error Number : 0

Error Description :

As per http://blogs.msdn.com/b/jorgepc/archive/2009/09/07/sql-server-2005-service-pack-does-not-update-all-the-installed-components.aspx

During the very beginning of the Service Pack setup process, the existing SQL Server services are discovered via WMI queries by the means of Wmiprvse.exeprocess; without the proper permissions, the process will fail to discover the installed components

That led us to suspect "Is the Service pack setup even able to enumerate/discover the SQL instances?

Here is how we investigated that:

We know that the SQL Server Configuration Manager tool also uses the same WMI queries to iterate through the services collection
When these tools generate the WMI queries, the SQL Server Web-Based Enterprise Management (WBEM) provider (Sqlmgmprovider.dll) is loaded into the Wmiprvse.exe process. Then, the SQL Server WBEM provider extracts and processes the information about the services of every instance of SQL Server 2005
So we decided to check if the SQL configuration manager on the problem node was able to enlist the services which would give us an idea if these WMI queries are working as expected

We open the SQL Server Configuration Manager, the right hand pane had the following error "The server threw an exception. [0x80010105]".

On the NODE2 and NODE1 we do not see these issues.

We also performed a small test to check WMI connectivity

Open the WMI Control console: Click Start , click Run , type wmimgmt.msc and then click OK.
In the console tree, right-click WMI Control , and then click Properties.
Select the Security tab.
Expand the Root folder and then navigate to Microsoft > SQLServer
Search for namespace ComputerManagement
Open the WMI Tester: Click Start , click Run , type wbemtest and then click OK.
Connect to the namespace ComputerManagement (root\microsoft\sqlserver\ComputerManagement) and click on Query

SELECT * FROM RegServices
SELECT * FROM SqlService

Make note if the query produced an error or if it was successful.

We see that in the Problem Node3, even after waiting for a long time we do not see any instances enumerated.

When we try the same on the NODE1 and NODE2 for the same namespace we see all the instances are being enumerated:

Connected to other namespace (e.g. root\cimv2, SELECT * FROM Win32_Service), it is all working fine

It appears that only the SQL namespace is having issue.

With this we confirmed that there a WMI connectivity issue on the Server for the SQL Namespace

However we had check the Setup logs to confirm if the Service pack setup has actually succeeded for the Database Engine Services, or if it was skipped due to any possible WMI issues?

We can see that setup able to enumerate all the details but weren't sure why it completes in ten seconds (Below is a snippet of setup log generated on NODE3 when the other two nodes were paused):

12/06/2014 13:10:24.272   Instance Details: INSTANCEX
12/06/2014 13:10:24.272     agentservicename = SQLAgent$INSTANCEX
12/06/2014 13:10:24.272     associatedhotfixbuild = 1520
12/06/2014 13:10:24.272     clustername = S02S702DR-CL6
12/06/2014 13:10:24.272     ftsservicename = MSFTESQL$INSTANCEX
12/06/2014 13:10:24.272     fullversion = 2005.090.4340.00
12/06/2014 13:10:24.272     hiveregpath = Software\Microsoft\Microsoft SQL Server\MSSQL.5
12/06/2014 13:10:24.272     id = MSSQL.5
12/06/2014 13:10:24.272     installsqldatadir = N:\MSSQL.5\MSSQL
12/06/2014 13:10:24.272     installsqldir = C:\Program Files\Microsoft SQL Server\MSSQL.5\MSSQL
12/06/2014 13:10:24.272     lcid = 1033
12/06/2014 13:10:24.272     name = INSTANCEX
12/06/2014 13:10:24.272     productcode = {A2E15F7E-A973-493A-AC36-B1841F8EE431}
12/06/2014 13:10:24.272     qfelevel = 4340
12/06/2014 13:10:24.272     servicename = MSSQL$INSTANCEX
12/06/2014 13:10:24.272     sku = ENTERPRISE
12/06/2014 13:10:24.272     sp = 3
12/06/2014 13:10:24.272     type = SQL Server Standalone Product
12/06/2014 13:10:24.272     vermajbld = 4035
12/06/2014 13:10:24.272     version = 9

"12/06/2014 13:10:44.133 Installing instance: INSTANCEX
12/06/2014 13:10:44.149 Enumerating passive cluster nodes
12/06/2014 13:10:44.149 No available passive nodes were found
12/06/2014 13:10:44.149 Finalizing cluster instance: INSTANCEX
12/06/2014 13:10:44.149 Waiting for all running passive nodes to complete their patch
12/06/2014 13:10:44.149 All running passive nodes have completed their patch
12/06/2014 13:10:44.149 Finalized cluster instance: INSTANCEX

12/06/2014 13:10:44.149 Enumerating passive cluster nodes
12/06/2014 13:10:44.149 No available passive nodes were found
12/06/2014 13:10:44.149 Finalizing cluster instance: INSTANCEX
12/06/2014 13:10:44.149 Waiting for all running passive nodes to complete their patch
12/06/2014 13:10:44.149 All running passive nodes have completed their patch

12/06/2014 13:10:44.149 Hotfix package completed
12/06/2014 13:10:44.149 Attempting to continue the 32 bit ngen queue
12/06/2014 13:10:53.431 Hotfix package closed"

We do not see it installing any files or creating a Log file!

****************************************
Compared to the good run on NODE2 pasted below we can see it creating a log file and installing msp files

++12/06/2014 06:56:02.818 Installing target: NODE2
12/06/2014 06:56:08.927 Installing file: sqlrun_sql.msp
12/06/2014 06:56:08.927 Copy Engine: Creating MSP install log file at: C:\Program Files\Microsoft SQL Server\90\Setup Bootstrap\LOG\Hotfix\SQL9_Hotfix_KB2463332_sqlrun_sql.msp.log

***************************************

So now we had a fair idea that the Service Pack setup is completing without actually patching the instance, because we possibly have issues discovering the installed components by the WMI queries

We now had to fix the SQL Server Configuration Manager, issue which was throwing the error "The server threw an exception. [0x80010105]".

We have a Public KB article that fixes the SQL configuration manager issue we are seeing:

As per: http://support.microsoft.com/kb/956013

To work around this problem, open a command prompt, type the following command, and then press ENTER:
mofcomp "%programfiles(x86)%\Microsoft SQL Server\number\Shared\sqlmgmproviderxpsp2up.mof"

We tried the above and it is reporting a success of parsing the .mof file. However it is stalled at "Storing data in repository" (it normally reports a "Done!" and comes out of the loop) Here it is just stalled..
Trying to restart the WMI service gets hung if we do it after parsing the .mof file.
We involved our Windows Platforms team to help us rebuild the WMI files, but we still received the same error when we try to open the SQL Configuration Manager.

What finally resolved it for us!

We came across http://support.microsoft.com/kb/2849344 , which speaks of the same "The server threw an exception [0x80010105]" error when the configuration manager is started.
However the KB article is applicable to Windows Server 2012 & SQL server 2008 R2 where the problem occurs because SQL Server 2008 R2 internally uses a deprecated 32-bit Windows .dll file (Msclus.dll). When Configuration Manager is started after the Cluster Automation feature is added, the system discovers that the necessary .dll file is missing, and this triggers the error

The work around provided by the KB is as follows

Note The 32-bit version of the MSClus.dll can be found in the "%windir%\syswow64" directory.
Register this .dll file by running the regsvr32 command from an elevated command prompt:
1. Click Start, type command promptin the Search programs and files box.
2. UnderPrograms, right-click Command Prompt, and then clickRun as administrator.
3. At the command prompt, type the following command, and then press ENTER:
  
  Regsvr32 MSClus.dll

The current customer's environment is Windows 2003. However we decided to give the above workaround a shot as we were also in a clustered environment.

After registering the above MSClus.dll ! Voila…the configuration manager no longer throws the exception

We patched the NODE3 once again, which took slightly longer this time.
Rebooted the node and checked the SQL Server build which now reports 090.5000.00 correctly J

SQL Server 2008 Add node setup hangs for hours at “Install SQL_DTS_VSTAConfiguration_install_Cpu64: RollbackCleanup: Removing backup files”

Pooja Kamath — Mon, 17 Mar 2014 22:13:07 GMT

We recently encountered a situation while trying to ‘Add a node’ to the SQL 2008 Cluster

The installation hung for hours at Install SQL_DTS_VSTAConfiguration_install_Cpu64: RollbackCleanup: Removing backup files

It may also hang at Install_VS_Shell_Cpu32_Action: RollbackCleanup: Removing backup files

We do not see any error messages from the SQL setup, however the setup just hangs and does not proceed

• From the System logs we should see the VS template warning and the particular msi which is not letting the SQL setup go through

Log Name:      Application
    Source:        Visual Studio - VsTemplate
    Date:          2/7/2014 7:55:01 PM
    Event ID:      1
    Task Category: None
    Level:         Warning
    Keywords:      Classic
    User:          N/A
    Computer:      ABC.XYZ.LOCAL
    Description:
    Error in Template (C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\ItemTemplates\CSharp\Code\1033\Class.zip), file (Class.vstemplate). Unregistered ProjectType (CSharp). This can be caused by an incorrect ProjectType or because the project type package is not installed.
    Event Xml:
    http://schemas.microsoft.com/win/2004/08/events/event">


        1
        3
        0
        0x80000000000000

        1046
        Application
        ABC.XYZ.LOCAL



        Error in Template (C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\ItemTemplates\CSharp\Code\1033\Class.zip), file (Class.vstemplate). Unregistered ProjectType (CSharp). This can be caused by an incorrect ProjectType or because the project type package is not installed.

Log Name:      Application
    Source:        MsiInstaller
    Date:          2/7/2014 7:52:49 PM
    Event ID:      1040
    Task Category: None
    Level:         Information
    Keywords:      Classic
    User:          TESTUSER\XYZSQLSVC
    Computer:      ABC.XYZ.LOCAL
    Description:
    Beginning a Windows Installer transaction: C:\SQL2008\x64\setup\vs_shell.msi. Client Process Id: 7512.
    Event Xml:
    http://schemas.microsoft.com/win/2004/08/events/event">


        1040
        4
        0
        0x80000000000000

        1043
        Application
        ABC.XYZ.LOCAL



        C:\SQL2008\x64\setup\vs_shell.msi
        7512
        (NULL)
        (NULL)
        (NULL)





< /Event>

If you look at the detail.txt you may find it is stalled at the below point

SSIS: Running: C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\vsta.exe /hostid SSIS_ScriptTask /setup

It does not indicate a failed execution of the script. Infact running this from the Command prompt manually would work, but it still didn’t let us proceed with the setup

Here’s how the Detail.txt would look like

14-02-08 04:39:57 Slp: Sco: Attempting to open registry subkey SOFTWARE\Microsoft\Microsoft SQL Server\100\SSIS
2014-02-08 04:39:57 Slp: Sco: Attempting to set value LogPackageExecutionToEventLog
2014-02-08 04:39:57 Slp: SetValue: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\100\SSIS, Name = LogPackageExecutionToEventLog
2014-02-08 04:39:57 Slp: Sco: Attempting to create base registry key HKEY_LOCAL_MACHINE, machine
2014-02-08 04:39:57 Slp: Sco: Attempting to open registry subkey
2014-02-08 04:39:57 Slp: Sco: Attempting to open registry subkey SOFTWARE\Microsoft\Microsoft SQL Server\100\SSIS
2014-02-08 04:39:57 Slp: Sco: Attempting to set value LogPackageExecutionToEventLog
2014-02-08 04:39:57 Slp: SetValue: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\100\SSIS, Name = LogPackageExecutionToEventLog
2014-02-08 04:39:57 SSIS: SetRegistryValues() done
2014-02-08 04:39:57 SSIS: STATUS: Install_ConfigNonRC completed
2014-02-08 04:39:57 SSIS: SSIS_Config: Install Timing: ConfigNonRC completed
2014-02-08 04:39:57 Slp: SlpConfigAction.ExecuteAction(actionId: SSISConfigAction_install_postmsi_Cpu64) completed
2014-02-08 04:39:57 Slp: Checkpoint: INSTALL:CONFIGNONRC:SQL_DTS_FULL
2014-02-08 04:39:57 Slp: Completed Action: SSISConfigAction_install_postmsi_Cpu64, returned True
2014-02-08 04:39:57 Slp: ---- Flush datastore cache before calling configuration action 'SQL_DTS_VSTAConfiguration_install_Cpu64'
2014-02-08 04:39:57 Slp: ---- Flush datastore cache before calling configuration action 'SQL_DTS_VSTAConfiguration_install_Cpu64'
2014-02-08 04:39:57 Slp: ----------------------------------------------------------------------
2014-02-08 04:39:57 Slp: Running Action: SQL_DTS_VSTAConfiguration_install_Cpu64
2014-02-08 04:39:57 Slp: Action Data:
2014-02-08 04:39:57 Slp:   Feature = SQL_DTS_Full_sql_is_Cpu64
2014-02-08 04:39:57 Slp:   Scenario = install
2014-02-08 04:39:57 Slp:   Timing = ConfigNonRC
2014-02-08 04:39:57 Slp:   ConfigObjectType = Microsoft.SqlServer.Configuration.SSIS.VSTAConfigurationPrivate
2014-02-08 04:39:57 Slp:   FeatureName = SQL_DTS_Full
2014-02-08 04:39:57 Slp:   FeatureCpuType = Cpu64
2014-02-08 04:39:57 Slp:   FeaturePackageId = sql_is
2014-02-08 04:39:58 Slp: Sco: Attempting to create base registry key HKEY_LOCAL_MACHINE, machine
2014-02-08 04:39:58 Slp: Sco: Attempting to open registry subkey
2014-02-08 04:39:58 Slp: Sco: Attempting to open registry subkey SOFTWARE\Microsoft\NET Framework Setup\NDP\v3.5
2014-02-08 04:39:58 Slp: Sco: Attempting to get registry value InstallPath
2014-02-08 04:39:58 SSIS: Running: c:\Windows\Microsoft.NET\Framework\v3.5\AddInUtil.exe -PipelineRoot:"C:\Program Files (x86)\Common Files\microsoft shared\VSTA\Pipeline" -Rebuild
2014-02-08 04:39:58 SSIS: Application succeded: c:\Windows\Microsoft.NET\Framework\v3.5\AddInUtil.exe -PipelineRoot:"C:\Program Files (x86)\Common Files\microsoft shared\VSTA\Pipeline" -Rebuild
2014-02-08 04:39:58 Slp: Sco: Attempting to create base registry key HKEY_LOCAL_MACHINE, machine
2014-02-08 04:39:58 Slp: Sco: Attempting to open registry subkey
2014-02-08 04:39:58 Slp: Sco: Attempting to open registry subkey SOFTWARE\Microsoft\VSTA\9.0
2014-02-08 04:39:58 Slp: Sco: Attempting to get registry value InstallDir
2014-02-08 04:39:58 SSIS: Running: C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\vsta.exe /hostid SSIS_ScriptTask /setup

We found that the the following Visual studio components were previously installed on the server before installing SQL Server

Visual Studio Tools for Applications 2.0
Visual Studio 2008 Shell (integrated mode)

The above issue happens to be due to the incompatibility of SQL Server with either a Visual Studio or .NET framework component

The incompatibility is explained in detail the blog :http://blogs.msdn.com/b/psssql/archive/2008/08/11/sql-server-2008-visual-studio-2008-sp1-and-net-framework-3-5-sp1-explained.aspx

We see these Stalls in the setup when it is trying to install the below components only

Business Intelligence Development Studio
Integration Services
Management Tools (Either Basic or Full)

Since this was a new server which was going to be a dedicated SQL server we Uninstalled the following components from the control panel

Visual Studio Tools for Applications 2.0
Visual Studio 2008 Shell (integrated mode)
SQL server Compact Edition

Repaired the .NET Framework 3.5 SP1 and then re-run the SQL setup

In most cases the issue is very clear with the Setup giving us a failed rule
Rule "Previous releases of Microsoft Visual Studio 2008" failed

But in our case, the setup only hung with No error messages or failed rule except for the the warning for VS template in the System log .

As summarized in the blog : http://blogs.msdn.com/b/psssql/archive/2008/08/11/sql-server-2008-visual-studio-2008-sp1-and-net-framework-3-5-sp1-explained.aspx we could follow the below steps based on environment (However If this dedicated Database server the easiest option would be uninstall the VS components before installing SQL server and reinstall them after the SQL setup is complete)

1) If you don't have Visual Studio 2008 installed and have never installed a SQL Server 2008 CTP build, SQL Server 2008 RTM comes with everything you need and you should not encounter this problem.

2) If have Visual Studio 2008 installed, you must install VS 2008 SP1 before you install SQL Server 2008 RTM (or don't pick features that require it -BIDS, Management Studio, SSIS)

3) If you do not have VS 2008 installed but are encountering this problem, do not attempt to install VS 2008 SP1. Rather, uninstall Visual Studio 2008 Shell (integrated mode) and Visual Studio Tools for Applications 2.0. You should then be allowed to proceed with the installation.

4) SQL Server 2008 RTM comes with the .Net Framework 3.5 SP1 final release. This requirement is independent of the binaries we need from VS 2008 SP1. There is no need for you to run any update to the .Net Framework after installing SQL Server 2008.

5) If you install Visual Studio 2008 RTM after installing SQL Server 2008 RTM, you must install VS 2008 SP1 to be supported if you are using the features in SQL Server that depend on VS 2008 SP1 (BIDS, Integration Services, or Management Tools

http://blogs.msdn.com/b/psssql/archive/2008/08/11/sql-server-2008-visual-studio-2008-sp1-and-net-framework-3-5-sp1-explained.aspx