Recently, when I was working with one of our customer, I came across an interesting issue.

Customer have two databases, in the same instance. Let’s say the databases are X and Y.

He have a table T1 in X and table T2 in Y.

Both the tables are exactly same in schema definition, number & type of indexes, amount of data and in almost all the specs.

Only difference is, one of those tables has 2 partitions.

Consider table T2 in database Y has the partitions.

They’re running a simple query like SELECT TOP (5000) * from T1 ORDER BY C1 (on database X – Where we don’t have partitions), the query is giving results almost instantaneously.

Same query when we run in a partitioned table, is taking very long time to finish.

I tried to reproduce the same scenario in my lab machine and was successful.

Please find the below exercise I performed to reproduce the issue.

1. Create a partition function for our testing

CREATE PARTITION FUNCTION PFT (integer)

AS RANGE RIGHT

FOR VALUES

(

125000, 250000, 375000, 500000,

625000, 750000, 875000, 1000000,

1125000, 1250000, 1375000, 1500000,

1625000, 1750000, 1875000, 2000000,

2125000, 2250000, 2375000, 2500000,

2625000, 2750000, 2875000, 3000000,

3125000, 3250000, 3375000, 3500000,

3625000, 3750000, 3875000, 4000000,

4125000, 4250000, 4375000, 4500000,

4625000, 4750000, 4875000, 5000000

);

GO

2. Create a partition scheme on that function

CREATE PARTITION SCHEME PST

AS PARTITION PFT

ALL TO ([PRIMARY]);

3. Create a table on the partition scheme (table with partitions)

CREATE TABLE dbo.T1

(

TID integer NOT NULL IDENTITY(0,1),

Column1 integer NOT NULL,

Padding binary(100) NOT NULL DEFAULT 0x,

CONSTRAINT PK_T1

PRIMARY KEY CLUSTERED (TID)

ON PST (TID)

);

4. Create a table without any partitions (Normal table)

CREATE TABLE dbo.T2

(

TID integer NOT NULL IDENTITY(0,1),

Column1 integer NOT NULL,

Padding binary(100) NOT NULL DEFAULT 0x,

CONSTRAINT PK_T2

PRIMARY KEY CLUSTERED (TID)

ON [PRIMARY]

);

5. Create an auxiliary table “Numbers” which stages temporary data that helps us in populating the tables T1 and T2

CREATE TABLE dbo.Numbers (n bigint PRIMARY KEY);

WITH

L0 AS(SELECT 1 AS c UNION ALL SELECT 1),

L1 AS(SELECT 1 AS c FROM L0 AS A CROSS JOIN L0 AS B),

L2 AS(SELECT 1 AS c FROM L1 AS A CROSS JOIN L1 AS B),

L3 AS(SELECT 1 AS c FROM L2 AS A CROSS JOIN L2 AS B),

L4 AS(SELECT 1 AS c FROM L3 AS A CROSS JOIN L3 AS B),

L5 AS(SELECT 1 AS c FROM L4 AS A CROSS JOIN L4 AS B),

Nums AS(SELECT ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS n FROM L5)

INSERT dbo.Numbers WITH (TABLOCKX)

SELECT TOP (10000000) n

FROM Nums

ORDER BY n

OPTION (MAXDOP 1);

6. Insert data into tables T1 and T2

T1

========

INSERT dbo.T1 WITH (TABLOCKX)

(Column1)

SELECT

(ABS(CHECKSUM(NEWID())) % 5) + 1

FROM dbo.Numbers AS N

WHERE n BETWEEN 1 AND 5000000;

T2

========

INSERT dbo.T2 WITH (TABLOCKX)

(Column1)

SELECT

(ABS(CHECKSUM(NEWID())) % 5) + 1

FROM dbo.Numbers AS N

WHERE n BETWEEN 1 AND 5000000;

7. Now if we see the partition information, for T1, using the below query

SELECT

t.name as TableName,i.name as IndexName,

p.partition_id as partitionID,

p.partition_number,rows, fg.name

FROM sys.tables AS t

INNER JOIN sys.indexes AS i ON (t.object_id = i.object_id)

INNER JOIN sys.partitions AS p ON (t.object_id = p.object_id and i.index_id = p.index_id)

INNER JOIN sys.destination_data_spaces dds ON (p.partition_number = dds.destination_id)

INNER JOIN sys.filegroups AS fg ON (dds.data_space_id = fg.data_space_id)

WHERE (t.name = 'T1') and (i.index_id IN (0,1))

Output

Tab Index partitionID part rows name

Name Name No.

T1 PK_T1 72057594039042048 1 125000 PRIMARY

T1 PK_T1 72057594039107584 2 125000 PRIMARY

T1 PK_T1 72057594039173120 3 125000 PRIMARY

T1 PK_T1 72057594039238656 4 125000 PRIMARY

T1 PK_T1 72057594039304192 5 125000 PRIMARY

T1 PK_T1 72057594039369728 6 125000 PRIMARY

T1 PK_T1 72057594039435264 7 125000 PRIMARY

T1 PK_T1 72057594039500800 8 125000 PRIMARY

T1 PK_T1 72057594039566336 9 125000 PRIMARY

T1 PK_T1 72057594039631872 10 125000 PRIMARY

T1 PK_T1 72057594039697408 11 125000 PRIMARY

T1 PK_T1 72057594039762944 12 125000 PRIMARY

T1 PK_T1 72057594039828480 13 125000 PRIMARY

T1 PK_T1 72057594039894016 14 125000 PRIMARY

T1 PK_T1 72057594039959552 15 125000 PRIMARY

T1 PK_T1 72057594040025088 16 125000 PRIMARY

T1 PK_T1 72057594040090624 17 125000 PRIMARY

T1 PK_T1 72057594040156160 18 125000 PRIMARY

T1 PK_T1 72057594040221696 19 125000 PRIMARY

T1 PK_T1 72057594040287232 20 125000 PRIMARY

T1 PK_T1 72057594040352768 21 125000 PRIMARY

T1 PK_T1 72057594040418304 22 125000 PRIMARY

T1 PK_T1 72057594040483840 23 125000 PRIMARY

T1 PK_T1 72057594040549376 24 125000 PRIMARY

T1 PK_T1 72057594040614912 25 125000 PRIMARY

T1 PK_T1 72057594040680448 26 125000 PRIMARY

T1 PK_T1 72057594040745984 27 125000 PRIMARY

T1 PK_T1 72057594040811520 28 125000 PRIMARY

T1 PK_T1 72057594040877056 29 125000 PRIMARY

T1 PK_T1 72057594040942592 30 125000 PRIMARY

T1 PK_T1 72057594041008128 31 125000 PRIMARY

T1 PK_T1 72057594041073664 32 125000 PRIMARY

T1 PK_T1 72057594041139200 33 125000 PRIMARY

T1 PK_T1 72057594041204736 34 125000 PRIMARY

T1 PK_T1 72057594041270272 35 125000 PRIMARY

T1 PK_T1 72057594041335808 36 125000 PRIMARY

T1 PK_T1 72057594041401344 37 125000 PRIMARY

T1 PK_T1 72057594041466880 38 125000 PRIMARY

T1 PK_T1 72057594041532416 39 125000 PRIMARY

T1 PK_T1 72057594041597952 40 125000 PRIMARY

T1 PK_T1 72057594041663488 41 0 PRIMARY

8. Now if we see the partition information, for T1, using the below query

SELECT

t.name as TableName,i.name as IndexName,

p.partition_id as partitionID,

p.partition_number,rows, fg.name

FROM sys.tables AS t

INNER JOIN sys.indexes AS i ON (t.object_id = i.object_id)

INNER JOIN sys.partitions AS p ON (t.object_id = p.object_id and i.index_id = p.index_id)

INNER JOIN sys.destination_data_spaces dds ON (p.partition_number = dds.destination_id)

INNER JOIN sys.filegroups AS fg ON (dds.data_space_id = fg.data_space_id)

WHERE (t.name = 'T2') and (i.index_id IN (0,1))

Output:

Tab IndexName partitionID partition_number rows name

Name

T2 PK_T2 72057594041794560 1 5000000 PRIMARY

So we confirmed that T1 is partitioned and T2 is not (By default SQL creates one partition for every table that’s why you’re seeing one row for T2) and both have 5,000,000 rows

9. Now let’s see the indexes on these tables : They have one clustered index each, which is their PRIMARY KEY

USE [partitions_perf_test]

GO

/****** Object: Index [PK_T1] Script Date: 21-04-2014 18:10:10 ******/

ALTER TABLE [dbo].[T1] ADD CONSTRAINT [PK_T1] PRIMARY KEY CLUSTERED

(

[TID] ASC

)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)

GO

USE [partitions_perf_test]

GO

/****** Object: Index [PK_T2] Script Date: 21-04-2014 18:10:15 ******/

ALTER TABLE [dbo].[T2] ADD CONSTRAINT [PK_T2] PRIMARY KEY CLUSTERED

(

[TID] ASC

)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]

GO

10. The query we’re going to fire is

select top(100000) * from T1 order by Column1

and

select top(100000) * from T2 order by Column1

11. Customer had another non clustered index on a column which is part of the order by clause i.e., Column1. So, I’m going to crate it now

create index nc1 on T1(Column1)

create index nc1 on T2(Column1)

Now we’re done with all the setup/configuration.

Let’s test the query and examine the explain plans.

Let’s start with T2 (non-partitioned table) first

It returned in 1 second.

The reason behind choosing an NC scan followed by a clustered index key look up can be explained as:

We have an order by clause on the column “column1” in the select statement. Optimizer is using the NC-Index to scan as the keys in the NC index are already arranged in the ascending order so we need to do few lookups, starting from left node, moving to root and then right (In-order traversal), on the B-Tree index to get the top 100,000 rows. This avoid the costly SORT operation to arrange the results in an order.

Let’s execute the same query on the partitioned table and see how I behaves,

Here, it’s not at all using the NC-Index. It’s just going for a clustered index scan followed by sort. And also it’s taking around 2 minutes to complete, in my machine. In customer’s environment it’s even more.

What might be the reason behind this behaviour?

I just observed the NC-Index’s properties on both the table – There’s an apparent change in the storage section.

T2: (Non-Paritioned)

T1: (Paritioned)

We created the NC-Index in the same way, using the same definition, on both the tables. But why the index on a Non-Partitioned table is created on the FILEGROUP whereas index on partitioned table is created on the partition scheme?

When a table is partitioned, the subsequent index we create, tries to align themselves as per the table’s structure. So the index will also be partitioned implicitly. i.e., a new B-Tree for every partition, with a new root and new structure, will be created. This is to improve the performance by making parallel localized searches when a partition needs to be traversed.

But, the question is, why my query performance is deteriorated when the index is partitioned?

The reason is: when the index is partitioned, SQL needs to go through each local index (index partition) and scan through it. So optimizer feel that that’s a heavy weighted operation and avoids using that index. This is exactly what’s happening for our NC-Index and it’s not being used.

When I just change the index definition on T1 by switching the storage to FILEGROUP from partition scheme, it also uses the NC index and returns the results fast.

CREATE NONCLUSTERED INDEX [nc1] ON [dbo].[T1]

(

[Column1] ASC

)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = ON, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)

--ON PST(Column1)

ON [PRIMARY]

GO

Removing “ON PST(Column1)” and adding ON [PRIMARY] makes the index created on the default filegroup i.e. PRIMARY.

Let’s test the query now:

This is exactly what the execution plan expected to be…

As an alternative to achieve this behaviour without changing the index structure is to use the query hint WITH (INDEX()). The query below also be executed using the same execution plan where it picks up NC-Index.

select top(100000) * from T2 WITH (INDEX(nc1)) order by Column1

As always, please feel free to get in touch with me with your comments or feedback.

Thanks,

Chandra

Recently, I was working on one of the customers on an issue. The resolution is simple but I don’t see much resources online on this.

So, wanted to write a blog on this quickly.

He’s using SQL 2005 (I didn’t say interesting because it is 2005 ;)) and installed SQL on a node (say Node-A) and now trying to do the obvious next step: yes, you guessed it right (‘AddNode’ on the other node).

While doing that he encountered with an error message like

Install : Failed

Log File : C:\Program Files\Microsoft SQL Server\90\Setup Bootstrap\LOG\Files\SQLSetup0020_XXXX_WI.log

Last Action : SetClusterDialogs

Error String : The setup has encountered an unexpected error while Preparing Installation Wizard. The error is: Fatal error during installation.

Error Number : 29528

--------------------------------------------------------------------------------

SQL Server Setup failed. For more information, review the Setup log file in %ProgramFiles%\Microsoft SQL Server\90\Setup Bootstrap\LOG\Summary.txt.

So, I quickly looked in the SQLSetup0020_XXXX_WI.log file, as pointed by the error message above – and it says

MSI (c) (90:88) [11:56:10:530]: Note: 1: 1729

MSI (c) (90:88) [11:56:10:530]: Transforming table Error.

MSI (c) (90:88) [11:56:10:530]: Note: 1: 2262 2: Error 3: -2147287038

MSI (c) (90:88) [11:56:10:536]: Transforming table Error.

MSI (c) (90:88) [11:56:10:537]: Transforming table Error.

MSI (c) (90:88) [11:56:10:537]: Note: 1: 2262 2: Error 3: -2147287038

MSI (c) (90:88) [11:56:10:537]: Transforming table Error.

MSI (c) (90:88) [11:56:10:545]: Note: 1: 2262 2: Error 3: -2147287038

MSI (c) (90:88) [11:56:10:545]: Product: Microsoft SQL Server 2005 (64-bit) -- Configuration failed.

MSI (c) (90:88) [11:56:10:545]: Windows Installer reconfigured the product. Product Name: Microsoft SQL Server 2005 (64-bit). Product Version: 9.4.5000.00. Product Language: 1033. Reconfiguration success or error status: 1603.

MSI (c) (90:88) [11:56:10:550]: Grabbed execution mutex.

MSI (c) (90:88) [11:56:10:550]: Cleaning up uninstalled install packages, if any exist

MSI (c) (90:88) [11:56:10:551]: MainEngineThread is returning 1603

I remember dealing with similar issues earlier, configuration fails by spitting out the “transforming table errors” when we have more than an IP resource configured in the Windows cluster.

And I see from the same log that they have 2 IP resources in the cluster

<Func Name='ValidateIpAddress'>

<Func Name='ValidateIpAddressX'>

Server1 XXX.XXX.XXX Clients 1

<EndFunc Name='ValidateIpAddressX' Return='0' GetLastError='0'>

<EndFunc Name='ValidateIpAddress' Return='0' GetLastError='0'>

<Func Name='IPAddressHelperAddToListView'>

MSI (c) (90!A0) [11:54:48:655]: Note: 1: 2262 2: ListBox 3: -2147287038

<EndFunc Name='IPAddressHelperAddToListView' Return='0' GetLastError='0'>

<Func Name='IPAddressDlgHelperRemoveFromCombo'>

<EndFunc Name='IPAddressDlgHelperRemoveFromCombo' Return='0' GetLastError='0'>

<Func Name='ValidateIpAddress'>

<Func Name='ValidateIpAddressX'>

Server1 YYY.YYY.YYY.YYY Clients 1

<EndFunc Name='ValidateIpAddressX' Return='0' GetLastError='0'>

<EndFunc Name='ValidateIpAddress' Return='0' GetLastError='0'>

<Func Name='IPAddressHelperAddToListView'>

I masked the IPs as XXX.XXX.XXX.XXX and YYY.YYY.YYY.YYY – you actually will see two different IPV4 addresses there.

As soon as I confirmed this, I asked him to follow the sequence of steps below

Remove an IP address -> do the “addNode” -> add the IP back

He did that and you know the outcome – it worked :)

As always, please feel free to get in touch with me with your comments or feedback.

Regards,

Chandra

Let’s do a simple exercise to understand how we do the encryption and decryption and we’ll try to replicate the scenario I was talking about.

First we’ll create 2 logins to play with and a database followed by table with few columns to perform cell level encryption.

Here we have two managers and they’re storing their employee details like id, name, SSN and rating details in a table called EmpTable. We’re encrypting the SSN and the rating details as they’re confidential and should only be visible to their respective managers.

Create login Man1 with password = 'Man1pwd'

Go

Create login Man2 with password = 'Man2pwd'

Go

create database CompanyDB

go

use CompanyDB

go

create user Man1;

go

create user Man2

go

create table EmpTable (Id int, name nvarchar(30), Manname varchar(20), SSN varbinary(60), Rating varbinary(4000))

go

--grant access to the table to both doctors

grant select,insert on EmpTable to Man1

grant select,insert on EmpTable to Man2

Then we’ll create a Database master key, two certificates for both the users and two symmetric keys using their respective keys

CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'CompanyDB1'

create certificate Man1Cert authorization Man1 with subject = 'Man1cert', START_DATE = '10/09/2013'

go

create certificate Man2Cert authorization Man2 with subject = 'Man2cert', START_DATE = '10/09/2013'

go

create symmetric key Man1Key authorization Man1 with algorithm = TRIPLE_DES ENCRYPTION BY certificate Man1Cert

go

create symmetric key Man2Key authorization Man2 with algorithm = TRIPLE_DES ENCRYPTION BY certificate Man2Cert

go

Now we’ll do the encryption

execute as login = 'Man1'

open symmetric key Man1Key decryption by certificate Man1cert

insert into EmpTable values (1, 'Chris','Man1', encryptByKey(Key_GUID('Man1Key'),'111-11-1111'), encryptByKey(Key_GUID('Man1Key'),'Exceeded Expectations'))

insert into EmpTable values (2, 'Charles','Man1', encryptByKey(Key_GUID('Man1Key'),'222-22-2222'), encryptByKey(Key_GUID('Man1Key'),'Met Expectations’))

insert into EmpTable values (3, 'James','Man1', encryptByKey(Key_GUID('Man1Key'),'333-33-3333'), encryptByKey(Key_GUID('Man1Key'),'Partially Met Expectations'))

close all symmetric keys

Revert

execute as login = 'Man2'

open symmetric key Man2Key decryption by certificate Man2cert

insert into EmpTable values (991, 'Dave','Man2',encryptByKey(Key_GUID('Man2Key'),'444-44-4444'), encryptByKey(Key_GUID('Man2Key'), 'Partially Met Expectations'))

insert into EmpTable values (1001, 'Dan','Man2',encryptByKey(Key_GUID('Man2Key'),'555-55-5555'), encryptByKey(Key_GUID('Man2Key'), 'Exceeded Expectations'))

insert into EmpTable values (1011, 'Steve','Man2', encryptByKey(Key_GUID('Man2Key'),'666-66-6666'), encryptByKey(Key_GUID('Man2Key'), 'Exceeded Expectations'))

close all symmetric keys

revert

If we try to select the data from these two columns which contain encrypted data, it looks like

select Id, name, Manname, convert(varchar,decryptbykey(SSN)) as SSN, convert (varchar,decryptbykey(Rating)) as Rating from EmpTable

Id     name   Manname SSN    Rating

1      Chris  Man1   NULL   NULL

2      CharlesMan1   NULL   NULL

3      James  Man1   NULL   NULL

4      Dave   Man2   NULL   NULL

5      Dan    Man2   NULL   NULL

6      Steve  Man2   NULL   NULL

Now we’ll try to decrypt the data while querying it.

execute as login = 'Man1'

open symmetric key Man1Key decryption by certificate Man1Cert

select Id, name, Manname, convert(varchar,decryptbykey(SSN)) as SSN, convert (varchar,decryptbykey(Problem)) as Rating from EmpTable

Id     name   Manname        SSN           Rating

1      Chris   Man1          111-11-1111   Exceeded Expectations

2      Charles Man1          222-22-2222   Met Expectations

3      James   Man1          333-33-3333   Partially Met Expectations

4      Dave    Man2          NULL          NULL

5      Dan     Man2          NULL          NULL

6      Steve   Man2          NULL          NULL

If we observe the user1 is seeing NULLs for the data rows that were encrypted by user2

Now we’re not closing the key opened and trying to see the data as user2

revert

execute as login = 'Man2'

select Id, name, Manname, convert(varchar,decryptbykey(SSN)) as SSN, convert (varchar,decryptbykey(Problem)) as Rating from EmpTable

Id     name   Manname       SSN           Rating

1      Chris  Man1          111-11-1111   Exceeded Expectations

2      CharlesMan1          222-22-2222   Met Expectations

3      James  Man1          333-33-3333   Partially Met Expectations

4      Dave   Man2          NULL          NULL

5      Dan    Man2          NULL          NULL

6      Steve  Man2          NULL          NULL

So, user2 is able to see the data that was encrypted by user1.

This is because, the keys that is required to decrypt this data is still open. And even though the login Doc 2 does not have the permission can still see the data that was encrypted using the Man1’s keys. This is only possible on the same session with impersonation.

 The query processor don’t validate if the owner of the key or the user who encrypted the data is doing the decryption or not. i.e. if a key, protecting the data/used to encrypt the data is open, whoever issues a select query against the encrypted columns will get the data in decrypted form, in the same session.

The point here is that when the keys are opened in a session and if there is a login or user where the caller has IMPERSONATE permissions on the another login/User, and if he changes the execution context to the new login, which really does not have permissions on the keys used for encryption, will still be able to see the data if the keys were not closed before changing the context. As in the above example.

The better way of changing the execution context in such cases where you do not want the data to be read would be to close the keys and then change the execution context as shown below:

execute as login = 'Man1'

open symmetric key Man1Key decryption by certificate Man1Cert

select Id, name, Manname, convert(varchar,decryptbykey(SSN)) as SSN, convert (varchar,decryptbykey(Problem)) as Rating from EmpTable

Id     name   Manname       SSN           Rating

1      Chris  Man1          111-11-1111   Exceeded Expectations

2      CharlesMan1          222-22-2222   Met Expectations

3      James  Man1          333-33-3333   Partially Met Expectations

4      Dave   Man2          NULL          NULL

5      Dan    Man2          NULL          NULL

6      Steve  Man2          NULL          NULL

close all symmetric keys

Revert

execute as login = 'Man2'

select Id, name, Manname, convert(varchar,decryptbykey(SSN)) as SSN, convert (varchar,decryptbykey(Problem)) as Rating from EmpTable

Id          name        Manname    SSN      Rating

----------- ----------- ---------- -------- -----------

1           Chris       Man1       NULL     NULL

2           Charles     Man1       NULL     NULL

3           James       Man1       NULL     NULL

991         Dave        Man2       NULL     NULL

1001        Dan         Man2       NULL     NULL

1011        Steve       Man2       NULL     NULL

(6 row(s) affected)

So it’s always suggestible to close your keys after reading the data, especially in the environments where same session’s state is shared by multiple client connections/users.

p.s. This feature is also useful in cases where a user want to share some/all of his data with the other users. For e.g. Manager1 want to share some/all of his employee’s details with other Managers. In this case instead of sharing his keys to those guys, he can just open his key and can allow others to read data from his session. He can close his key after the other guys completed querying the data.

As always, please feel free to get in touch with me with your comments or feedback.

Thanks,

Chandra

Fortunately we have an option in BPA to export the recommendation data into an XML file. But the problem here is it is very difficult for people to go through that XML (it’s very bulky) and understand that to make a good sense out of the recommendations. Some of the customers of mine expressed an idea of materializing this XML data which is very difficult to understand into relational data (represents through tables) which is very easy to understand.

The other benefits of doing that are

· It enhances the ease of understanding

· You can run reports and determine how many alerts are there, how many errors are there, how many warnings are there etc…

· You can build your own custom applications on top of that data, prepare beautiful charts, reports and show them to your senior management

· We can prioritize our action plans like first we want to see how many warnings/errors/informational messages are there and based upon that we want to decide which to address first etc…

And the list goes on based upon your creativity…the whole point here is bringing that data into much usable format

A sample snippet from the XML file generated by the BPA looks like this

<Objs Version="1.1.0.1" xmlns="https://schemas.microsoft.com/powershell/2004/04">

<Obj RefId="0">

<TN RefId="0">

<T>Microsoft.BaselineConfigurationAnalyzer.CoreInterface.Model</T>

<T>System.Object</T>

</TN>

<ToString>Microsoft.BaselineConfigurationAnalyzer.CoreInterface.Model</ToString>

<Props>

<S N="Id">SQL2008R2BPA</S>

<S N="Company">Microsoft</S>

<S N="Name">SQL Server 2008 R2 BPA</S>

<Version N="Version">1.0</Version>

<S N="LastScanTime">10/03/2013 13:29:35</S>

<Obj N="SubModels" RefId="1">

<TN RefId="1">

<T>System.Collections.Generic.List`1[[System.String, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]</T>

<T>System.Object</T>

</TN>

<LST>

<S>AnalysisServices</S>

<S>Engine</S>

<S>IntegrationServices</S>

<S>Replication</S>

<S>ReportingServices</S>

<S>Setup</S>

</LST>

.

.

.

.

<Obj RefId="19">

<TN RefId="5">

<T>System.Object[]</T>

<T>System.Array</T>

<T>System.Object</T>

</TN>

<LST>

<Obj RefId="20">

<TN RefId="6">

<T>Microsoft.BaselineConfigurationAnalyzer.CoreInterface.Result</T>

<T>System.Object</T>

</TN>

<ToString>Microsoft.BaselineConfigurationAnalyzer.CoreInterface.Result</ToString>

<Props>

<S N="ResultNumber">1</S>

<S N="ResultId">4196529084</S>

<S N="ModelId">SQL2008R2BPA</S>

<S N="SubModelId">Engine</S>

<S N="RuleId">0</S>

<S N="ComputerName">localhost</S>

<S N="Context">SQL2008R2BPA</S>

<S N="Source">localhost</S>

<S N="Severity">Warning</S>

<S N="NeutralSeverity">Warning</S>

<S N="Category">Security</S>

<S N="Title">Engine - Authentication Mode</S>

<S N="Problem">The authentication mode is not set to recommended value</S>

<S N="Impact">Windows Authentication is the default authentication mode, and is much more secure than SQL Server Authentication. Windows Authentication uses Kerberos security protocol, provides password policy enforcement with regard to complexity validation for strong passwords, provides support for account lockout, and supports password expiration</S>

<S N="Resolution">Configure Authentication mode to Windows mode when possible</S>

<Nil N="Compliance" />

<S N="Help">https://go.microsoft.com/fwlink/?LinkId=184392</S>

<S N="Excluded">False</S>

</Props>

</Obj>

<Obj RefId="21">

<TNRef RefId="6" />

<ToString>Microsoft.BaselineConfigurationAnalyzer.CoreInterface.Result</ToString>

Please find the exercise I performed to push this valuable XML data into a table in the SQL server database

1). Modified the first line i.e. the Objs element – Changed it to something like this (<Objs xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="https://www.w3.org/2001/XMLSchema">)

2). Deleted the unnecessary content from the XML file (Till Ref ID 19 from the top)

3). Replaced "Nil" with "S". The resultant XML schema looks like this

<Objs>

<Obj>

<Props>

<S> </S>

</Props>

</Obj>

</Objs>

- Only one Objs element

- Multiple Obj elements inside Objs element

- Only one Props element Inside Obj element

- Multiple S elements inside Props

4). Create the table to put the XML data

CREATE TABLE EXPRTRESTRCT19TBL (XMLDATA XML, LOADDATETIME date)

5). Prepare the insert statement by replacing the content between quotes with your new XML content.

To see the complete insert statement please click on the Insert_Code.SQL

After insert the table TEST2_CMCK_CSSPRDDB03_EXPRTRESTRCT12 contains the formatted XML data

6). Create a table temp_values using the statement below

"create table temp_values (UID integer identity,ID Integer,value NVARCHAR(MAX))"

7). Run the stored proc to populate temp_values table

declare @idoc int,

@xmldata xml

Select @xmlData = XMLDATA from dbo.EXPRTRESTRCT19TBL --MAKE SURE YOU CHANGE THE TABLE NAME TO THE ONE YOU CREATED IN STEP-4

EXEC sp_xml_preparedocument @idoc OUTPUT, @xmldata

insert into temp_values (id,value)

Select * from Openxml(@idoc,'Objs/Obj/Props/S',1)

With (

ID NVARCHAR(MAX) '../S',

Value NVARCHAR(MAX) 'text()'

)

EXEC sp_xml_removedocument @idoc

8). Create a final result table which stores all the formatted output i.e. the final output

drop table final_result_exprtresrct19

create table final_result_exprtresrct19 (ResultNumber NVARCHAR(MAX),ResultId NVARCHAR(MAX),ModelId NVARCHAR(MAX),SubModelId NVARCHAR(MAX),RuleId NVARCHAR(MAX),ComputerName NVARCHAR(MAX),Context NVARCHAR(MAX),

Source NVARCHAR(MAX),Severity NVARCHAR(MAX),NeutralSeverity NVARCHAR(MAX),Category NVARCHAR(MAX),Title NVARCHAR(MAX),Problem NVARCHAR(MAX),Impact NVARCHAR(MAX),

Resolution NVARCHAR(MAX),Compliance NVARCHAR(MAX),Help NVARCHAR(MAX),Excluded NVARCHAR(MAX));

--MAKE SURE YOU CHANGE THE TABLE NAME TO THE ONE YOU CREATED IN STEP-4

9). Execute the stored proc which takes input from temp table and formats the data and push it into the final result table we created in step-8

-- REPLACE THE Set @index = <WITH THE VALUE YOU GET FROM SELECT COUNT(*) FROM TEMP_VALUES>

Declare @counter int,

@index int

Set @counter = 1

Set @index = 1278

While @counter <= @index

Begin

insert into final_result

select

max(case when UID=@counter then value else '0' end) as ResultNumber,

max(case when UID=@counter+1 then value else '0' end) as ResultId,

max(case when UID=@counter+2 then value else '0' end) as ModelId,

max(case when UID=@counter+3 then value else '0' end) as SubModelId,

max(case when UID=@counter+4 then value else '0' end) as RuleId,

max(case when UID=@counter+5 then value else '0' end) as ComputerName,

max(case when UID=@counter+6 then value else '0' end) as Context,

max(case when UID=@counter+7 then value else '0' end) as Source,

max(case when UID=@counter+8 then value else '0' end) as Severity,

max(case when UID=@counter+9 then value else '0' end) as NeutralSeverity,

max(case when UID=@counter+10 then value else '0' end) as Category,

max(case when UID=@counter+11 then value else '0' end) as Title,

max(case when UID=@counter+12 then value else '0' end) as Problem,

max(case when UID=@counter+13 then value else '0' end) as Impact,

max(case when UID=@counter+14 then value else '0' end) as Resolution,

max(case when UID=@counter+15 then value else '0' end) as Compliance,

max(case when UID=@counter+16 then value else '0' end) as Help,

max(case when UID=@counter+17 then value else '0' end) as Excluded

from temp_values

Set @counter = @counter + 18

End

-- We have to make sure that this count i.e. 18 is the number of 'S' elements under the 'props' element

9). after executing the above stored procedure we’ll have all the XML data formatted in a nice tabular format in the final results table

A simple select statement from the final results table looks like this

For example if you want to concentrate first on all the warnings, you can issue a simple select statement like

SELECT * FROM FINAL_RESULT WHERE SEVERITY=’Warning’;

So this way we can arrange the BPA data in a more meaningful way and run reports and queries on those tables to make more sense/use of it.

As always feel free to provide your feedback/comments on the article.

Thanks,

Chandra

Lot of times we get into a situation where we need to restore databases from backup files (.BAK) stored in a UNC/network share.

There are two ways we can do this

• By giving the path in the RESTORE DATABASE command like RESTORE DATABASE FROM DISK = \\server_name\shared_drive\backup_file_name.bak
• By giving the path of the file (\\server_name\shared_drive\backup_file_name.bak) in the management studio restore wizard as shown below

One of the big questions asked by lot of people in forums is why we’re unable to explore/locate a UNC path/ a network drive in this wizard.

For e.g. I have 2 local drives C, D and a network drive in my PC and a network drive Z. please find the screen shot below to confirm that.

But I see only C and D in SSMS (SQL Server management Studio) restore Wizard.

Why is this happening? I tried to find the answer through Profiler (the tool that was made in heaven, by god J).

I was doing a restore through the restore wizard, reached the above shown screen which displays the directories and files in it. In the background I invoked profiler trace.

Please find the profiler trace output below

One interesting thing I found is the procedure “dbo.xp_fixeddrives”. This procedure is called in the background when we invoke the restore wizard in GUI. And this procedure get all the available drives, free space available in each drive etc

And if we run it in the query window directly, itgives the below output

If we observe the complete flow,

· First we’re calling xp_fixeddrives

· Getting the names of fixed drives. Here in my case C,D

· Passing them to another procedure called xp_dirtree, one at a time (This procedure takes an argument for e.g. logical drives like C, D and lists all the folders and files present in that drive in a loop)

Below screenshot shows the flow of actions I mentioned above, step by step.

So this procedure xp_fixeddrives can see only the fixed drives on the local computer if we execute it without any parameters.

But if you notices there’s another statement like “xp_fixeddrives 1” which is executed too. This actually lists all the network drives that are mapped to this computer.

But when we execute it in SSMS we won’t see anything. This is because to list the network drives by this procedure the share must be mapped as a network drive in the session in which
SQL Server is running. If you start sqlservr.exe from command line, SQL Server will see any drives you have mapped in your login session. When you run sqlservr.exe as a service, however,
SQL Server runs in a separate session that has no relation to your login session. You need to map the
drive in the session that SQL is running in.

So as a work around, to make network drives visible I did the following steps.

1). Created a directory in the remote computer

2). Shared it with “Everyone” – Actually We just need to make sure that the service account under which SQL is running is having permissions to access the shared folder).

3). Enabled XP_CMDSHELL by issuing

sp_configure 'XP_CMDSHELL' , 1

reconfigure with override

4). Mapped the network drive in the same session as SQL is running by issuing

xp_Cmdshell ‘net use p: \\csm1\bkp_test’

5). Now executed xp_fixeddrives with argument 1

6). The restore wizard now shows network drives

So finally we’re able to see the network drives J

But one caveat here is this will go off once we restart the SQL server. So to prevent that we can create a stored procedure like below and make it as a “Startup Procedure” or procedure that runs on every startup.

CREATE PROCEDURE [dbo].[startupnetmap]

AS

exec xp_Cmdshell 'net use p: \\csm1\bkp_test'

go

And enable it to run at startup by giving

exec sp_procoption 'startupnetmap','startup','true'

So this is why SSMS doesn’t show network drives and this is how we can work around the issue.

Feel free to share your comments and feedback on this.

Regards,

Chandra

“The resource 'disk p' could not be moved from cluster group 'Available Storage' to cluster group 'SQL Server (MSSQLSERVER)' “

By looking at the error message we can infer that it’s something related to the inability of the installer to move clustered disk resource to the group/role created by the SQL installer in this case SQL Server (MSSQLSERVER).

Some of the errors in your detail.txt looks like below

01) 2013-12-02 09:20:57 Slp:         The resource ‘Disk P’ could not be moved from cluster group 'Available Storage' to cluster group 'SQL Server (MSSQLSERVER)'. Error: There was a failure to call cluster code from a provider. Exception message: Generic failure . Status code: 183. Description: Cannot create a file when that file already exists.
(01) 2013-12-02 09:20:57 Slp:         .
(01) 2013-12-02 09:20:57 Slp:     HResult : 0x86d80035
(01) 2013-12-02 09:20:57 Slp:         FacilityCode : 1752 (6d8)
(01) 2013-12-02 09:20:57 Slp:         ErrorCode : 53 (0035)
(01) 2013-12-02 09:20:57 Slp:     Data:
(01) 2013-12-02 09:20:57 Slp:     resourceName = Disk P
(01) 2013-12-02 09:20:57 Slp:     groupName1 = Available Storage
(01) 2013-12-02 09:20:57 Slp:     groupName2 = SQL Server (MSSQLSERVER)
(01) 2013-12-02 09:20:57 Slp:     errorMessage = There was a failure to call cluster code from a provider. Exception message: Generic failure. Status code: 183. Description: Cannot create a file when that file already exists.
.(01) 2013-12-02 09:20:57 Slp: WatsonData = Microsoft.SqlServer.Configuration.Cluster.ChangeClusterGroupOperationException@53

(01) 2013-12-02 09:20:57 Slp: SQL.Setup.FailureCategory = ConfigurationFailure

(01) 2013-12-02 09:20:57 Slp: WatsonConfigActionData = INSTALL@CONFIGNONRC@SQL_ENGINE_CORE_INST

(01) 2013-12-02 09:20:57 Slp: WatsonExceptionFeatureIdsActionData = System.String[]

(01) 2013-12-02 09:20:57 Slp: Stack:

(01) 2013-12-02 09:20:57 Slp: at Microsoft.SqlServer.Configuration.Cluster.ClusterResource.ChangeResourceGroup(ClusterGroup g)

(01) 2013-12-02 09:20:57 Slp: at Microsoft.SqlServer.Configuration.ClusterConfiguration.ClusterDiskPrivateConfigObject.MoveClusterDisksToGroup(ClusterDiskPublicConfigObject diskConfig)

(01) 2013-12-02 09:20:57 Slp: at Microsoft.SqlServer.Configuration.ClusterConfiguration.ClusterDiskPrivateConfigObject.Install(ConfigActionTiming timing, Dictionary`2 actionData, PublicConfigurationBase spcb)

(01) 2013-12-02 09:20:57 Slp: at Microsoft.SqlServer.Configuration.SqlConfigBase.SlpConfigAction.ExecuteAction(String actionId)

(01) 2013-12-02 09:20:57 Slp: at Microsoft.SqlServer.Configuration.SqlConfigBase.SlpConfigAction.Execute(String actionId, TextWriter errorStream)

(01) 2013-12-02 09:20:57 Slp: Inner exception type: Microsoft.SqlServer.Configuration.Cluster.ClusterProviderDetailedException

(01) 2013-12-02 09:20:57 Slp: Message:

(01) 2013-12-02 09:20:57 Slp: There was a failure to call cluster code from a provider. Exception message: Generic failure . Status code: 183. Description: Cannot create a file when that file already exists.

(01) 2013-12-02 09:20:57 Slp: .

(01) 2013-12-02 09:20:57 Slp: HResult : 0x86d70002

(01) 2013-12-02 09:20:57 Slp: FacilityCode : 1751 (6d7)

(01) 2013-12-02 09:20:57 Slp: ErrorCode : 2 (0002)

(01) 2013-12-02 09:20:57 Slp: Data:

(01) 2013-12-02 09:20:57 Slp: ExceptionMessage = Generic failure

(01) 2013-12-02 09:20:57 Slp: StatusCode = 183

(01) 2013-12-02 09:20:57 Slp: Description = Cannot create a file when that file already exists.

(01) 2013-12-02 09:20:57 Slp: WatsonData = Microsoft.SqlServer.Configuration.Cluster.ClusterProviderDetailedException@2

(01) 2013-12-02 09:20:57 Slp: Stack:

(01) 2013-12-02 09:20:57 Slp: at Microsoft.SqlServer.Configuration.Cluster.WmiClusterResource.ChangeResourceGroup(String group)

(01) 2013-12-02 09:20:57 Slp: at Microsoft.SqlServer.Configuration.Cluster.ClusterResource.ChangeResourceGroup(ClusterGroup g)

(01) 2013-12-02 09:20:57 Slp: Inner exception type: System.Management.ManagementException

(01) 2013-12-02 09:20:57 Slp: Message:

(01) 2013-12-02 09:20:57 Slp: Generic failure

(01) 2013-12-02 09:20:57 Slp: HResult : 0x80131501

(01) 2013-12-02 09:20:57 Slp: Data:

(01) 2013-12-02 09:20:57 Slp: WmiErrorCode = Failed

(01) 2013-12-02 09:20:57 Slp: WatsonData = Failed@183

(01) 2013-12-02 09:20:57 Slp: Description = Cannot create a file when that file already exists.

(01) 2013-12-02 09:20:57 Slp: ErrorType = 1

(01) 2013-12-02 09:20:57 Slp: Operation = ExecMethod

(01) 2013-12-02 09:20:57 Slp: ParameterInfo = MSCluster_Resource.Name="Disk S"

(01) 2013-12-02 09:20:57 Slp: ProviderName = WinMgmt

(01) 2013-12-02 09:20:57 Slp: StatusCode = 183

(01) 2013-12-02 09:20:57 Slp: Stack:

(01) 2013-12-02 09:20:57 Slp: at System.Management.ManagementException.ThrowWithExtendedInfo(ManagementStatus errorCode)

(01) 2013-12-02 09:20:57 Slp: at System.Management.ManagementObject.InvokeMethod(String methodName, ManagementBaseObject inParameters, InvokeMethodOptions options)

(01) 2013-12-02 09:20:57 Slp: at Microsoft.SqlServer.Configuration.WMIInterop.Resource.MoveToNewGroup(String Group)

(01) 2013-12-02 09:20:57 Slp: at Microsoft.SqlServer.Configuration.Cluster.WmiClusterResource.ChangeResourceGroup(String group)

(01) 2013-12-02 09:20:57 Slp: Watson Bucket 2

Original Parameter Values

So we cancelled the installer and tried to move the disk resource (disk p) to the SQL role manually, through the failover cluster manager console. When I do a right click on the resource (disk) and click on the option move, the pop-up just goes away as if it has successfully moved the resource without throwing any errors but when I go and check in the resources tab, the disk which I tried to move still shows up under the ‘Available Storage’ instead of showing under the SQL resource group.

As I’m not a clustering expert, I didn’t do much research on this behaviour. But I observed this phenomenon when we use a disk resource that's having a LUN mounted as the base mount point and other LUNs mounted as folders inside it and dependencies setup among the mounted folders and the base mount point. The issue might be because of SQL installer tries to establish the dependencies at the install time, again, but as the dependencies are already there it fails to set them up (Again it’s based upon an assumption but not substantiated with any proofs).

So the workaround was,

1. End the setup

2. Go to the failover cluster manager

3. Create a role– Just click on the roles (left pane), and click “create empty role” on the right pane

4. A new role will be created with a name like “New Role”. Rename it to SQL Server (MSSQLSERVER) or whatever instance name you want

5. Move the disks using the command Move-ClusterResource "<DISK NAME>" "SQL Server (<SQL INSTANCE NAME>)" – MAKE SURE YOU LAUNCH POWERSHELL AS ADMINISTRATOR

Ref: https://technet.microsoft.com/en-us/library/ee461049.aspx

6. Launch the setup for the failover cluster installation and in the cluster resource group section select the pre-created role in the step 3 and continue with the setup (refer to the screen shot below)

As we already moved the resource to the new role installer will skip the disk movement action and setup will be completed successfully.

Another workaround using which i was able to move the installation forward is,

1.     End the setup
2.     Go to Control Panel/Programs and uninstall the database engine ONLY. Leave any other SQL Server installed components intact
3.     Open the cluster Manager and remove the Virtual Server (You can find it under the resources section of the role created by SQL installer), but leave the rest of the SQL Server resource group intact
4.     Move the disks using the command Move-ClusterResource "<DISK NAME>" "SQL Server (<SQL INSTANCE NAME>)" – MAKE SURE YOU LAUNCH POWERSHELL AS ADMINISTRATOR
5.     Begin the SQL cluster install again, specifying only the database engine. The installer will find and use the resource group and rebuild the virtual server.

The issue is not occurring in every environment. But from what I noticed, it's occurring when we select disk resource that's having a LUN as the base mount point, point and other LUNs mounted as folders

As always, please feel free to pass your comments/feedback.

Thanks,

Chandra