But then the customer came up with the Information, that the OS will be Windows Server 2012 R2. OK, older SQL on newer OS is very often a case where we need a service pack as minimum requirement. And yes, there is an article in MSDN.

https://support.microsoft.com/kb/2681562/en-us

SP1 is needed for the Support of SQL 2012 on Windows Server 2012 R2. In the past there was often something like this, just remember SQL 2008 on Windows Server 2008 R2. It was needed to just install SQL 2008 RTM and apply the SQL SP afterwards.

There are already media out there for SQL 2012 including SP1, but my customer only had the SQL 2012 RTM Version. So we tried it with the RTM Version, and wanted to apply the SP1 afterwards, because there was no recommendation until this time if SQL 2012 RTM with SP1 install is needed, or only SQL 2012 RTM and apply the SP after that.

So we started the Installation from the SQL 2012 RTM media, it all went well, the Progress bar with the insallation started.... and it was just nearly in the middle, when a message box came up, that the SQL Browser Service could not start. Hmm... went to services console, start.... no, error, could not start. So a look into the Windows Eventlog was needed. There was a nice little message...The SQL configuration for SQL is inaccessible or invalid. Hmm...

OK, we cancelled the setup and observed that was installed in 'Programs and features'. Setup Support files, SQL Browser, SQL Server (64 Bit), nearly everything was listed what is known for a successful installation. So we decided to de-install everything. It worked, only the 'SQL Server (64 Bit)' did not find any instance. OK, this really comes at the end of a SQL cluster installation, we stopped earlier.

I wanted to cleanup all the stuff in 'Program Files' and so on, but it was locked somehow.... OK, we did a reboot on both servers. Still no cleanup on Node 1 of the already installed bits in Program files etc. possible. So we left them there.

We now decided to go a different way. There is a great thing in SQL 2012 setup, called SMART SETUP. If the server is in the internet it will check there for updates. Server, in the internet..... huuuuhhhhh. No, never ever. But there is another option, you can point SQL Server Setup to a directory with the updates. So, we already had SQL 2012 SP1 setup there (This is around some 900 MB). We decided to go also for a cumulative update (CU), 6 was the actual one, so we downloaded this also. We created a directory on drive C of the server (C:\SQLUpdates, name it as you want, it is only an example here...) and copied both updates into this directory. Be careful: The CU update is in a self extracting zip, so unzip is first to get the exe and copy this to the directory.

So, how does SQL Setup now finds the updates?

 Easy.... but not with a nomal double click on Setup.exe. Open a command line, and type 'SETUP.EXE /Action=installfailovercluster /updatesource=C:\SQLupdates'

When you install a normal SQL instance, the Action is 'install', when you add a node to a SQL Cluster the Action is 'addnode'.

We started the Setup from the SQL 2012 RTM media this way, on one of the pages at the beginning SQL Server tells you what it found (Service Pack 1 and another update on top of this, right) and... the Setup ran until the end, and our Failovercluster instance was there (It was fine with all the stuff alredy in Program Files etc.... very great Setup). We did the same on node 2, with Action as 'addnode' (Copy the updates locally to node 2, or share the Directory and call in /updatesource= from an UNC path. ) and it was fine. The Cluster was up and running.

In the meantime, the article above says that there might be Setup issues and SQL 2012 SP1 with Minimum CU3 should be used. Ah, really, I can prove this, you better go with a CU on top of SP1 to be on the safe side. Check the MSDN article frequently, it is updated from time to time with new Information.

I did a lot of partitioning, in DWH scenarios (to load data in and out, to get rid of old data, effective queries with partition elimination) and in OLTP (old data out without blocking of a delete query, partition elimination).

With column store indexes in SQL 2012 you need to do partitioning to get data into the table if the columnstore index is already there. No insert, update, delete possible with activated column store index.

In my last project, an OLTP like DWH, I partitioned 25 tables, just for the purpose to get rid of old data, and to use partition elimination when querying the data and processing the OLAP cubes on top of the data.

When you want to snap a Partition out (and also into a table) you need to have a table with the exactly same structure as the partitioned table, and it must be in the same filegroup as the partition you want to work with. Most people use the ‘select top 0 * from dbo.foo into dbo.foo_copy’. This works, but try this with data compression or when your table just has calculated columns. It will fail.

So, OK, other idea (and what I used mostly in the past): I create a table dbo.PartitionConfig, which has a row for each partitioned table, and columns like ‘tablename’, ‘create_table_statement’, ‘PK_statement’, 'Index_statement’, ‘drop_statement’ and so on. With this I can set the commands configurable and in my partitioning code I just get the data from the table, nothing hardcoded. Dynamic SQL with the commands, and go. Works fine.

But with this the next problem came up: imagine you have different systems (Dev, Test1, Test2, ProdTest and Production). There will be the time, when you need to do an ‘alter table’ to a partitioned table with data. No problem, this works, but alter table adds the column at the right end of the table. If you support ‘full builds’ and ‘update builds’ in your building environment there will be a time, when the create table statement for the ‘full build’ will have a different column order than the ‘update build’ based on alter table commands. Believe me, in a project with 5 or more SQL Dev guys this will happen :-).

What is the problem with this? When switching out or in data in partitioning SQL Server does not care about the column names, so being in wrong order will be a problem. My config table was filled with a SQL script, and I was not willing to do a ‘new build’ and an ‘update build’ script just for the column order.

So this was the point to do something new. So I wrote some SP code to read the metadata from the partitioned table, do a ‘create table’ statement with this, find out the primary key, the compression state of the partition (yes, true, you can compress a partition each for its own, with difference in page or row compression). And, find out about calculated columns!

For the ease of this example I just do not use a partitioned table here, my code also works without partitions. Why this works? Well, also a not partitioned table is a table which consists of one partition, the whole table :-).

Attached you will find the Procs and Scripts to setup the Scenario, they are commented inside the code. Here are only the calls and what happens.

This is my table example:

CREATE
TABLE PartTest

(      ID INT NOT NULL,

       DayDate DATE NOT NULL,

       DayTime TIME(0) NOT NULL,

       AnyValue decimal(10,2) NOT NULL,

       AnyString VARCHAR(255) NOT NULL,

       CalcColumn AS 5 * ID,

       CONSTRAINT PK_PartTest PRIMARY KEY (ID, DayDate)

) WITH (DATA_COMPRESSION = ROW)

GO

First of all, the copy of the table already exists, so I have a procedure to drop it.

EXEC [dbo].[usp_DropSwitchOutTable] 'dbo.PartTest_Out'

Then, just create the copy using the stored procedure. The first parameter is the source table, the second one is the name of the table I want to create as a copy, the third is the filegroup in which the copy table should be created (it must be the same as the source table when switch in and out of partitions should work, as already mentioned).

EXEC [dbo].[usp_CreateSwitchOutTable] 'dbo.PartTest', 'dbo.PartTest_Out', 'PRIMARY'

Check out your database, it now has a second table, it should have the same structure as the source table and is in the filegroup you specified. How do I find out in which filegroup my
partition is, if I have a table with partitions spread over multiple groups? You find out in the metadata, just check my attached function.

dbo.usf_GetFilegroupForPartition

Parameters are Name of the partition schema, and the Partition number. How to get this number, see later in this article. How to get the partition schema? Oh, just use my attached function

dbo.usf_GetPartitionSchemaForTable

All you need is the table name in the form of schema.tablename.

Next step is to add the primary key to the copy table.

EXEC [dbo].[usp_CreatePrimaryKeySwitchOutTable] 'dbo.PartTest', 'dbo.PartTest_Out'

Check it out, there will be a PK now on the copy.

Finally, find out the compression and apply it to the copy table.

EXEC [dbo].[usp_SetSwitchOutTableCompression] 'dbo.PartTest', 'dbo.PartTest_Out', 1

Check out your copy table, it will now have row compression. The third parameter is the partition number, without partitions in the table this is 1, if you want to use a partition inside your table you must find out the number from the metadata. The easiest way to do so is the $partition metadata function, you just need to know the name of the partition function of your table, and by which criteria it is partitioned, so if it is by day, just throw in the date, and you get the Partition ID or number in which the data for that day is in. Check out https://msdn.microsoft.com/de-de/library/ms188071.aspx for the details of the function.

If you do not know the name of the partition function of your table, use my attached function.

dbo.usf_GetPartitionFunctionForTable

All you need is the table name in the form of schema.tablename.

So, this is all the magic.

Do a test on your own, just add a column to the table with alter table, and run the code again. The copy table should have the row added also. Or try to create the table in a filegroup other than PRIMARY. Or, much better, setup a partitioned table and use my code.

What to do with the created copy of the table? Well, just identity the partition you want to switch out of your partitioned table and use the alter table switch Partition statement to move the data out, drop the copy table, and the data is gone. Don’t forget to so a merge on your partitioned table afterwards to get rid of the ‘hole’ in your table.

Partitioning.zip

I have installed it, and did the example that comes with the CTP. Ok, it worked, but I
wanted to upload some of my own data and create a hadoop table on my csv ‘big’
data.

But how?

I have inspected the scripts from the example and found out, that HDInsight is just a
structure of data files. So uploading the files is necessary.

The right thing to use for any action in HDInsight is the Hadoop console, the link is on
the desktop.

To upload data one should in the first step create a directory for the data. This
can simply be done by using the hadoop command line tool in the console.

hadoop dfs -mkdir /test/data

easy….

To upload data:

hadoop dfs -copyFromLocal C:\temp\testdata.csv /test/data/testdata.csv

easy….

Get rid of the data again?

hadoop dfs -rmr -skipTrash /test

Be careful, -rmr means recursive, so it kills also all subdirectories and files in
there. And… -skipTrash means that is skips the trash, so it is gone immediately
with no way to get it back.

For better use of the commands I have changed some of the CTP examples powershell
scripts to be more generic. The powershell must be called in the hadoop command
window. The powershell syntax there in General

powershell -ExecutionPolicy unrestricted –F <SCRIPTNAME>.ps1 <ARGUMENTS>…

I have created three powershell scripts, create_Directory.ps1, import_Data.ps1 and
remove_Directory.ps1.

create_Directory.ps1:

powershell -ExecutionPolicy unrestricted –F create_Directory.ps1 <DIRECTORY>

<DIRECTORY> can be /test, /test/data, test/data/whatever… and so on.

import_Data.ps1:

powershell -ExecutionPolicy unrestricted –F import_Data.ps1 <SourceDirectory> <SourceFile> <targetDirectory> <targetFile>

remove_Directory.ps1:

powershell -ExecutionPolicy unrestricted –F remove_Directory.ps1 <DIRECTORY>

Be careful, the calls inside are recursive and with skip trash.

The powershell scripts are attached. Have fun :-)

Perhaps I do another blog post on how to create a simple table out of the imported data….

Scripts.zip

OK, tough goal. But why?

To encrypt a column in the database (be aware: this has nothing to do with
transparent data encryption, TDE, this here is about encryption of a single
column in a table) you create the column as varbinary(128). Then you need a
key, symmetric or asymmetric, to encrypt and decrypt. Or a certificate.

So, what can one use here?

Encrypt by Key:

• Symmetric or asymmetric
• Key will be generated in the database
• You need an additional master key in the database, which protects this key
• You open the key, and encrypt
• You open the key, and decrypt
• There are different algorithms (DES, AES etc…)

Encrypt by certificate:

• Create a certificate
• Encryption using certificates is based on asymmetric algorithms

So, OK, first stop here: the requirement was, that the dimension table can hold some
100 000 rows. But asymmetric encryption is not recommended for lots of rows.
So, encrypt by certificate only works with asymmetric algorithms, so this is
not an option.

So we need to go for Encryption by key. But here our next problem comes up: We must
open the key first to use it.

We should reflect on how the data from a SQL Server database is read from SSAS. It
can be done directly from the table, or, the recommended way, using a view. But
in both ways it is not possible to use an ‘open key’ in the SQL code….

One possibility is to use a view, and inside the view use a stored procedure with
open query. In the procedure you can run any code you want. I tried that, and
it worked…. But, should I really go for this?

Just think a little bit more about it. And, there is a solution. The key must open
itself when needed, and really, there is a possibility to do that. All you need
is a combination of symmetric and asymmetric keys, a SQL Server key and some
permissions for the right account….

Let’s do it using demo code…

First we need a demo DB:

use master

go

CREATE DATABASE Crypt

go

use Crypt

GO

Then we need a database master key:

-- Database Master Key.... we need one….

if not exists (select
* from sys.symmetric_keys where symmetric_key_id = 101)

create master key encryption by password =
'Password123'

Do not forget the password that you set here!!! If you need to restore the database
later on a different instance of SQL Server, you need that, otherwise the
encrypted data stays encrypted forever….

To protect (encrypt) this key, we connect it to the SQL instance the DB is running
on. This prevents people from opening the master key when stealing a backup,
and we need it for the automatic opening later. The Service Master Key opens
automatically when the instance starts running and decrypts the database master
key. But: the database master key is encrypted by the service master key, so
the DB is bound to the SQL instance for that to work:

ALTER MASTER KEY ADD
ENCRYPTION BY SERVICE MASTER KEY

go

Now we create an asymmetric key. This will not be used for the column encryption
(asymmetric is bad for a lot of data, you remember….), it is used to protect a
symmetric key we create afterwards:

IF NOT EXISTS (select
* from sys.asymmetric_keys where name = 'STORE_CRYPT_ASYM')

BEGIN

      Create asymmetric key STORE_CRYPT_ASYM

            with algorithm = RSA_2048;

END

Use any algorithm here which is available, I went for the RSA_2048.

Now we create the symmetric key, which will be encrypted by the asymmetric key we just
created. With this trick, we do not need to encrypt the symmetric key with a
password, which must later be used for an ‘open key’…..

IF NOT EXISTS (select
* from sys.symmetric_keys where name = 'STORE_CRYPT')

BEGIN

create symmetric key STORE_CRYPT with
algorithm = DES

           encryption by asymmetric key STORE_CRYPT_ASYM

END

Here we use the DES on the symmetric keys, it is just my favor, use any that is
supported there, look at books online.

Now we create a table:

CREATE TABLE Demo

(

      ID int identity(1,1) not null PRIMARY KEY,

      CryptColumn varbinary(128) not null,

      InsertDateTime datetime not null

)

GO

For the inserts into the table, we need to open the symmetric key first…

open symmetric key
STORE_CRYPT decryption by Asymmetric key STORE_CRYPT_ASYM

This works because we are sysadmin on the SQL instance. It also works as a dbo in
the database. For others to do that, you need the right permissions set…. More
later to come here on that topic.

We do an insert into the Table…

insert into dbo.Demo

(CryptColumn, InsertDateTime)

select EncryptByKey(Key_GUID('STORE_CRYPT'), 'Dies ist ein Test'), GETDATE()

Close the key afterwards

close symmetric key STORE_CRYPT

We do a select on the table…

All fine, not readable with a normal select.

Now we create a role in the database

if not exists (select * from sys.sysusers where name = 'DeCryptColumns' and issqlrole = 1)

      create role DeCryptColumns

go

We give this role the two rights necessary to use the Keys…

grant view definition on symmetric key::STORE_CRYPT to [DeCryptColumns];

go

grant control on asymmetric key::STORE_CRYPT_ASYM to [DeCryptColumns]

go

A select on the table with the decrypt column code inside… and, just remember…. We did
not open the key as we did for the insert!

select ID,

CONVERT(VARCHAR,
DecryptByKeyAutoAsymKey(AsymKey_ID('STORE_CRYPT_ASYM'), NULL, CryptColumn)) AS CryptColumn,
InsertDateTime FROM dbo.Demo

This happens when you have the right to use the keys, they open by itself and
decrypt.

If you don’t have the rights, your just get NULL instead of the clear text in the
CryptColumn data column.

So, all you need to do, to have this in clear text in the SSAS cube is…..

• Use a view to select the data for the cube and include the DecryptByKeyAutoAsymKey
  for the column to be decrypted
• Create the account used for processing the cube as login in the SQL Server instance
  (you must do so in any case, otherwise you have no chance to read the data)
• Map the login additionally to the DeCryptColumns role in the database. I suppose
  you gave the account already the db_datareader role to be able to read data in
  the database to fill the cube
• Process the cube, and the column is in clear text when browsing the cube

The data is readable in the cube, but encrypted in the table. So someone without rights
to the keys only sees NULL, and someone who steals the database backup also
sees encrypted data, or NULL.

So I added a script task in SSIS and build up the whole query, including the values from the variables, in VB.net in the script task, and gave it back as another variable. I have used the variable with the query inside than as datasource in the data flow task and... it was fast, also in SSIS.
It seems to me that sometimes passing variables with the placeholders ? in SSIS really generate bad execution plans in SQL Server. I changed some other things in the query, but the biggest effort was to not use the placeholders for the variables and build the query in a script task completely and just pass it to the data source.

By the way... the query (together with two other queries which I have changed accordingly) had a runtime of 3 hours for the SSIS package.... my changes brought it down to 1 min 40 seconds !

Today SQL Server databases are in nearly 99% of all cases installed as
case-insensitive. So you get your data to import from somewhere, directly from
a database, by Excel Files, by csv-Files, and you just import to SQL Server.
During import to your data warehouse there are some lookups necessary, to look
for IDs in referential tables etc. All the import stuff on your DWH works fine,
for month, and then, suddenly, peng! The job failed last night. After hours of
investigation you found out that in one of the imports a string that you use
for a lookup task is written in different upper and lowercase than before, and
this crashes your lookup task, which is configured to fail the component if a
no lookup match happens.

So, my hint is: always compare strings in lowercase (or Uppercase, will also work)

in SSIS if you cannot make sure that

the upper and lowercase on the important lookup columns always comes in the
right case. Just imagine, your data comes from UNIX, so upper and lower case is
really a difference there, but in your system, on SQL Server, case insensitive,
those strings are the same.

What I do in SSIS: I add all strings that I need for comparison in a derived column
task to lower(column) in addition to the original column. When doing comparisons,
i.e. in a lookup task, I do not use the reference table as data source, I
always write a query, and make the lookup strings lower(Column) in the
recordset. Then I compare the derived lower column with the lower column from
the lookup. And, voila, the lookup always works, regardless of what comes from
the outside world into your DWH :-)

Is it possible to do a transactional replication from SQL Server to Sybase?

• Yes, it is. You can replicate via the Sybase OLE DB driver, if you set it up as a linked Server. You can only setup this replication using the SQL Server system SPs for replication.

Is it supported?

• Hmm, it is not explicitly not supported :-). Transactional replication is supported for so called non-SQL Subscribers, but all the examples are only for Oracle as replication target.

Which replication will work?

• Transactional, thats it. And only push is possible.

What if I want to replicate from Sybase ASE to SQL?

• You can use the Sybase Replication Server, but this is not for free. It costs per processor core of the replication machine. Hint: use a virtual machine as replication server and give it one processor. Saves you money

Can I replicate via stored procedures as in SQL Server?

• YES. You can create SPs in Sybase for Update, Insert and Delete and configure those to be called when the data comes in.

Thats it. If you want to know more about, give me a mail. It is too much to write down here.

No, there is a tool from Microsoft, the Microsoft Assessment and Planning Toolkit (MAPS). I agreed with the customer that we will use it. When using it I found out some specials which I will talk about here.

1. Installation

Installation is easy, download the tool from the web, run setup, and….. no, not done. First of all, the tools needs a SQL Server instance to store the data. And, the special thing is, the SQL Instance cannot be anywhere, it must be on the computer where you install the tool. During setup it gives you the chance to download SQL 2008 Express, which is really good for the job here, but you can also pre-install your own SQL Instance using Express, Standard, Enterprise or Developer Edition. But there is one thing you must be aware of: the installation must be a named instance and the name must be MAPS. Really true.

2. Installation 2nd

When installing there is the prerequisite that Office 2007 SP2 or Office 2010 must be on the computer. The MAPS tool can automatically create nice decision maker ready Excel sheets with results, but just for collecting data there is really no need to have Office on the computer. It would be more useful to not have this need, just install and collect to SQL. Then backup the SQL database, copy to your laptop, restore there rand make the rest of the work there, including creating those nice Excel sheets. This is the way I did it; I took the results with me and did the documentation in my home office. But, nevertheless, the installation needs it, so prepare your customer to have the installation media ready.

3. Collection of data

The collection runs automatically. First of all you must configure a database on the MAPS, instance, the tool asks you when starting if you want to create a new one, or use an existing one. You should create a new one when you want to do a new collection, choose an existing one when you want to use already collected data.

On a new collection: first of all you must run the ‘Inventory and assessment wizard’. This one will collect basic data about the servers, OS version, SQL version, services running etc. Which server to collect can be specified in many ways, by a text file in which the servers are just entered, each one manually in the tool etc. I will not explain every option here, use the tool and discover yourself. You can enter one user account for collecting the data, of enter a number of user accounts and the tool will try all of them one after another on each of the configured servers until it finds one which has access. This is useful if the servers are in different domains; this was the case at my customer.

After the successful run of the ‘Inventory and assessment wizard’ you can collect the performance data. Make sure that you capture all times, busy times, month endings (creation of bills etc.). In my case we let the tool collect the data for 6 weeks! 30 minutes is the minimum time, by the way, otherwise the tool is doing nothing :-)

4. Collection, be careful!

At my customer the collection failed. But why? OK, it took me some time, but then I found out that the WMI service was disabled at the servers, and the tool does WMI calls. So enable and start the service, and it works…. But then it stopped…  ok, a lot of customers have group policies which disable the WMI service because of security. So, my hint: Clarify with the customer before you start, because I have the experience at my customers that it takes some days to change the group policies (and in addition, there will be some discussions with the AD staff for what you need WMI :-)).

5. Collection, be very careful!!!

The program must be opened while collecting; it is not a windows service! So, you must lock the console and wait. In my case at the customer a different person started the collection and locked the console, so the person I worked with at the day we wanted to stop the collection just used its administrative account to log out the account (which is a hard program stop from process perspective).

OK, we did this, logged on to the computer, started the MAPS program, when asked we took the existing database we collected the data into, went to the section ‘Server consolidation’, and then to the option ‘Performance metrics results’…. And the MAPS tool told us ‘sorry, I do not have any collected performance data’. WHAT?

OK, there is the option of just opening the database in Management Studio, there are tables called dbo.Performance_disk, dbo.Performance_memory etc…. and all of them are full of data! But why has the tool the opinion there is no data?

There is a table called dbo.Performance_statistics… and this is empty. I checked on my database on my laptop with a successful collection, and there is a row in the database for each server I have collected. OK, so what happened? Easy…. The MAPS tool collects for days and weeks and months, and write the data to the SQL database, but… this table is written when the collection is ended, but we…. Stopped the program by a hard logoff of the user… and then it does not write the table, and… the tool says ‘sorry’ :-(.

Solution? Easy… we just started a new collection, the tool ask if we want to overwrite the data (NO, NO, NO!!!), or if we want to keep the data (YES YES YES!!! :-)). We collect…… 31 minutes (as you remember mentioned before, 30 minutes is the minimum), after that we close the collection in the tool, and we are done. The table is filled now, and the MAPS program shows the results. YES! :-)

6. One last hint

The collected data is a nice playground for your documentation. The documentation written by the MAPS tool is ok, but only gives number of collections of data, minimum value, maximum value, average…. But perhaps you want to know how often the processor was above 90 % during the collection time for a special server, or stuff like that. All of that you find easily in the table in the database. T-SQL is your friend, the tables are named dbo.Performance_ and the rest tells you whats inside, disk, memory, processor……. :-)

That’s it for today. By the way, this info here is for MAPS 5.0. There is new version out, 5.5, but this was in beta when I did my job and not public available. So I decided to use the officially released 5.0 version. Are my hints here still valid for 5.5… no idea. Find out yourself :-)