1. Il servizio SQL Server viene “stoppato” con il comando: NET STOP
2. Il servizio SQL Server viene “stoppato” tramite SSMS
3. Il servizio SQL Server è clustered e si verifica un failover automatico o manuale
4. In tutte le condizioni in cui il servizio SQL Server viene “stoppato” senza che prima venga effettuata un operazione di CHECKPOINT

Tale comportamento è documentato nei books online:
https://msdn.microsoft.com/library/ms186775(v=sql.110).aspx
“The identity property on a column does not guarantee the following:

• Uniqueness of the value – Uniqueness must be enforced by using a PRIMARY KEY or UNIQUE constraint or UNIQUE index.
• Consecutive values within a transaction – A transaction inserting multiple rows is not guaranteed to get consecutive values for the rows because other concurrent inserts might occur on the table. If values must be consecutive then the transaction should use an exclusive lock on the table or use the SERIALIZABLE isolation level.
• Consecutive values after server restart or other failures –SQL Server might cache identity values for performance reasons and some of the assigned values can be lost during a database failure or server restart. This can result in gaps in the identity value upon insert. If gaps are not acceptable then the application should use a sequence generator with the NOCACHE option or use their own mechanism to generate key values.
• Reuse of values – For a given identity property with specific seed/increment, the identity values are not reused by the engine. If a particular insert statement fails or if the insert statement is rolled back then the consumed identity values are lost and will not be generated again. This can result in gaps when the subsequent identity values are generated.

These restrictions are part of the design in order to improve performance, and because they are acceptable in many common situations. If you cannot use identity values because of these restrictions, create a separate table holding a current value and manage access to the table and number assignment with your application.”

Il seguente script riproduce la problematica:

CREATE TABLE [TestTable] ([C1] INT IDENTITY, [C2] CHAR (100));

GO

INSERT INTO [TestTable] VALUES ('Transaction 1');
INSERT INTO [TestTable] VALUES ('Transaction 2');

aprire un command prompt con privilegi elevati (run as administrator)
net stop MSSQLSERVER
net start MSSQLSERVER

INSERT INTO [TestTable] VALUES ('Transaction 3');
INSERT INTO [TestTable] VALUES ('Transaction 4');

SELECT * FROM [TestTable]

Il risultato che si ottiene è il seguente:

Generalmente la discontinuità nei valori dei campi IDENTITY non crea problemi applicativi, a patto che tali campi vengano utilizzati come chiavi surrogate, cioè i valori non devono avere un significato di bussiness:
https://en.wikipedia.org/wiki/Surrogate_key
al contrario di quello che succede per le chiavi naturali:
https://en.wikipedia.org/wiki/Natural_key un tipico esempio di una chiave naturale in letteratura è il SSN simile al, molto più nostrano, Codice Fiscale.

Ad essere sinceri anche a me, sull’istanza del mio portatile, si sono verificati numerosi gap sugli IDENTITY, probabilmente in corrispondenza di alcuni shutdown non proprio da manuale, che ogni tanto mi capita di fare. Nella figura che segue un esempio:

Nel mio caso la discontinuità dei valori non mi ha creato nessun problema perché ho usato il campo IDENTITY come chiave surrogata, non mi ero accorto del gap se non dopo parecchio tempo, quando sono andato a modificare dei record direttamente sul DB, perché non ho avuto ancora il tempo di implementare la modifica di un determinato valore dalla mia applicazione.

Se al contrario si è scelto di dare un significato di bussiness, ai valori generati tramite attributo IDENTITY, e un eventuale gap crea un problema, si può utilizzare uno dei seguenti workaround.

Workaround 1 (retrocompatibilità)

Utilizzare il trace flag 272 tra gli startup parameters dell’istanza. Da notare che la stringa da aggiungere è –T272 (la “T” deve essere uppercase).
Questo trace flag forza il SQL Engine a generare gli IDENTITY emulando le versioni pre 2012, cioè verrà effettuata una scrittura sul transaction log per ogni valore.
L’attivazione di questo trace flag potrebbe creare un impatto sulle performances.
Il trace flag è deprecato e nelle prossime versioni potrebbe essere rimosso.
Per verificare questo workaround possiamo modificare lo script presentato prima come segue:

DBCC TRACEON(272, -1)
GO
CREATE TABLE [TestTable] ([C1] INT IDENTITY, [C2] CHAR (100));
GO

INSERT INTO [TestTable] VALUES ('Transaction 1');
INSERT INTO [TestTable] VALUES ('Transaction 2');

aprire un command prompt con privilegi elevati (run as administrator)
net stop MSSQLSERVER
net start MSSQLSERVER
INSERT INTO [TestTable] VALUES ('Transaction 3');
INSERT INTO [TestTable] VALUES ('Transaction 4');

SELECT * FROM [TestTable]

Il risultato che si ottiene è il seguente:

Workaround 2 

Effettuare manualmente il commando CHECKPOINT prima di riavviare l’istanza o di effettuare un failover.
Tale soluzione è praticabile solo durante le operazioni programmate.
Per verificare questo workaround possiamo modificare lo script presentato prima come segue:

CREATE TABLE [TestTable] ([C1] INT IDENTITY, [C2] CHAR (100));
GO

INSERT INTO [TestTable] VALUES ('Transaction 1');
INSERT INTO [TestTable] VALUES ('Transaction 2');

CHECKPOINT

apriere un command prompt con privileggi elevati (run as administrator)
net stop MSSQLSERVER
net start MSSQLSERVER

INSERT INTO [TestTable] VALUES ('Transaction 3');
INSERT INTO [TestTable] VALUES ('Transaction 4');

SELECT * FROM [TestTable]

Il risultato che si ottiene è il seguente:

Workaround 3 (Sequence)

Sostituire l’uso dell’IDENTITY con una SEQUENCE con l’argomento NO CACHE, come descritto nei books online https://msdn.microsoft.com/en-us/library/ff878091.aspx:

CREATE SEQUENCE s1 AS INT
START WITH 1
NO CACHE;

l’uso del NO CACHE avrà l’effetto, come nel caso dell’IDENTITY nelle versioni antecedenti a 2012, di produrre una scrittura di un record sul transaction log per ogni valore generato. Questo chiaramente annulla l’incremento di performance guadagnato con la generazione “a batch” delle sequenze di valori.
A seguire due esempi di utilizzo della SEQUANECE che emulare l’IDENTITY:

CREATE TABLE t
(
Id INT PRIMARY KEY DEFAULT NEXT VALUE FOR s1,
val INT NOT NULL
);

INSERT INTO t (val) VALUES (2)

oppure:

CREATE TABLE t
(
Id INT PRIMARY KEY,
val INT NOT NULL
);

INSERT INTO t (id, val) VALUES (NEXT VALUE FOR s1, 2)

Per testare anche questo workaround possiamo modificare il solito script per utilizzare la SEQUENCE:
CREATE SEQUENCE s1 AS INT
START WITH 1
NO CACHE;

CREATE TABLE [TestTable] ([C1] INT, [C2] CHAR (100));
GO

INSERT INTO [TestTable] ([C1], [C2]) VALUES (NEXT VALUE FOR s1, 'Transaction 1')
INSERT INTO [TestTable] ([C1], [C2]) VALUES (NEXT VALUE FOR s1, 'Transaction 2')

apriere un command prompt con privileggi elevati (run as administrator)
net stop MSSQLSERVER
net start MSSQLSERVER

INSERT INTO [TestTable] ([C1], [C2]) VALUES (NEXT VALUE FOR s1, 'Transaction 3')
INSERT INTO [TestTable] ([C1], [C2]) VALUES (NEXT VALUE FOR s1, 'Transaction 4')

SELECT * FROM [TestTable]

Il risultato ottenuto è quello corretto:

Come controprova, se per caso, avessimo dimenticato il NO CACHE, avremmo ottenuto un gap (provate a eseguire nuovamente lo script commentando il NO CACHE):

Nota: nel caso degli IDENTITY e della SEQUENCE otteniamo dei gap di valori differenti, ciò è dovuto alle diverse dimensioni del buffer che contengono i valori generati in modalità batch.

Conclusione

A causa della struttura degli indici di SQL Server:
https://technet.microsoft.com/en-us/library/ms177443(v=sql.105).aspx
https://technet.microsoft.com/en-us/library/ms177484(v=sql.105).aspx
le SEQUENCE o di campi con attributo IDENTITY, sono ottimi candidati per essere usati come primary key clustered questo perché permetteranno di avere:

• row locator sugli indici nonclustered compatti
• datatype performanti per le operazioni di join tra tabelle (la join tra tabelle con clausola di ON su campi int o è molto più performante della stessa operazione su campi ad esempio varchar(255)).

Si suggerisce quindi di progettare le basi dati in modo tale che i valori delle primary key, prodotti con SEQUENCE o tramite l’attributo IDENTITY, siano delle chiavi surrogate.
Se non fosse possibile cambiare la logica implementata all’interno del database si consiglia di utilizzare il workaround 3.
Se non fosse possibile nemmeno modificare il codice è possibile usare il workaround 1.

Marco Giordani

• [TechNet] Welcome to the Microsoft ODBC Driver 11 for SQL Server on Linux (https://technet.microsoft.com/en-us/library/hh568451(v=sql.110).aspx).

While we can use a standard Microsoft Azure image for this I opted for a full blown on premise hyper-v based installation.

First step: acquire a linux distribution

For this tutorial we’ll use CentOS. You can get the install image from here: https://www.centos.org/ .

Second step: install CentOS

I will not go into the details here, just make your Hyper-V settings include a network adapter bound to an external network or a NATed internal network. In this lab I assume the hypervisor hosts the SQL Server instance so we need connectivity between CentOS (virtual) and the hypervisor. Please make sure your firewall-ipsec configuration allows that. I will also assume internet access for the virtualized OS (for upgrades and so on).

To keep things simple we can pick the CentOS wizard suggestions:

Pick an hostname:

Allow the installer to use all the virtual hard disk space:

Installation type “minimal” would suffice (we don’t need a GUI after all) and we can add what's needed afterwards anyway:

After the installation and the following reboot you need to login as root with your chosen password.

Note: you should create another user to avoid abusing root. This is not shown here for the sake of simplicity. If you’re interested take a look at useradd, passwd and visudo.

Third step (optional): install Linux Integration Services Version for Hyper-V

Get them from here: https://www.microsoft.com/en-us/download/details.aspx?id=34603 . Since it’s an ISO we just need to load it using the Media->DVD Drive->Insert disk… hyper-v menu:

To mount the DVD in CentOS just issue there two commands:

sudo mkdir /mnt/dvd

and

sudo mount /dev/dvd /mnt/dvd

(note: if you are root you don’t need the sudo command).

Now you should be able to cd into /mnt/dvd and pick the right version (in my case RHEL63):

cd /mnt/dvd/RHEL63

Now issue the install.sh script with

sudo ./install.sh

And wait.

When the installation is over, you have to reboot your CentoOS server:

sudo reboot

That’s it. Now your CentOS installation should be a better hyper-v guest :).

Fourth step: enable the network connection and get an IP

To enable the internet connection at boot and get a DHCP leased address we need to edit the /etc/sysconfig/network-scripts/ifcfg-eth0 file. To do this we issue a

sudo vi /etc/sysconfig/network-scripts/ifcfg-eth0

Just edit the file and set ONBOOT=yes instead of ONBOOT=no. Make sure that BOOTPROTO is dhcp so we’ll get an address from the DHCP server.

Note: vi can be awkward if you are used to the easy-yet-powerful Microsoft editors. If you have problems please let me know.

Now at reboot we should have a valid IP.

To test it perform a full system update:

sudo yum update

While we are at it, install some useful commands:

sudo yum install wget nano tar gcc

And reboot again.

Fifth step: Download and install Microsoft® SQL Server® ODBC Driver 1.0 for Linux

Go here and download the package: https://www.microsoft.com/en-us/download/details.aspx?id=28160.

For this purpose I’ve created a folder in my home directory called dev an downloaded the package there. You might want to download it using Windows and then upload it using SFTP protocol or just wget it:

Extract it:

tar xvf sqlncli-11.0.1790.0.tar.gz

Now just issue the ./build_dm command:

./build_dm

And follow the on-screen instructions.

As you can see we still need to install the driver with:

cd /tmp/unixODBC.24884.23543.20110/unixODBC-2.3.0 (your path will be different!)

and then

sudo make install

As last thing to do we should go back to our dev folder:

cd ~/dev/sqlncli-11.0.1790.0

And issue a sudo ./install.sh install command:

sudo ./install.sh install

And we are done!

Sixth step: connect to SQL!

Now we can use the well known sqlcmd utility to query our database. In this lab I’ll use SQL authentication but know that Kerberos authentication is supported too.

To connect to my SQL instance (default with port 1433) all I need is:

sqlcmd –s myserver –U username –P password

Now I can issue a very common TSQL statement and check it:

SELECT name FROM sys.databases

We succeeded querying a SQL Server database from a brand new hyper-v hosted CentOS virtual server!

Happy coding,

Francesco Cogno

Let’s talk about a simple example: the wind speed measurement. Suppose we have a telemetry telling us the actual speed of the wind. The data will be roughly like this:

Notice that we have 11 equidistantiated samples (one each second). This will translate very well in a graph:

Looking at the graph, you can clearly see that there was a spike lasting two seconds. Otherwise the value is more or less constant.

Now we suppose we lose the connection for 2 minutes with the telemetry and we lose data. When the connection is restored we receive another 8 samples:

The graph becomes this one:

 You see the problem here? Even though the legend shows the correct time, just looking at the graph we might not be aware of the data loss. Even worse, we might think that between the two spikes there were only few seconds. This is not good:

What should we do? There is no right answer; generally I tend to keep the last value until a new one is available. This creates a flat line: it’s clear that something went wrong. It is also clear that minutes passed between a spike and the other:

Now let’s go back to the original dataset:

This time we receive a flood of data (more than one each second):

 This leads to this misleading graph:

We seem to have a huge spike lasting very long! The problem here is – again – that the time scale is not linear:

What can we do in this case? I tend to suggest using the weighted average: this is mathematically correct even though this may cause loss of information. The graph with the linear time will be:

Aside of having a lower spike, it’s also evident that the spike lasted just a few seconds.

A more formal definition of what I’ve done here is kindly given by a contributor who wished to remain anonymous:

Just to explain a little, consider this simplified example. Given a linear time axis:

We add two samples at Cp(0)  and at Cp(1). The values are f(Cp(0)) and f(Cp(1)). The goal here is to get the value at each x(i).

The first sample is exactly at the x(0) . This means that

.

That’s easy.

Now since we don’t have samples between x(0) and x(1) we choose maintain the same value.

This means that

. 

In other words we are weighting the value for relative weight in the interval:

.

Of course since  x(1)-x(0) is our deltaT, we have:

.

 Now, we add another sample Cp(2). This sample is after x(2) but before x(3).

What will be the value of f(x(2))?

It will be the value of f(x(1)) until a new sample is found (Cp(1)). From that on will be f(Cp(1)). To even things out we just need to weight the values for their duration:

Wow. Notice that since: 

we can write the function this way:

.
This is the same as:

.

See what we are getting to? I will not cover here the other scenarios because I think you got the hang of it. The algorithm here is easy: for each period calculate the weighted average of its samples (taking account of the previous sample if need be). The code is very easy indeed:

public List<DateValuePair> Push(DateValuePair dvp)
        {
            List<DateValuePair> lDVPs = new List<DateValuePair>();

            if (fFirst)
            {
                Start(dvp);

                // Add itself as first event
                lDVPs.Add(dvp);

                return lDVPs;
            }

            DateTime NextStep = CurrentStep.Add(Step);

            while (dvp.Date >= NextStep)
            {
                double dUsedMS = Math.Min((NextStep - CurrentTime).TotalMilliseconds, 1000);
                AccumulatedValue += (dUsedMS * Current.Value) / 1000.0D;
                lDVPs.Add(new DateValuePair(-1) { Date = NextStep, Value = AccumulatedValue });
                AccumulatedValue = 0;

                CurrentStep = NextStep;
                CurrentTime = NextStep;
                NextStep = NextStep.Add(Step);
            }

            double dRemainining = Math.Min((dvp.Date - CurrentTime).TotalMilliseconds, 1000);
            AccumulatedValue += (dRemainining * Current.Value) / 1000.0D;
            CurrentTime = dvp.Date;

            Current = dvp;

            return lDVPs;
        }

We expose a method to be called for each sample. Notice that for each call we may receive an arbitrary number of results. This is expected as we cannot know in advance how many time frames have passed between a sample and the next one.

The algorithm revolves around an accumulator (called AccumulatedValue) that holds the weighted partial value for a given interval. As you can see, I am arbitrarily setting the  to 1 millisecond: this should be enough for most scenarios but feel free to change if you need. With the accumulator in place it’s just a matter of looping through the samples until you complete an interval. 

Our last task is to expose the method via T-SQL. There are many ways of doing this but given the limitation of SQLCLR dataset passing I opted for using the context connection (see https://msdn.microsoft.com/en-us/library/ms131053.aspx). This is not elegant at all but gets the job done. If you think of a better approach please let me know.

The definition is:

CREATE FUNCTION [Normalization].TimeNormalize(@statement NVARCHAR(MAX), @days INT, @hours INT, @minutes INT, @seconds INT, @milliseconds INT)
RETURNS TABLE(
	[EventTime]			DATETIME,
	[NormalizedValue]	FLOAT
) 
AS EXTERNAL NAME [ITPCfSQL.Azure.CLR].[ITPCfSQL.Azure.CLR.TimeNormalization.TimeNormalizer].TimeNormalize;
GO

As you can see, we should pass this SP the TSQL statement that will produce the dataset to normalize. The dataset must have two columns, the first one being a DATETIME and the second a number. The other parameters specify the time interval used for the normalization. You can go as low as one millisecond (as explained above).

Let’s try a demo. First we create a staging table:

CREATE TABLE tbl(Evt DATETIME, Val FLOAT);

 And we put some events in it. The events are hardly evenly spaced:

INSERT INTO tbl(Evt, Val) VALUES('20140101 17:00:10.000', 0);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:00:10.123', 230);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:00:10.145', 244);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:00:10.153', 240);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:00:10.165', 248);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:00:10.175', 246);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:00:10.215', 100);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:00:10.218', 14);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:00:11.218', 13);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:00:12.218', 13);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:00:13.218', 13);

Now let’s see our data with the linearized timescale (using our algorithm):

SELECT * FROM tbl;
SELECT * FROM [Normalization].TimeNormalize('SELECT * FROM tbl', 0, 0,0,1,0);

This gives us:

Notice here a how the same starting information we are getting two different graphs: the first one is clearly misleading (look at the scale: it goes up to 500!), the second is much easier to understand. Don’t misunderstand me here: I’m not saying that the first graph is wrong. It is not. It’s just misleading because we expect the graph to have linear axis.

Last demo. Let’s simulate a connection loss. To do so, we just skip some time in our last sample:

TRUNCATE TABLE tbl;
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:00:10', 14);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:02:11', 13);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:02:12', 14);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:02:13', 14);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:02:14', 13);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:02:15', 33);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:02:16', 7);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:02:17', 33);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:02:18', 30);
INSERT INTO tbl(Evt, Val) VALUES('20140101 17:14:17', 14);
SELECT * FROM tbl;
SELECT * FROM [Normalization].TimeNormalize('SELECT * FROM tbl', 0, 0,0,1,0);

This is the result (notice I’ve blurred the normalized values just because they are too many):

The two graphs are once again very different.

Long story short: just because a graph is correct don’t assume the viewer will interpret it in the right way. Sometimes it’s best to help him a little, even if that means blending information.

PS: I’ve almost forgot to mention: you will find the method (along with the stored procedure) for free in the Microsoft SQL Server To Windows Azure helper library (https://sqlservertoazure.codeplex.com/) open source library.

Happy Coding,

Francesco Cogno

Summary:

SQL Server 2014 introduces significant new features toward a deeper integration with Microsoft Azure, thus unlocking new scenarios and provide more flexibility in the database space for IaaS data models. This technical article will cover in depth SQL Server 2014 Data Files on Azure Blob storage service, starting from step-by-step configuration, then providing guidance on scenarios, benefits and limitations, best practices and lessons learned from early testing and adoption. Additionally, a fully featured example of a new (Windows Server Failover Clustering) - like feature will be introduced to demonstrate the power of Microsoft Azure and SQL Server 2014 when combined together. 

Happy coding,

Francesco Cogno

In order to inspect sqlservr.exe we will use a powerful tool called Dependency Walker. You can find it in many Microsoft distribution kits; the author, however, keeps an update page with the latest binaries: you can get them here: https://dependencywalker.com/.

Dependency walker allows you – among other things - to inspect dynamic DLL handling by user code. That is exactly what SQL Server is doing when loading your native compiled code into its own address space.

SQL Server will perform these tasks:

For each natively compiled entity. Note that is just an oversimplification meant to show what happens in that specific moment. There is a lot more involved before (Mixed Abstract Trees, Pure Imperative Trees, etc…) and afterwards of course.

First, we need to start dependency walker and ask it to open the main SQL Server executable. If you have trouble finding the right exe you can use the SQL Configuration Manager:

Once open Dependency walker will show you the module linked by the executable. You will find the familiar DLLs here:

This is, however, the static representation of sqlservr.exe. Dependency walker is able to inspect a running instance of that executable much like a debugger does. This is very helpful since we want to show what happens during a native compilation.

Select Start profile from Profile menu:

As you can see we are starting sqlservr.exe from scratch. You will need to shut down the running instance first (there cannot be two instances owning the same files) and copy the relevant parameters from the configuration manager. If you want more details on how to start SQL Server from command line please refer here: https://technet.microsoft.com/en-US/library/ms180965(v=SQL.105).aspx.

You will see the console popping out: just wait for SQL Server to finish its initialization (note: if you already have many natively compiled entities the startup will take a while; this is normal and the reason of if should become clear in a moment).

Before proceding further let’s look at the Window menu of Dependency Walker:

Notice that SQL Server has spawned another process during its startup: CONHOST.exe. DW will show the latest process spawned as default so we need to select sqlservr.exe back before proceeding further.

If you want you can clear the Dependency Walker’s log window now:

Now that our instance is running and Dependency Walker is actively monitoring it we can create a demo database on it (we can use SSMS, sqlcmd, whatever you like):

CREATE DATABASE [NativeDemo]
ON  PRIMARY 
( NAME = N'NativeDemo_data', FILENAME = N'C:\Demo\NativeDemo_data.mdf'),  
FILEGROUP [InMemoryOLTP_InMemory] CONTAINS MEMORY_OPTIMIZED_DATA  
( NAME = N'NativeDemo_memopt', FILENAME = N'C:\Demo\NativeDemo_memopt') 
LOG ON ( NAME = N'NativeDemo_log', FILENAME = N'C:\Demo\NativeDemo_log.ldf')
GO

Now we try and create a natively compiled table. Note that as soon as we start the batch Dependency Walker will start to output info (that’s what we wanted in the first place) and slow down the batch considerably:

USE [NativeDemo];
GO
CREATE TABLE [Customer](
    [PKIDBucketCount64] INT NOT NULL PRIMARY KEY NONCLUSTERED HASH WITH (BUCKET_COUNT = 64),
    [AnINTField] INT NOT NULL,
	[AVarchar200Field] VARCHAR(200) NOT NULL,
    [ANVarchar500Field] NVARCHAR(500) NOT NULL
 )
WITH (MEMORY_OPTIMIZED = ON,  DURABILITY = SCHEMA_AND_DATA  );
GO

If all went well we should have our own table. Now go back to Dependency Walker and look at the Window menu:

As we can see there are two new processes captured by Dependency Walker: CL.exe and LINK.exe. These are the compiler and the linker processes spawned by SQL Server for the native compilation of our table.

Let’s start selecting CL.exe.

If we scroll all the way up in the log we will find something like this:

Worth noting here are the command line parameters. In my case they were:

Program Arguments: /O2 /d2Zi+ /GL /Z7 /Gy- /I "C:\Program Files\Microsoft SQL Server\MSSQL12.P00\MSSQL\Binn\Xtp\gen\include" /W3 /wd4101 /wd4723 /wd4724 /wd4049 /WX /GS /fp:strict /LD "xtp_t_6_277576027.c" hkgenlib.lib hkruntime.lib hkengine.lib hkgenexp.exp hkcrt.lib hkk32.lib hkversion.obj /link /LIBPATH:"C:\Program Files\Microsoft SQL Server\MSSQL12.P00\MSSQL\Binn\Xtp\gen\lib" /LIBPATH:"C:\Program Files\Microsoft SQL Server\MSSQL12.P00\MSSQL\Binn\Xtp\VC\lib" /noentry /nodefaultlib /incremental:no /ltcg /wx /FUNCTIONPADMIN

If you have experience in C Windows coding you will recognize many of them. Just to point a few, notice the /7Z flag that helps debugging; for the performance point of view notice the /O2 flag that asks the compiler to perform code optimization favoring speed over size (see https://msdn.microsoft.com/en-us/library/8f8h5cxt.aspx) while the /fp:strict flag forces the floating point operations to favor precision over speed (see https://msdn.microsoft.com/en-us/library/e7s85ffb.aspx).

Now if we switch to SQL Server process, we should see something like this:

Notice two things here:

• SQL Server calls LoadLibrary (see https://msdn.microsoft.com/en-us/library/windows/desktop/ms684175(v=vs.85).aspx) to import in its own process the DLL just compiled.

• SQL Server calls GetProcAddress to find the address of the exported functions (see https://msdn.microsoft.com/en-us/library/windows/desktop/ms683212(v=vs.85).aspx).

Now SQL Server 2014 can call the compiled functions just pointing at the (local) address given by GetProcAddress.

Happy Coding,

Francesco Cogno

Non-blocking operators, on the contrary, can produce output before having processed all the input rows. Think about at this query:

SELECT a,b, a+b FROM Table

SQL Server can tell the exact result of a+b of each row as soon as it has read the row itself. In this case SQL Server will not wait for the whole table to be read from disk and will stream the data to the output as soon as it’s ready.

As you might have guessed by now non-blocking operators are generally better: the less bottlenecks your execution plan has the better.

The SQL Server optimizer, of course, knows when it’s best to use – and where – these operators on its own. You should however be mindful of the implications of a blocking function in your code, especially when dealing with slow-to-retrieve datasets. As a real example suppose we want to access the OpenData Lombardia and perform some statistical calculations. The data is available as an URL: https://www.dati.lombardia.it/api/views/sd8x-w4h3/rows.csv?accessType=DOWNLOAD (rows are available in other formats such as JSON and XML but for the purpose of this article we will work with CSV).

In order to show a real example I’ve created a much bigger version of the recordset (exactly 1,305,487,617 bytes) replicating the original dataset.

Using the CLR we can access the remote recordset and return as SQL NVARCHAR rows. A first blocking approach can be:

[SqlFunction(
    DataAccess = DataAccessKind.None,
    SystemDataAccess = SystemDataAccessKind.None,
    FillRowMethodName = "_StreamLine",
    IsDeterministic = false,
    IsPrecise = true,
    TableDefinition = (@"Line NVARCHAR(MAX)"))]
public static System.Collections.IEnumerable BlockingFileLine(
    SqlString fileName)
{
    List<string> lStreams = new List<string>();

    using (System.IO.FileStream fs = new System.IO.FileStream(
                    fileName.Value,
                    System.IO.FileMode.Open,
                    System.IO.FileAccess.Read,
                    System.IO.FileShare.Read))
    {

        using (System.IO.StreamReader sr = new System.IO.StreamReader(fs))
        {
            string str;
            while ((str = sr.ReadLine()) != null)
                lStreams.Add(str);
        }
    }

    return lStreams;
}

This undoubtedly works as expected generating a line-by-line TVF. However if we look at the memory footprint (I suppose a recently started instance for the sake of simplicity):

This is far from optimal. Look at the virtual_memory_committed_kb delta: it’s a LOT. SQL went from 13264 KB to 4132364 KB. It’s almost 4GB just for this function (~3.5 times the initial dataset)!

In SQL CLR functions we can – and should - use lazy evaluation instead: your main function is expected to return a System.Collections.IEnumerable instance. This is good: IEnumerable instances are not required to be able to tell the exact number of items in advance. This means that we can populate an IEnumerable while it’s being traversed. Using the streaming approach we don’t need to read all the file in advance. We can just read the relevant bits when requested.

In the previous example we returned a List (that implements IEnumerable). The list was populated in advance (hence the heavy memory usage). To use a streaming approach we need to return a custom class that implements IEnumerable directly:

[SqlFunction(
    DataAccess = DataAccessKind.None,
    SystemDataAccess = SystemDataAccessKind.None,
    FillRowMethodName = "_StreamLine",
    IsDeterministic = false,
    IsPrecise = true,
    TableDefinition = (@"Line NVARCHAR(MAX)"))]
public static System.Collections.IEnumerable StreamFileLine(
    SqlString fileName)
{
    System.IO.FileStream fs = new System.IO.FileStream(
                    fileName.Value,
                    System.IO.FileMode.Open,
                    System.IO.FileAccess.Read,
                    System.IO.FileShare.Read);

    return new LineStreamer(fs);
}

Here our LineStreamer class is just a wrapper around our real implementation:

public class LineStreamer : Streamer, System.Collections.IEnumerable
{
    public LineStreamer(System.IO.Stream stream)
        : base(stream)
    { }

    System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
    {
        return new LineStreamerEnumerator(stream);
    }
}

The real code is in LineStreamerEnumerator class. That class will implement System.Collections.IEnumerator allowing it to be traversed in a foreach loop. In order to achieve streaming we should produce a row only when explicitly requested. We should also discard any non essential information in order to minimize the memory footprint.

Here we go:

public class LineStreamerEnumerator : System.Collections.IEnumerator, IDisposable
{
    protected System.IO.StreamReader sr = null;
    protected string _CurrentLine = null;

    public LineStreamerEnumerator(System.IO.Stream s)
    {
        sr = new System.IO.StreamReader(s);
    }

    object System.Collections.IEnumerator.Current
    {
        get { return _CurrentLine; }
    }

    bool System.Collections.IEnumerator.MoveNext()
    {
        _CurrentLine = sr.ReadLine();
        return _CurrentLine != null;
    }

    void System.Collections.IEnumerator.Reset()
    {
        throw new NotImplementedException();
    }

    #region IDisposable / destructor
    void IDisposable.Dispose()
    {
        if (sr != null)
        {
            sr.Dispose();
            sr = null;
        }
    }

    // Finalizers are not supported in SQL CLR
    //~LineStreamerEnumerator()
    //{
    //    if (sr != null)
    //        sr.Dispose();
    //}
    #endregion
}

Notice that I’ve implemented the IDisposable interface explicitly. SQL Server will call Dispose on our enumerator as soon as it’s done with it. This will give us a chance to cleanup any lingering streams. We cannot use finalizers in SQLCLR as the example above illustrated.

Let’s try streaming in action:

Here the memory delta is a LOT less (107728 – 16844 = 90884 kb). We lowered our memory footprint from 4GB to 100MB.

Impressive isn’t it?

Of course this is an ad-hoc example so you should not expect such big gains in real world. It is however a good practice to avoid blocking operators whenever possible.

More on the topic:

• [MSDN] IEnumerable Interface (https://msdn.microsoft.com/library/system.collections.ienumerable(v=vs.110).aspx).

• [MSDN] IEnumerator Interface (https://msdn.microsoft.com/library/system.collections.ienumerator(v=vs.110).aspx).

• [MSDN] IDisposable Interface (https://msdn.microsoft.com/library/system.idisposable(v=vs.110).aspx).

Remember also that most .Net streams have automatic buffering so you don’t have to worry about it. This feature fits nicely in the streaming pattern; try it using the System.Net.WebRequest to understand why (or just look here if you’re lazy :)).

Happy Coding,

Francesco Cogno

here are the slides of our recent speech at SQLSaturday 264 in Ancona (https://www.sqlsaturday.com/264/eventhome.aspx).

You will also find the demo scripts. Feel free to share your thoughts.

Happy Coding,

Francesco Cogno (@MindFlavor) and Giuseppe Zagarrio (@ZagaMimmo)

[View:~/cfs-file.ashx/__key/communityserver-blogs-components-weblogfiles/00-00-00-87-37/6-degrees-of-separation.pdf:550:0]

[View:~/cfs-file.ashx/__key/communityserver-blogs-components-weblogfiles/00-00-00-87-37/6254.Demos.zip:550:0]

Ever since the MSDN post about SQL Server XI, available from SQL 2014 (see  https://blogs.msdn.com/b/igorpag/archive/2013/10/23/create-a-sql-server-2014-database-directly-on-azure-blob-storage-with-sqlxi.aspx) there is a lot of buzz about Azure Storage in the SQL Server community. One recurring question is: What is the Shared Access Signature (SAS) and why SQL Server XI needs it?

First some background: in the Azure storage platform, each blob is contained in a, er, container. Each container, in turn, is related to a specific Storage Account:

So, in order to access a blob you need to specify:

• The account.
• The container.
• The blob name itself.

So, for example, suppose you have a storage account called frcogno. In that account you have created a blob called database.mdf and placed it into the container mindflavor.

In the Azure world, this is a REST URI:

 https://frcogno.blob.core.windows.net/mindflavor/database.mdf

 Of course, unless you’ve specified the container or the blob to be publicly accessible, you cannot access it directly. The Get Blob operation (https://msdn.microsoft.com/en-us/library/windowsazure/dd179440.aspx) requires you to specify the Authorization header. Without it any access will fail. But what is required to specify the authorization header? Simply put you need the Azure Account shared key. You can find them in the portal:

As you can see the access keys are linked to your Azure Storage account:

Our picture now shows that in order to access https://frcogno.blob.core.windows.net/mindflavor/database.mdf we need the Shared Access Keys of the account frcogno. But what if we have two containers?

As you can see, knowing the Shared Access Keys of frcogno would allow you to access both containers. Aside from that, each REST API call must be independently authorized. Since the authorization involves a SHA256 algorithm it will be hard on your CPU (suppose 50K IOPS: it means up to 50K SHA256/sec!).  

In order to avoid the continuous authorization we should create a reusable signed request. In the Azure platform it’s done by using a “self-signed” URI (or, more correctly, Shared Access Signature URI).  That is, an URI that contains the authorization header once and for all.

Referring here we find exactly how to generate that URI: https://msdn.microsoft.com/en-us/library/windowsazure/dn140255.aspx; don’t worry, though, you can use the easier T-SQL method (just install the SQL Server to Windows Azure helper library available here: https://sqlservertoazure.codeplex.com/).

Using the library we can call the GenerateDirectBlobSharedAccessSignatureURI function to do the job for us:

SELECT [Azure].GenerateDirectBlobSharedAccessSignatureURI(
	'https://frcogno.blob.core.windows.net/mindflavor/database.mdf',
	'cQhAGiYCTw09dg+i4YkciTiG2g0g5Sn5AMvKD2bnG9csjxuLOf8fNq7dKoY4I48LEFLTUGDZXxNxBz0oxGSUNQ==',
	'rwd',
	'b',
	'2010-01-01', '2015-12-31');

The parameters are:

1. The URI you want to sign. In our case is 'https://frcogno.blob.core.windows.net/mindflavor/database.mdf'.
2. The Storage account shared Key.
3. The permission to grant the SAS URI (in this case we give it Read, Write, Delete. Please refer to https://msdn.microsoft.com/en-us/library/windowsazure/dn140255.aspx for the list of permissions available).
4. The type of resource. In this case we use b as Blob. If we were to grant a container it would have been c.
5. The validity interval of the SAS URI. In this case we specify from '2010-01-01' to '2015-12-31'.

The result would be a rather complex URI that will allow you to access the resource without knowing the Azure account shared key.

 Notice that in this case we are creating a SAS URI for the single blob. We could have done it at container level. In that case it would be possible to access all the blobs in that container. For example this SAS URI would allow access to any blob in the mindflavor container:

Now all you have to do is to create the credential in SQL Server:

CREATE CREDENTIAL [https://frcogno.blob.core.windows.net/mindflavor]
WITH IDENTITY='SHARED ACCESS SIGNATURE'
SECRET='sv=2012-02-12&st=2009-12-31T23:00:00Z&se=2015-12-30T23:00:00Z&sr=c&sp=rwdl&sig=OR4xtXnPXe%2BYHpZ9J5EiZ0Fm9R664wc7%2BK0RVELffIQ%3D';

And now SQL Server can manage the blobs in the mindflavor container. And, more importantly, not in any other container of the same Azure Storage Account.

There are a couple of things to note:

• The SAS URI has a validity time frame. Outside that timespan the SAS URI would expire.
• Knowing the SAS URI will allow uncontrolled access to the container. Make sure to not share it.

The second point leads to another thought: what if the SAS URI is stolen? Is there a way to revoke the permissions?

In our case we’ll have to change the Azure Account Shared key (the one we used to generate the SAS URI). That will, of course, invalidate every single SAS URI generated with it. This is far from ideal so the Azure platform has another solution for you.

You can generate a access policy of a resource and then grant access to that policy. If the SAS URI is stolen all you have to do is to revoke that single policy.

But that will be shown in another post since it requires you to play with the container ACLs. More on that later.

Happy Coding,

Francesco Cogno

as a sample of how to use Microsoft SQL Server To Windows Azure helper library let me show you how to perform a backup to azure using the TSQL. As prerequisite, of course, you should have downloaded and installed the helper assembly (for how to do that please refer to the project documentation). 

In our sample we will be using the Azure page blob. The page blob is well suited for random access allowing us to upload the backup in data chunks. It's useful since we can resend only the failing chunks instead of the whole backup. In order to use a page blob we need to define its maximum size first. In a way it acts like a sparse file: your tell the file system how big the file should be and then start putting content in it.

Of course we should put an exclusive lease on the blob. We should release the lease only when the upload procedure is complete. Failing to do so might allow readers to get a non complete copy (we all know the SQL isolation levels enough to know how bad that could be). In this sample, however, will leave that out to a later post.

So in order to upload a backup we should follow this macro-code:

1. Backup to temporary storage.
2. Create the page blob with maximum size = backup size.
3. Split the backup in pages of reasonable size (as usual, big chunks are most efficient but a failed upload would mean more data to retransmit).
4. Create a table to keep track of the uploaded chunks.
5. Upload the pages verifying the outcome.

Let's get down to the code:

Backup to temp storage

This is easy:

SET NOCOUNT ON;
BACKUP DATABASE [DemoAzureCLR] TO DISK='C:\temp\DemoAzureCLR.bak' WITH INIT, COMPRESSION;
GO

Nothing fancy here.

Create the page blob with maximum size = backup size

This is much more interesting. First use a pretty CLR function that returns us the backup size.

DECLARE @size BIGINT = [Azure].GetFileSizeBytes('C:\temp\DemoAzureCLR.bak');

 Now that we have the size we can call the correct [Azure] stored procedure: CreateOrReplacePageBlob:

-- Create the "sparse" page blob.
EXEC [Azure].CreateOrReplacePageBlob    
	'your_account',
	'your_shared_key', 1,
	'your_container',
	'DemoAzureCLR.bak',
	@size

Of course you should replace the parameters with the one of your Azure container.

Split the backup in pages of reasonable size

Now we must calculate in how many chunks we want to split the backup file. I like to upload 32KB for each call so we divide the total size by 32*1024. Note that since we round up to the nearest integer we might end up with unused space. This must be accounted during the last upload: if we upload too much data - even if zeroed - the backup would be unusable.

DECLARE @PageCount INT = CEILING(CONVERT(FLOAT, @size) / (1024.0*32.0));

Create a table to keep track of the uploaded chunks

To keep things simple we can use a simple table with an integer ID and a text field. The Chunk field identifies the chunk (page) number. Any non null string in the text field indicates that the chunk is yet to be uploaded. Since we know that the upload function will return null in case of success and a text in case of error all we have to do is to iterate between non-null rows, try to upload the corresponding chunk and then update the text field.

First the table definition:

IF EXISTS(SELECT * FROM tempdb.sys.objects WHERE name = '##progressTable')
DROP TABLE ##progressTable;

CREATE TABLE ##progressTable(ID INT IDENTITY(1,1) PRIMARY KEY CLUSTERED, Chunk INT NOT NULL, OpStatus NVARCHAR(MAX) NULL);

Then we initialize it (rember: a null value means "chunk uploaded successfully!"):

-- Prestage the progess table
DECLARE @iCnt INT = 0;
WHILE @iCnt < @pageCount BEGIN
	INSERT INTO ##progressTable(Chunk, OpStatus) VALUES(@iCnt, 'ToUpload')
	SET @iCnt += 1;
END

Upload the pages verifying the outcome

All that's left to do is, for each non null row, upload the corresponding chunk:

DECLARE @pageNum INT, @ID INT;
DECLARE @len INT;
WHILE(EXISTS(SELECT * FROM ##progressTable WHERE OpStatus IS NOT NULL))
BEGIN
	SELECT TOP 1 @pageNum = Chunk, @ID = ID FROM ##progressTable WHERE OpStatus IS NOT NULL;

	SET @len = 1024*32;
	IF ((1024*32*@pageNum)+@len)>@size 
		SET @len = @size -(1024*32*@pageNum);

	PRINT 'Uploading page number ' + CONVERT(VARCHAR, @pageNum) + '. ' +  CONVERT(VARCHAR, @len) + ' bytes to upload.';

	UPDATE ##progressTable SET OpStatus = [Azure].PutPageFunction(
		'your_account',	'your_shared_key', 1,
		'your_container',
		'DBPartition.bak',
		[Azure].GetFileBlock(
			'C:\temp\DBPartition.bak',
			1024*32*@pageNum,
			@len,
			'Read'),
		1024*32*@pageNum,
		@len,
		NULL,
		NULL,
		0,
		NULL)
	WHERE ID = @ID;
END

Note that the T-SQL UPDATE statement is very handy in this case. Note, also, that we account for the last upload: since it might not be aligned to the 32KB boundary we check to make sure to upload only the valid data.

Notice that in my sample execution the last upload should not be 32KB since the backup isn't 32KB-aligned. Our code, however, handles that:

Notice that the last chunk is smaller than the previous ones. As usual, we can query the container to find our brand new blob (using the ListBlobs TVF):

SELECT * FROM [Azure].ListBlobs(
	'your_account',
	'your_shared_key', 1,
	'your_container',
	1,1,1,1,
	NULL);

Here is our backup:

You can try to download the blob using your browser (the URI is there for a reason!) and restore it. You will find that your backup was saved to the cloud successfully.

Last but not least, even though there is no MD5 show by the ListBlobs function each upload was MD5ed separately and checked server-side by azure. So, since every chunk is checksumed the whole file is checksumed as well :). Want to know how this is done by the library? Check the open source code in Microsoft SQL Server To Windows Azure helper library project in CodePlex.

 Here is the complete script for this sample:

IF EXISTS(SELECT * FROM tempdb.sys.objects WHERE name = '##progressTable')
DROP TABLE ##progressTable;

SET NOCOUNT ON;
BACKUP DATABASE [AdventureWorks] TO DISK='C:\temp\AdventureWorks.bak' WITH INIT, COMPRESSION;
GO

DECLARE @size BIGINT = [Azure].GetFileSizeBytes('C:\temp\AdventureWorks.bak');

CREATE TABLE ##progressTable(ID INT IDENTITY(1,1) PRIMARY KEY CLUSTERED, Chunk INT NOT NULL, OpStatus NVARCHAR(MAX) NULL);

DECLARE @PageCount INT = CEILING(CONVERT(FLOAT, @size) / (1024.0*32));

PRINT CONVERT(VARCHAR, @pageCount) +' 32KB-pages to upload (total ' +  CONVERT(VARCHAR, @size) + ' bytes). Added ' + CONVERT(VARCHAR,(@PageCount*1024*32) - @size) + ' bytes as padding.';

-- Prestage the progess table
DECLARE @iCnt INT = 0;
WHILE @iCnt < @pageCount BEGIN
	INSERT INTO ##progressTable(Chunk, OpStatus) VALUES(@iCnt, 'ToUpload')
	SET @iCnt += 1;
END

-- Create the "sparse" page blob.
EXEC [Azure].CreateOrReplacePageBlob    
	'your_account',
	'your_shared_key', 1,
	'your_container',
	'AdventureWorks.bak',
	@size

PRINT '"Sparse" blob created.'

DECLARE @pageNum INT, @ID INT;
DECLARE @len INT;
WHILE(EXISTS(SELECT * FROM ##progressTable WHERE OpStatus IS NOT NULL))
BEGIN
	SELECT TOP 1 @pageNum = Chunk, @ID = ID FROM ##progressTable WHERE OpStatus IS NOT NULL;

	SET @len = 1024*32;
	IF ((1024*32*@pageNum)+@len)>@size 
		SET @len = @size -(1024*32*@pageNum);

	PRINT 'Uploading page number ' + CONVERT(VARCHAR, @pageNum) + '. ' +  CONVERT(VARCHAR, @len) + ' bytes to upload.';

	UPDATE ##progressTable SET OpStatus = [Azure].PutPageFunction(
		'your_account',	'your_shared_key', 1,
		'your_container',
		'AdventureWorks.bak',
		[Azure].GetFileBlock(
			'C:\temp\AdventureWorks.bak',
			1024*32*@pageNum,
			@len,
			'Read'),
		1024*32*@pageNum,
		@len,
		NULL,
		NULL,
		0,
		NULL)
	WHERE ID = @ID;
END

SELECT * FROM [Azure].ListBlobs(
	'your_account',
	'your_shared_key', 1,
	'your_container',
	1,1,1,1,
	NULL);

Happy Coding,

Francesco Cogno

since my last SQLRally presentation in Amsterdam people kept asking for a working build of the SQLCLR Windows Azure REST API  DLL. Many snippets of it can be found in this blog; however most of you don't really care about the implementation boilerplate. On the contrary, some of you might actually like to contribute to this library. In order to please everyone I've created - with the help of a fellow PFE Matteo Teruzzi - a GIT enabled open source project hosted on CodePlex. You can find it here (https://sqlservertoazure.codeplex.com/).

For those unfamiliar with CodePlex, just know that you can:

1. Download a precompiled DLL to import in your Database (easy). You can also download a sample script to test the library in a matter of minutes.
2. Download the source code to build your own copy (I'll explain in a moment why you should do that).
3. Just browse come class straight from your browser.
4. Post comments!
5. Create issue tasks for the community to correct.
6. Start a discussion for a feature to implement.

Right now the project needs - badly - a proper documentation (as it is now there's just an empty page) and a bunch of willful testers.

Please note that this is a pet project so I cannot guarantee any support. The full license is here (https://sqlservertoazure.codeplex.com/license); it's a the standard Mozilla Public License Version 2.0.

As I said the library can be used in its compiled form. I, however, encourage you to build you own version.

First of all you should replace the dummy snk file with your own. This is fairly straightforward, all you have to do is to right click every library and change the strong name key file:

If you don't have a strong name key file you can generate one using the sn.exe utility that ships with the Microsoft SDK. The syntax is just:

sn -k <output_file_name.snk>

Second if you want to use the [Azure.Embedded] schema you need to customize the Connections.xml file embedded into the library:

As you can see in the image above there are two account already configured (a dummy one and the simulator). All you have to do is to add a Connection section for each account you want to be able to address without specifying all the parameters in the T-SQL (as I've done in my SQLRally presentation). For example, if you have an Azure Storage account called myazurestorageaccount and one of your shared keys is Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw== you can add a section like this one:

<Connection>
  <LogicalName>azurestorage</LogicalName>
  <AccountName>myazurestorageaccount</AccountName>
  <SharedKey>Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==</SharedKey>
  <UseHTTPS>True</UseHTTPS>
</Connection>