1. A text parser that will calculate the relative weight of each word in a given text (we'll use some classical texts as source). The text parser will also expose a JSON REST interface.
2. An AngularJS app that will get the JSON data and then pass it to timdream's WordCloud component in a suitable manner.

The final result is like this:

You can find the complete source code here: https://github.com/MindFlavor/word_count_rust_angular2. Due to the nature of this blog we will discuss here the data management parts only. Our logic is pretty simple: count each occurrence of a word and use it to determine how big the word will appear in the cloud.

AngularJs 2 App

There are many open source JavaScript libraries that render word clouds, for this post we'll use timdream's one: http://timdream.org/wordcloud/. Timdream's library comes complete with Typings which is helpful working with TypeScript. This time I will skip the steps required to startup a TypeScript project in Visual Studio Code. If you need more help please refer to my previous post: How to render SQL Server acyclic blocking graphs using Visual Studio Code, TypeScript, NodeJS and TreantJS – Part 1.

Service

We will start with the service. The service is responsible to retrieve the data (an HTTP GET) and to map it to the strongly typed class. The class for our word weight is a two-liner:

Using Promises, along with the TypeScript's arrow operator, makes the code simple and easy to follow.

Component

The Angular component requests the service as prerequisite. It does it in the class decorator under the name providers:

Now just inject the mock service instead of the real one and test the app.

WordCount service

The WordCount service should simply count the occurrences of every word in a given file. This is basically the textbook example of map/reduce. To spice the things up a bit we also want to:

1. Exclude noise words (based on a dictionary).
2. Consolidate similar words (singular/plural, synonyms, etc...).

We also want to create a multithreaded word count service. If we are CPU-bound this should help to speed up the computation on multithreaded machines. This kind of programs are - this example excluded - hard to code. Fortunately there are new languages built from the ground up to tackle the concurrency problems. One I particularly like is Rust (https://www.rust-lang.org). While targeted to systems programming Rust can be successfully used for other tasks too. In our case it forces us to think about ownership of shared resources, which is good for multithreaded applications (you can find more about it in this blog post https://blog.rust-lang.org/2015/04/10/Fearless-Concurrency.html).

1. Load a file. For each word:
   1. Check if is a noise word.
   2. Lookup the most relevant synonym (if any). If found use it instead of the original word.
   3. Increase the word's count by one.
2. Present the total per word ordered by occurrence count.

If we were able to apply the serial logic in parallel we could come up with:

1. Load the file. For each word:
   1. Find an available thread and dispatch the word to it.
2. Each thread will, for each word received:
   1. Check if it is a noise word.
   2. Lookup the most relevant synonym (if any). If found use it instead of the original word.
   3. Increase the word's count by one.
3. When all the words are dispatched and processed, present the total per word ordered by occurrences.

The data race

The parallel algorithm is good in theory but poses some immediate questions: "How do I dispatch a word to a thread?" or "How can a thread know when there are no more words to process?".

The biggest problem, however, lies in the data race between threads. The step Increase the word's count by one will be performed by multiple threads in parallel and, unless we mediate the access to the counter somehow, problems will arise. This is nasty because, in most languages, your code will compile and run even if you were modifying the same object from multiple threads. The result can be a segfault if you're lucky or, way worse, miscalculations.

Bottom line you're own your own. If you are a superstar coder you will prevent problems and end up with an efficient concurrent algorithm. But if you're a normal person like me, chances are you will end up scratching your head at the pseudo random errors.

The good news is, with Rust, you cannot forget about such things. Rust will force you to prevent data races or your code won't even compile.

Even the apparent innocuous concurrent access to a shared vector should be safe. With Rust it must be safe (thankfully). Let's see how.

Concurrent access to a shared resource

Let's try to share an immutable vector across threads. A first approach could be:

Transferring ownership

Rust's new error system will suggest you to move (transfer) the ownership of the vector to the thread. It makes sense: if work relinquishes the ownership of v to the thread it will no longer deallocate v. v would get deallocated at the end of the thread (as it should).

Let's try it (notice the move keyword before the closure):

The error this time is harder to interpret. The hint is in the note: v does not implement `Copy` trait. Why Rust tries to copy our vector instead of simply transferring ownership? Because we are transferring the ownership of one vector to three threads. Since it's impossible Rust tries to send three separate copies of our vector to our three threads. But since by default our structures cannot be copied it doesn't work (luckily because it's not something we wanted to do in the first place).

Go on and try to remove the for loop and spawn a single thread instead of three. In that case the move will work.

Reference-counted pointers

Clearly moving the ownership of the vector to our threads cannot work without copying it for each thread. This solution is a waste of memory since we know our vector to be immutable. We should be able to access it concurrently without mutexes of any kind. But without copies we cannot move the vector and without move the thread will outlast the vector.

So, how do we solve the lifespan problem?

One answer is to use reference counted pointers. These wrappers will keep count of how many references are present at any time and only deallocate the inner object when nobody else is referencing it. It's roughly what a GC-based language would do. Only here we get deterministic deallocation (and also the circular reference problem).

In our case it means the inner object will be referenced by the work function and by each thread. So there are 4 references (1 for work + 1 for each thread). When work terminates the reference count will simply drop to 3. And the vector won't be deallocated (yet).

You can read more about it here: https://doc.rust-lang.org/std/sync/struct.Arc.html and here https://doc.rust-lang.org/book/concurrency.html.

That's our final, working, code:

Channels

Putting all together

Now that we are able to share safely an immutable structure between threads and to send ownership using channels we can implement our concurrent word counter method:

This example of parallel map => parallel reduce => serial reduce is surprisingly fast and, last but not least, cross platform.

I'm sure better performance can be achieved further optimizing the code but I'll leave it up to you :).

In the previous blog post (you can find it here: http://bit.ly/bcts1) we've seen how to setup a NodeJs REST application in Visual Studio Code. We've also seen how easy is to debug the application without leaving the IDE. In this second part, we will see how to wire some data to a gorgeous library: TreantJS. In our example, we wanted to be able to graphically visualize the SQL Server acyclic blocking graph. Let's review the server code:

As you can see we are returning a strongly-typed array - in fact a promise to be fulfilled. We use an interface in order to render unit testing using mock services easier.

Next step

What we need now is to create an html page that will call our web service, do some data manipulation and then render the graph. We want to do it in TypeScript (obviously) and we want to use Visual Studio Code.

Debugger for Chrome

There are many techniques for debugging a TypeScript transpiled code. Visual Studio Code, using the Debugger for Chrome plugin, has first class support to web debugging. You can set breakpoints in the code and, upon hit, the browser will pause and surrender control to Visual Studio Code. From there, you will be able to inspect the variables, call stack, execute a step over, step into, and so on.

Dual debugging

We want to be able to debug the NodeJs server and the Chrome webapp so we can follow the data from end to end. But how to debug a NodeJs server and a Chrome webapp at the same time in the same Visual Studio Code solution? Ideally we want to setup two different Visual Studio Code solutions, one for the NodeJs server and another for the webapp. But the webpage will be sent by the server (so the code must be there), not by the webapp solution. There are many ways to solve this chicken-and-egg solution. I usually end up using symbolic links but any vendoring technique should be fine.

In my case I will have the final web folder in the NodeJs solution (to be exported as static files by express). I will not, however, use it from there. I will, instead, create a new solution specific for the webapp and link the web folder from the NodeJs solution. This way I can work on my webapp solution and publish only the transpiled files (which I would want to do anyway). I'll be able to debug the webapp from my source TypeScript files since I'll configure the source maps. The downside of this technique is that is very git unfriendly. You can solve this limitation with git submodules but it's beyond the scope of this article (for more info, see here:https://git-scm.com/docs/git-submodule). This is how the webapp folder appears in my solution. Notice I've also linked the typings folder so the code will compile successfully in both solutions.

Once set - I will skip the details now - you will be able to hit breakpoints in Chrome even if the source code comes from another solution. Here I show you how the debugging works.

TypeScript and browsers

One of the main problems when working with browsers instead of NodeJs is the complexity of module loading. Fortunately there are many high-quality module loaders. I use SystemJs (https://github.com/systemjs/systemjs) because I find it clear. Your mileage may vary, pick the one that better suits your needs. Here is a very simple SystemJs loader (extracted from index.html):

Mapping JSON to strongly typed entries

Our client webapp, however, receives the JSON which is not strongly typed. In order to get the WhoEntry class back all we have to do is to map the JSON to our class:

TreantJS

Root blocking SPIDs

Notice in the code we relaxed the assumption of having only one root blocking SPID in a given scenario.

Recursively add blocked SPIDs

The recursive function can be functional - better - or purely side effecting like this one:

TreantJS again

Final result

Let's try our solution. Fire the NodeJs server (remember to configure the SQL Server connection properties or use the mock service) and open a browser pointing to http://localhost:3000. You should see the blocking graph (or nothing if you don't have any blocking, play with TRANSACTIONs to create one!).

For example, suppose to have this output from sp_who:

Our code will create this JSON:

TreantJS ignores the extra fields, like whoEntry in our case. We added it to make the recursive function easier.

This JSON, given to TreantJS with a very basic CSS, gives us the expected result:

As you can see from the picture above, someone (SPID 54) left a transaction open! All in all, the sloc is minimal and the result is useful. If you export the connection string you will have a handy tool to impress fellow DBAs: all you need is a NodeJs installation. Even your laptop can do that!

SQL Server uses blocking - among other technologies - to ensure ACID transactions. There are a lot of lock types in SQL Server, ranging from Shared and Exclusive, Page and Range or even table level. You can find more about these here: [TechNet] - Lock Modes. Whenever SQL server tries to acquire a lock on a resource and that resource is already owned by another, incompatible lock, blocking occurs. Depending on how long - and how often - these blocks persist the execution slowdown becomes noticeable. We will use this premise in order to show how to create a simple SQL Server-backed web site using Visual Studio Code, NodeJS and TypeScript. Since the post is very long I'll split it into two separate ones. In the first part we'll setup Visual Studio code for TypeScript + NodeJs debugging. We will also create a skeleton web service using express and mssql packages. In the second part we will create a TrantJs page rendering the data provided by the REST API just created. You can find the following post here: http://bit.ly/bcts2.

How to inspect blocking

There are many ways to find out the SPIDs (process IDs) involved in blocking. For the purpose of this article, we will focus on the sp_who stored procedure. This little helper will give back the current list of processes active in SQL Server along with the blocking process, if any. You can find more details here: https://msdn.microsoft.com/en-us/library/ms174313.aspx.

The column blk is either zero - no blocking - or the SPID blocking the session. In the example we see the SPID 54 blocked by 55. The SPID 54 blocks SPID 56 and SPID 62 in turn. The SPID 56 blocks SPID 60. These blocked processes are a directed acyclic graph (if it were a cyclic graph we would have had a deadlock instead).

Suppose now you are a DBA tasked on killing the offending SPID. But you cannot kill SPIDs at random: for example, killing SPID 56 is not guaranteed to resolve the problem because the SPID 60 might end up blocked by SPID 55. So, in general, you want to kill the root SPID - that is, the one blocking one or more SPIDs without being blocked by someone else. In our example the root SPID is SPID 55. It's easy to find the root in our graph but it's hard to do with the sp_who output because we will have to re-create the graph in our heads first (and when there are hundreds of blocked processes the task becomes daunting).

Architecture

For our tool we will use many open source technologies. We will use NodeJs with Express for the backend and TreantJs for the graph.

Setup

npm and TypeScript

Generally speaking setting up a TypeScript project is cumbersome - especially if you are spoiled like me by Visual Studio. You need to have Node and npm installed (if you don't, go here). First create a working folder, go in it and initialize the project:

Transpilation and debugging

outDir

sourceMap

The sourceMap field takes care of that. emitDecoratorMetadata and experimentalDecorators are not strictly needed in our project but are useful and required if you plan to use metadata-heavy frameworks (such as Angular 2).

We also tell the TypeScript compiler (tsc) to avoid transpiling the node_modules and the typings module files. Now let's add a dummy app.ts file with this code:

Task runner for building the solution

Debugging

program

sourceMaps

outDir

Install required packages

Also install the typings in order to exploit TypeScript's strong type checking:

.gitignore

I've also excluded the output folder of the transpilation (dist), the typings folder and the .vscode specific folder.

Create a dummy program

Let's create a dummy program to test that everything is in place. We will create a web server that responds to the URL /sp_who sending back the result of the stored procedure as JSON:

The result will be browsable from http://localhost:3000/spwho:

Hide noisy folders from Visual Studio Code

Our solution is cluttered with many noisy folders. We have the  node_modules , the  typings  folders which are automatically handled by npm and typings, respectively. We do not need to concern with them so we want to hide them from the Visual Studio Code folder view. All we have to do is to add the  "files.exclude"  property to the workspace settings:

This will leave less clutter in the Visual Studio Code folder tree allowing you to focus on your code.

Moving on

• A functional, unauthenticated web server serving static pages.
• A functional, unauthenticated web server serving REST verbs in JSON.
• A way to interact with SQL Server.
• We are able to debug step by step in our TypeScript source code, directly in Visual Studio Code.

All we have to do is to render the data graphically. We need to serve some static JavaScript that will interact with our REST API service. We will use AJAX for that. However, this is the topic of the following blog post (http://bit.ly/bcts2).

Intro

Spark

HDInsight

File upload

Zeppelin

CSV load

From CSV to case class

1. SensorId
2. An event time
3. Some kind of float value
4. A status constant

Graphs in Zeppelin

Dimension file

BI

Stacked bars

Execution plan

Getting started with Azure Data Lake store

Open Data Lombardia

AzureRmDataLakeStoreItem -Account <account> -Path <local_path> -Destination <remote_path>

First job

1. The Csv and Tsv SerDe. In U-SQL terminology we call them Extractors and Outputters. So we extract from extractors and output to outputters.
2. We can specify the data type directly in the extractor. This is great: we don't have to parse the type ourselves as long as its string representation is parsable in the specified type. In case of problems we can instruct Azure Data Lake to either throw an error - which we can debug directly in VS - or to ignore the offending row. Sometimes, especially when munching trillions of rows for statistical analysis, handling a mis-formatted row is simply not worth the effort.

Aggregate, enter BI

An ORDER BY clause used outside of an OUTPUT statement must have a FETCH subclause..

Join to denormalize

e.SensorId == s.SensorId

Insight!

show me the temperature evolution of one specific sensor: the one in Lambrate, Milan.

Going on

Azure Event Hub

Microsoft Azure offers a very appealing IoT service. It's called Event Hubs (https://azure.microsoft.com/en-us/services/event-hubs/). Basically it gives you and endpoint in which you can send your events. The events are stored for a retention period and can be retrieved in a FIFO manner by consumers. Each consumer, optionally, can have its own "location" so you can consume your events at different pace. The event hub, therefore, is perfect for the IoT devices as it will unburden them of the store requirement: just collect your data and send it to your event hub for later processing.

The following image depicts the architecture. As you can see the Event Hub is a declination of the Service Bus Azure service.

Rust

Rust is a - relatively - new system language that allows you to create fast applications that compile with minimal (or no) runtime. It doesn't have a garbage collector so you don't have the processing hiccups and the memory bloat typical of GCs languages. What makes it so special, compared to C/C++, is a series of features (lifetimes, borrow checker, default immutability and so on) created to ensure thread safety and memory safety. Last but not least the community is great and helpful regardless of your programming skill. You can get everything you need to start programming in Rust from here: https://www.rust-lang.org/ so don't take my word for it and start playing with Rust: it will be a rewarding experience.

Personal considerations aside, you can see how Rust is a good candidate for IoT devices: low processing power and scarce memory are constraints better handled by system languages. There are excellent blog posts on how to cross-compile Rust code to work with various IoT devices so I will not cover that point. There are also many libraries already built enabling you to interact with low level hardware (without having to ASM yourself) so you can start collecting data from your sensors with ease.

But what about sending it to Azure Event Hub?

REST

As there is no official Azure SDK for Rust yet you might wonder how to access the great functionalities without resorting to external languages (i.e. Python or C#). It turns out the Azure platform exposes most functionalities as REST endpoints. So it's just a matter to post to the right URL. Here is the documentation: https://msdn.microsoft.com/en-us/library/azure/dn790664.aspx.

The body will contain the event data.

With Rust is fairly easy to perform a POST request. For this example I will be using the Hyper crate. The only difficulty is how to encode the Authorization header: without it Azure will reject our events with 401 Unauthorized. The header requires a SAS token (SAS being Shared Access Signature) in a specific format.

SAS Token

This post helps us to build our SAS token: Shared Access Signature Authentication with Service Bus. Long story short we need to encode the URL and the Expiry attribute with the HMAC SHA256 based on our shared key. The server will perform the same operation (it knows the URL and the expiry attribute because we will pass them in the request) with the same key (shared) and if the result matches it will allow the request.

RUST's way to Azure Event Hub

Let's start with the transposition of above diagram. Here I will show extracts of the code. You can of course grab the complete project from GitHub (https://github.com/MindFlavor/pusheventhub) and compile try it yourself.

generate_signature

submit_event

Worth noting here is the event_body parameter: it is a tuple with a Reader and the expected length. You can use a struct here - it would be better - I used the tuple because I'm lazy . Also to note the expected HTTP Result code: 201 (Created). Anything else we treat as failure and send back to the caller in the AzureError enum (not shown here because boring, please refer the GitHub project for it at azure_error.rs).

Conclusion

I hope this will get started with Rust and Azure Event Hub. These two exciting technologies will allow you to start playing with the IoT world with minimum investment. But, hey, now that I've got my device data in the Event Hub what can I do? This is something for another blog post; in the meantime, check out a previous post of mine that explains how to consume Event Hub events, analyse them in a streaming manner using Stream Analytics and to generate intelligence with PowerBI: SQL Server intrusion detection using Azure Event Hub, Azure Stream Analytics and PowerBI.

Always encrypted feature promises you to protect your data from prying eyes and does that simply by storing the encrypted form only. SQL Server is totally unaware of the plain text data so nobody, not even sysadmins, can read your data. This is good because this separates the duties from sysadmin - who handles the SQL instance - and data admin - who handles the data consistency.

You might wonder: if the data is always encrypted, how can we get the unencrypted form? Must we write the cryptographic code ourselves? The magic here is done by the Enhanced ADO.net library (from .Net 4.6 on). The library will handle the decryption/encryption automagically without special code: we will create our queries without worrying about encryption and the library will take care of everything. We don't even need to be aware of encrypted columns since the library discovers them at runtime. Isn't it cool? Let's see it in action.

For my example we are about to create very simple weight log table:

Of course your thumbprint will be different. Note that we can specify a custom key store provider (more on this here: https://msdn.microsoft.com/en-us/library/mt147923.aspx): here we are accepting the windows certificate store. Also note that we are not storing the certificate in SQL Server, we are merely storing its address in the certificate store.

After we have created the column master key definition we must create the Column encryption key:

This will generate something like this:

This table is virtually the same as before but here we are specyfing the COLUMN_ENCRYPTION_KEY and the ENCRYPTION_TYPE. COLUMN_ENCRYPTION_KEY specifies the encryption key (so, yes, you can pick a different key for each field) and ENCRYPTION_TYPE specifies whether the encryption will be deterministic or not.

Deterministic or not?

Deterministic encryption will give the same ciphertext from the same data. So, for example, if we encrypt "OliveOil" once and we get the encrypted string "Bassaeedd33543aaa" we will always get "Bassaeedd33543aaa" for "OliveOil". This is good because we can write a query like this: SELECT * FROM encTable WHERE food="Bassaeedd33543aaa" and benefit from index seeks. It's bad because we could infer from other rows the unencrypted data. So if security is paramount we must use the randomized - ie non-deterministic - encryption and filter our data client-side.

We cannot insert rows using SSMS with a simple:

Some restrictions apply (https://msdn.microsoft.com/en-us/library/mt163865.aspx). The most notable is the need to use parametrized queries... but you are not using concatenated tsql anyway, right ;)?

As you can see this code is fairly standard: the library will inspect the table (using sys.sp_describe_parameter_encryption system SP) and if it finds encrypted fields will look for the relative certificate, encrypt the value in the parameter and send the ciphertext. If you are worried about the performance impact of this operation you might want to fine tune its behaviour using the relevant SqlCommand overload (see https://msdn.microsoft.com/en-us/library/mt147923.aspx).

The full program code is in this gist. You can clone, compile and execute with: 1. clone with: git clone https://gist.github.com/MindFlavor/0a5302058a19ddc99105 2. Compile it: csc .a5302058a19ddc99105\Program.cs 3. Execute it (pass your SQL instance as parameter): Program.exe <your_instance_here>

It will go like this:

If you get:

it's because you forgot to import the certificate in the local machine's certificate store (or in current user, depending of the configuration). Also note that you need the private key so if you have imported the public key only you will get something like this:

Are you curious on what a sysadmin will find on SQL Server? Let's try it out:

Now my weight log is safely encrypted. Bear in mind that sysadmin can always modify the data so don't forget to audit the table access. Audit should always complement other security features.

Happy coding, Francesco Cogno

In this post I will show you how to use the concurrent features of the GO (ie GOlang) language in order to perform some maintenance tasks in parallel. This is just a showcase of features so don't expect a enteprise-ready functionality . The problem we address in this case is the repetitive task of checking our databases for consistency. Internet is full of great quality scripts to address the issue. I will only show an alternative way of doing it as an excuse to show how easy is to reach SQL Server from GO and how to use its excellent concurrent features.

First of all, you need to install GO in your environment. Please go here https://golang.org/ and install the required binaries. Next you need to set one very important environmental variable called GOPATH: it will store all your source code (well, that's note exactly true but it's enough for here). Select a convenient folder for it (for example C:\dev\go), open the system settings panel and select Advanced system settings:

From here, click Environmental variables:

And add your GOPATH variable, either to your account (better) or system-wide.

To check your configuration just open a command prompt and issue go version. If everything is ok you should see the version installed:

In order to connect to SQL Server there are many options available. The most straightforward way is to use the ODBC driver, but for the sake of this post we will use the excellent open source implementation by denisenkom. You can find it here: https://github.com/denisenkom/go-mssqldb. Wait, you don't need to clone his repository yourself. All you have to do is issue go get "github.com/denisenkom/go-mssqldb" and the go runtime will clone the package for you and install it in your GOPATH folder.  GOlang drivers allow you to code without tying yourself to a specific implementation. You can switch to another driver simply changing the driver signature. Please note that Microsoft does not support these open source drivers: you will need to contact the authors for support.

Open connection

Let's see how it's done:

Query preparation

Once we have a valid connection (ie without err) you must prepare your statement using the Prepare method of the receiver *sql.DB:

Query execution

Now we can call the Query() method. This method is peculiar because can accept and arbitrary number of parameters. As you might have guessed, these are the query parameters defined before. So in the case of SELECT database_id, name FROM sys.databases (no parameters) all we have to do is:

Data retrieval

This function returns the row collection as first parameter. All we have to do is to enumerate it and extract the data calling the Scan method of *sql.Rows:

Note: DBCC CHECKDB WITH TABLERESULTS is undocumented, unsupported and different between SQL Server versions. It's shown here just as an example.

The Scan code becomes:

Which will give you different .

So in order to tell this difference you have to pass to Scan a pointer to a pointer to a string (that is *(*string)) so in case of NULL the pointer will point to nil, in case of an empty string the pointer will point to "". Don't worry, you cannot forget this because you will get a runtime error if your pass a simple pointer to string (that is *string).

You might have spotted in the previous snippets the defer statement. It basically tells the compiler to call your function as soon as the exit of the scope. The defer is smart enough to be LIFO so if you do:

1. open DB
2. defer close DB
3. open query
4. defer close query
5. open row
6. defer close row

You are actually telling this:

1. open DB
2. open query
3. open row
4. close row
5. close query
6. close DB

That is what we want. The defer statement is handy because allows you to logically link the open and close operations without scattering them in your code in reverse order.

This special statement starts a concurrent function. The function can be anything and will be scheduled by the runtime without you having to do it manually. This is excellent for long running blocking operations such as database calls. So suppose we have our long running function; we can do:

There are may ways to do that, we can create mutexes (ok, but hard to maintain), shared objects (don't do that! ) or use channels.

The cEnd := make(chan int) creates the channel (empty). The function will add an item when it's done (the syntax is cEnd <- <item>). Notice how the go func can access the channel even if it's declared outside its scope and not passed as parameter. This is possible since this functions are closures.  Now from outside we just need to get from the channel a number of items equal the number of go funcs started:

What we do is: 1. For each command line parameter start a go func 2. For each go func: 3. Open the connection 4. Get the database list 5. For each database: 6. Perform DBCC CHECKDB

This will execute steps 3-6 in parallel (that's what we wanted in the first place).

This is a sample execution with 3 servers as parameter:

Notice how the different executions are interleaved. Each server processes the entries without waiting for the others (as expected).

Going on

Well as you have guessed we still lack the proper interpretation of the CheckDB response. Right now we just tell the user that the CheckDB has started and has completed but we should also tell him if his databases are corrupt or not . I'll leave to you this task with a hint. There are two very useful packages you might want to import: strings and regexp.

Happy coding, Francesco Cogno

A couple of days ago I was asked this question:

As a DBA, what can Stream Analytics do for my on-premise SQL Server?

The answer is easy to come by: to handle the ERRORLOG. SQL Server has its own streaming log: it's called ERRORLOG (yes, the capitalization is intended) and gives us a cartload of information. DBAs look for it when asked questions like: "what the heck happened? ". Since SQL Server 2005 it's also the standard output of the login auditing feature https://technet.microsoft.com/en-us/library/ms175850.aspx. In a nutshell, SQL Server logs entries like

The problem is, in order to detect malicious attacks we should be able to tolerate scattered failures. We don't want to raise an alarm if a users mistypes a password once in a month. In other word, we need to detect failed login attempts in a custom timespan. If we detect, for example, 40 failed logins from a specific IP within 5 minutes we should definitely raise an alarm. Another example is to detect whenever there are 5 failed logins for the same account in 1 minute. For the purpose of this post we will implement the latter example; I'll leave up to you to modify it in order to detect IP-based attacks.

What we need to do

We need to:

1. Scan the ERRORLOG file and extract login entries.
2. Store the entries in a persistent, scalable event store.
3. Group those entries using a tumbling window of 1 minute. Pick only the failed logins and group them by login name.
4. Identify the entries whose count is greater than 5.
5. Create a dynamic dashboard to show the alert.

Azure to the rescue

Luckily most tasks are addressed by a specific Azure feature. We will use, respectively:

1. Some custom C# code (will show in a bit).
2. Azure Event Hub (http://azure.microsoft.com/en-us/services/event-hubs/).
3. Azure Stream Analytics (http://azure.microsoft.com/en-us/services/stream-analytics/).
4. Azure Stream Analytics.
5. PowerBI (https://powerbi.microsoft.com/).

So what we'll end up having is:

Let's start creating the event hub collector.

Event hub

Create the event hub using the New... button in the Azure portal:

Make sure to create a consumer group for your event hub:

Create the event hub policy (it will allow our client to send data):

The dashboard should now contain the connection string. Copy it for later.

C# code

I will not dwell long here on the ERRORLOG scanner code. You can get it from github here https://github.com/MindFlavor/StreamFromErrorlog. In order for it to work on your environment make sure to configure the App.config file.

I will only highlight how to connect to Event Hub.

First you need an instance of EventHubClient class:

_eventHub = EventHubClient.CreateFromConnectionString(connectionString, eventHubName);

Here the connectionString is the one you got from the Azure portal before. The eventHubName is, well, the event hub name .

Now to send an event you just have to call the Send method. I've picked JSON as message format:

string ser = JsonConvert.SerializeObject(le);

EventData ed = new EventData(Encoding.UTF8.GetBytes(ser))
{
    PartitionKey = le.Client
};

Very easy. In case you're wondering, the PartitionKey is just a way to spread your messages in different partitions. The Azure framework will dynamically change the partitioning function based of the number of available partitions.

Stream Analytics

Now get back to the portal and create a new Stream Analytic job:

Input

Now add your event hub input:

You can download some sample data if you want to check it. Make sure to grab some as it will help during the query design phase:

Query

Remember we wanted to find the failed logins attempt on the same username in a 5 minute timespan. We just have to use the Tumbling Window function like this:

SELECT
    System.Timestamp as WindowEnd,
    login, COUNT(*)
INTO
    [elpowerbi]
FROM
    [errorloginput]
TIMESTAMP BY 
    EventTime
WHERE
    [failed] = 1
GROUP BY TUMBLINGWINDOW(mi, 5), login
HAVING COUNT(*) > 3

Make sure to specify in the FROM clause the input name you've chosen in the previous step. The INTO clause must match the output name (we haven't created it yet).

If you click Test and supply the sample data you can make sure the query works as expected:

As you can see, in my sample data there are 4 failed attempt to authenticate as username crack.

Output

Add now an output and pick PowerBI as destination. Remember to name it as the INTO clause of your query. The dataset name will be the dataset created in PowerBI. Be warned: if a dataset with the same name exists in your PowerBI subscription your existing data will be lost.

When done start your stream analytics job. If everything goes as expected you will see the status of the job going to running (or idle when the data has been processed):

PowerBI

The last task is to create the dashboard in PowerBI. You will find there the newly created dataset (note: if there's no data there will be no dataset ):

Click on it and create some graphical representation of your liking. In my case I've created this dashboard:

Here I can clearly see that I had three different account failing to authenticate: crack, kkk and NT AUTHORITY\SYSTEM. This is just what we wanted. But! More magic! The PowerBI dashboard is completely automatic! In few moments the graphs updated themselves into this:

I can see in a glance that:

• The kkk failed logins stopped happening. Since it happened just a few times I can infer it was a simply human error.
• The crack failed logins keep happening at the same rate. This smells as machine generated and could be a brute force attack indeed !.

Conclusion

This is just a sample but you can grasp the power of this solution. With just a simple query you can extract streaming information and visualize it with a dynamic dashboard. You can collect all your SQL Servers logs in the event hub and create an enterprise wide dashboard (with drill down, too). The events will shape themselves and animate your dashboard without having to write a single line of code.

Azure magic!

First and foremost, remember this is just a preview feature so it could change in the future.

Now, what is row-level security? Basically it’s an application transparent custom row filter.

In other words you now have the power to decide which row to show right after it has been fetched from a table. 

Let me show you an example. You have a table with 5 rows, as shown below:

Up until now, depending of your permission set, you are either query the table or you don’t. In other words, the recordset composition is entirely dependent of the search predicate. Since we don’t have WHERE arguments all the rows are returned. Using row-level security we can choose which row to return adding a custom filter to every query performed against our table. So, for our example, we could filter the “private” rows returning only the public rows:

The other rows are still in the database, they're just filtered.

All you have to do is provide SQL Database with a function SQL will call for each row fetched from the table. If your function returns a value SQL will not filter the row. If your function returns an empty recordset the row will be filtered out.

Simple isn't it?

In truth you have to perform two steps. Beside our filter function you have to tell SQL Database which table to filter. This is accomplished with a new statement: CREATE SECURITY POLICY. Given that security policy, SQL is able to call your function for each access to the specific table automatically.

Here https://msdn.microsoft.com/library/7221fa4e-ca4a-4d5c-9f93-1b8a4af7b9e8.aspx you can find some simple examples to get started.

Note however some important details: these filters are in query phase only. There is nothing preventing you to insert rows you would not be able to query afterwards.

From now on I'll use another, more realistic, example (full source code is available here: https://gist.github.com/MindFlavor/61bf05e357cddbeffb02). Suppose we have a table with sensitive information. Each row has a clearance level needed to retrieve it. Your users have a specific clearance level. You want to let each user retrieve only the rows he’s allowed to.

So we have a table specifying the username and the granted security level (let's call it [Sec].[securitylevel]):

(here GrantedLevel specifies the “security clearance” for the user). And a table like this:

(Here RequiredLevel is the minimum “security clearance” needed to fetch the row).

We have to create our custom function. Note that it must be an inline table value function. You cannot use multi statement table value functions here. This is good performance-wise. Our function should receive the required clearance level in input and, checking the current user, tell SQL if it should return the row examined.

This is easy. All we have to do is to query our [Sec].[securitylevel]  table to see if the currently context user has the required security level:

Note how we linked the function to our table. Also note that the function will receive the RequiredLevel field for each row to evaluate. Think about it as an implicit CROSS APPLY function.

If we query the table (as dbo):

 

We receive 10 rows, 1 for each security clearance.

Now look at the execution plan: instead of a simple clustered index scan we find: 

The join operator is our inline function predicate:

So you can clearly see how SQL Database injected our filtering function in the statement above.

Now if we were to create 3 users:

1. VIP user with clearance level 10.
2. AverageGuy with clearance level 5.
3. PowerlessGuy with clearance level 1.

We could test our filtering function just changing execution context. For example PowerlessGuy will only see 1 row:

While AverageGuy will get 5 rows. Note how COUNT works consistently:

 Moreover, since SQL injects our filtering function down in the execution tree, join expressions get filtered too as expected:

 

With this execution plan:

 

Happy coding,

Francesco Cogno