As part of the SQL Server 2012 launch, I've contributed a presentation on StreamInsight 2.0, which is available either on TechNet or the Microsoft Virtual Academy.

You can get the code associated with the video here code.

Enjoy!
-Isaac

Although I’m guessing that people who really are into maps have a slightly higher view of Mercator than he implies.

Cheers,
-Isaac

I just wrapped up my talk the 2011 PASS Summit. Here are a few quick thoughts:

StreamInsight seems like a technology that could benefit a large number of customers, for a large number of scenarios. But relatively few people even know what SI can do. So although the title of my talks was, “What’s new in Microsoft StreamInsight?”, I turned it into more of a dev-ish, “What the Heck is StreamInsight?” This seemed successful: people seemed attentive; I got a lot of good questions; very few people waked out—all good signs. I’ll see what the session feedback looks like, and if you have any comments, please paste them here or shoot me an email. I’ll be talking again at Connections, and want to use all the feedback I can to refine that talk.

I promised folks at the talk that I’d post my code. I’ve cleaned it up a little—added comments and some copyright info to save myself from our legal department—but it really is the code I used during the talk. Hopefully both of these will show up with the talk on the PASS website, but you can also download my code here.

One final thought: I’ve underestimated how hard it would be to get back in the blogging groove. But I’ll keep trying.

Cheers,
-Isaac

To answer one question he asks in his post: StreamInsight doesn’t require SQL Server Compact, but if you use it for you use it for your metadata provider, then the server metadata (applications, queries, and the like) will be persisted across restarts. This is required if you want to use resiliency for fairly obvious reasons.

Cheers,
-Isaac

As I mentioned some a few posts ago, I recently made a move to the StreamInsight team. Well, we’ve just released the new version, StreamInsight 1.2, so it’s time to start talking about what we’ve been up to.

One of the big features in the new version is resiliency, which helps you build highly available systems that use StreamInsight. This post gives a high-level, no-code overview of what this functionality does and how it can be used. More detailed posts will follow.

To motivate the feature, consider a few use cases:

1. You have a query that consumes sensor data—say the temperature of a machine—and computes the average temperature over the last day. It’s not the end of the world if StreamInsight were to miss a few minutes of data: the missed events will likely have little impact on the overall average. But if these missed events were caused by an outage, you do care (a) that your app comes back up quickly, (b) that you don’t lose the substantial state that was built up for the aggregation, and (c) that the results you get when SI comes back up are close to what you would have gotten had SI not gone down.

2. You have a query that consumes Twitter data and sends you email every time someone mentions Justin Bieber. For this query, you have the same requirement (a) for speedy recovery as above, but (b) and (c) are no longer sufficient: you really don’t want to miss an important alert should StreamInsight go down. On the other hand, you don't mind getting duplicate alerts. I.e., you want the same events you would have gotten without the outage—perhaps with some duplicates.

3. You have a query that consumes stock price data, and produces trade events based on some secret algorithm. Now you have all of the requirements from the Bieber-bot—fast recovery, equivalent events—but now it could be catastrophic if you got duplicate trades. I.e., you want completely equivalent outputs whether or not there was an outage.

The form of resiliency introduced in StreamInsight 1.2 allows SI to take and restore checkpoints. A checkpoint is a serialized form of all of the internal state SI keeps for a query. When SI restarts, it will automatically restart a previously-running query and restore its state from the checkpoint, effectively putting the query at a known point when the checkpoint was taken.

You can control when checkpoints are taken: take them more frequently and you’ll lose less in a crash; take them less frequently and you’ll keep the impact on the system low. Let’s consider how this would be used in our first use case:

1. Your query is happily chugging along, and you’re asking the system to take a checkpoint every five minutes. At some time—say 12:59—some doofus trips on the power cord of the machine and brings it down. Realizing what he’s done, he plugs it back in and restarts everything. At 1:05 everything’s ready and he restarts StreamInsight, which reads back the latest checkpoint from 12:55 and is ready to start accepting events again at 1:07.
   
   In this case, SI was down from 12:59 to 1:07, and all events that should have been received during this time were lost. In addition, all events received between 12:55 and 12:59 were effectively lost as well, since they weren’t recorded in the last checkpoint before the system went down. These lost input events may cause missed output events, and can also affect aggregate results that overlap the outage. Still, you’re happy because the system is back up, you’ve only lost 12 minutes of data, and your long-term aggregates are roughly intact.

Perhaps all of this is fine for a silly temperature reading, but what if someone were to tweet a Bieber sighting during the outage? To avoid missing this critical information, StreamInsight needs some help: something needs to keep track of the events that occurred during the outage, as well as those that occurred before the outage but after the last checkpoint, and be ready to send them along when SI comes back up.

That something is the input adapter—or more properly, whatever the input adapter is connected to. In other words, there has to be a component sitting in front of StreamInsight that keeps track of recent events and can replay them for SI. And this component has to be independent and highly available as well.

If we have this in place, we’re ready to tackle our second use case:

2. Your query is happily Biebering along, taking checkpoints every minute, but you have a historian sitting in front of your query. This historian is pulling tweets from Twitter and persisting them before sending them on to StreamInsight. As before, our doofus trips over the cable at 12:59 and restarts StreamInsight at 1:05. But now as part of its recovery routine, SI will ask its input adapter to replay the event it’s missed and the historian will guarantee that SI will see all that it has missed since 12:55 when it took its last checkpoint.
   
   And this is great: StreamInsight came back, and you haven’t missed a thing. But you notice something funny: the report of Justin’s new coif that issued at 12:57 came through twice. Once before the outage, and once after. Why? Because SI forgot that it issued that event before the outage, and when the events replayed, it got issued again.

And that’s not bad. Unless events have consequences—which is certainly the case with stock trades. To eliminate these duplicates, StreamInsight needs a little more help, this time on the output. As SI produces events, the output adapter—again, more properly, whatever the output adapter is connected to—has to remember them.

When StreamInsight comes back up after an outage, it will essentially tell its output adapters the last time it remembers. It also guarantees that every event that it produced after that time will be produced again. Since the output adapter knows what these events were, it can remove the duplicates when it sees them instead of acting on them. And this is what we need in our third use case:

3. Your query is happily trading away, taking checkpoints every minute. You also have a historian in front of the query recoding stock prices, and a historian on the end recording trades. As before our doofus—who we have somehow failed to fire—trips over the cable at 12:59 and restarts things at 1:05. During recovery, StreamInsight gets the input to replay all stock info since 12:55, so no input is missed. It also notifies its output that all trades that happened between 12:55 and the outage will be sent again, so when the duplicate trades are issued, the output adapter just drops them on the floor.
   
   And now we have full stream equivalence. You don’t miss a trade, and you don’t get any duplicates.

Hopefully this gives a feel of what’s possible with checkpointing. I’ll have more to say on specifics over the next few weeks. In the meantime, be sure to pick up the release and take a look at the documentation. And there’s a full end-to-end demo available on CodePlex as well.

Cheers,
-Isaac

As I mentioned in my previous post, I recently moved from my old home in the SQL Server engine to the StreamInsight team. As I continue to ramp up here, I want to take a few posts to talk about what I was working on back in the engine: the contained database.

To illustrate the problem contained databases are intended to solve, let’s consider a simple scenario: I take a database D that lives on server S, detach it, and attach it to server T:

This is one example of database movement, but there are many movement scenarios—recovery, upgrade, load balancing, consolidation, moves to production—and many movement techniques—backup/restore, detach/attach, mirroring. But how well do these techniques satisfy the scenarios?

In a word: poorly. As an illustrative example, let’s take a look at what happens with authentication.

SQL Server security is a little complex, but for our purposes it’s enough to consider two core artifacts: logins and users. Logins are instance-level objects (stored in master) and users are database-level objects (stored in the user database). Each of these are responsible for authorization in their respective domains, i.e., used to grant permissions at the instance and at the database respectively. But only logins are used for authentication. So in order to even log on to the server you have to have a login.

Users and logins are associated with each other through a matching identifier called a SID, and in order for a person to connect to and use a database, he must have a user in the target database and matching login on the instance. And here’s where our problem comes from: while users are stored in and move with the database, logins are not and do not. They’re left behind:

The result of this is that someone who could connect to D when it lived on S may no longer be able to do so when it moves to T because their login is missing. This is reparable, of course: you just need to manually add the required logins to the new instance. It’s also not easily automatable, since the target instance may already have a different login with the same name, which would cause a collision.

Were this only a problem for logins, we’d have one nit to work around. But of course it isn’t. The same applies to any other server-scoped object on which the database depends: agent jobs, error messages, linked servers, etc.

I’m going to blow my punch line here: we solved this the authentication problem with contained databases. But before moving on to the details, I’m going to touch on some more subtle problems. In particular, collations. Stay tuned.

Cheers,
-Isaac

Many years ago I kicked up this blog. It started with a trickle of general posts on SQL Server programmability topics. As our spatial support took off, it was quickly consumed by spatial-related posts.

And then it languished…

…and now its back. Let me quickly explain via a quick Q&A with myself:

Where did you go?

Through the 2008 release, I was the program manager for spatial. I led spatial projects, thought spatial thoughts, and dreamt spatial… well only occasionally. In any case, as we ramped up the next release, I started working on what we now call “contained databases”. Since these weren’t public yet, I couldn’t say too much to say about them. And since I wasn’t involved so much in spatial, I didn’t have so much to say there, either.

Why are you back?

After spending six years in the SQL Server engine, I’ve taken a new job. I am now a program manager with StreamInsight. And with this new adventure comes a renewed vigor on the blogging front.

What is this blog about now?

Since most of what I’m working on is centered around StreamInsight, StreamInsight would be a good guess. I’ll probably have a thing or two to say about contained databases as well. And heck, I may even do some spatial.

Stick around and we’ll find out.

Cheers,
-Isaac

Over at his blog, Nicklas Avéns questions the behavior of Filter. He's right: Filter is a little odd. It is also very much by design.

Before we look at filter, let's review the data flow for a typical spatial query. A spatial selection generally has two internal components: a primary filter, which is designed to be fast, but which may produce extra results; and a secondary filter, which is slower, but only works to remove the false positives passed by the primary filter.

The primary filter is accomplished by using the spatial index. The secondary filter requires calls to actual geometry routines, and will be much more expensive. But the secondary filter also guarantees index-independent—and geometrically correct—results.

Without an index, most operations will just skip the primary filter. The result is slow, but returns the same results as the full primary–secondary pathway.

Filter is designed for a pretty common, but somewhat odd use case. Imagine that you want to display a portion of a map: you want to retrieve from the database those objects that would show on the map, but if you happen to retrieve a few extras you don't care—they'll just fall off the display.

In other words, false positives are okay. In fact, they're often preferred, because the overall performance is can be much better if you can skip the expensive geometry operations and instead render some additional objects.

So we added Filter, which simply skips the secondary filter and gives you the quick index-only results.

But what if you don't have an index? We had two choices, either (a) return an error, or (b) revert to STIntersects. Note that both of these options have index-dependent behavior: that ship sailed when we decided to skip the secondary filter.

We chose to allow the operation, but revert to STIntersects in the non-indexed case. This gives us clear semantics: the correct results are always a subset of what we return. It's true that for SQL Server, these are somewhat odd semantics, since they are index-dependent.

But this is the only place in spatial where we have such behavior. If you want consistent results regardless of index, use STIntersects or any of the other methods we provide. If your use case fits the one I described above, you may very well want Filter.

Call it useful, call it odd. Call it Filter.

Cheers,
-Isaac

As part of a larger effort to document our protocols, SQL Server has just released documentation on the structure of our spatial types.  The document is also available in PDF.  Note that all of this is preliminary, and while I doubt it will change much, it could.  I had very little to do with this effort; you have Ed to thank.

It turns out that this structure is pretty simple, and there are folks out there who have fairly easily reverse-engineered it.  Now that we’re publishing the structure, let me make one very specific warning: if you are going to play around with the binary format, do not set the valid bit (v) unless you’re absolutely sure the object is valid.

Setting this should not cause SQL Server itself any problems: you won’t crash anything.  We just can’t guarantee that you’ll get the most predictable results.  And things get very difficult to fix.

Let’s work an example.  Consider the following linestring: LINESTRING (0 0, 10 0, 5 0, 10 0).  This is clearly invalid according to the OGC, since it overlaps itself.  We can put this into a geometry element and play with it:

declare @g geometry = 'LINESTRING (0 0, 10 0, 5 0, 10 0)'

select @g.STIsValid()            -- 0

select @g.MakeValid().STLength() -- 10

We can’t run the STLength() without first making the object valid or we’ll get an error.  We can now pull out the binary version of this to play with:

select @g

And we get:

0x000000000100040000000000000000000000000000000000000000000000000024400000000000000000000000000000144000000000000000000000000000002440000000000000000001000000010000000001000000FFFFFFFF0000000002

We’re particularly interested in the highlighted byte, which contains a number of flags, all currently false.  Most of these aren’t particularly dangerous to play with.  For example, the lowest-order bit tells SQL Server that the item contains Z values.  If you flip it, you’ll just end up with an error, since the data that follows doesn’t actually contain any Z values:

declare @g geometry = 0x000000000101040000000000000000000000000000000000000000000000000024400000000000000000000000000000144000000000000000000000000000002440000000000000000001000000010000000001000000FFFFFFFF0000000002

Msg 6522, Level 16, State 1, Line 1

A .NET Framework error occurred during execution of user-defined routine or aggregate "geometry": 

System.IO.EndOfStreamException: Unable to read beyond the end of the stream.

The valid bit, the third-least significant, is a little touchy, though.  Let’s see what happens if we flip that on our instance and run a few operations:

declare @g geometry = 0x000000000104040000000000000000000000000000000000000000000000000024400000000000000000000000000000144000000000000000000000000000002440000000000000000001000000010000000001000000FFFFFFFF0000000002

select @g.STIsValid()  -- 1

select @g.STLength()   -- 20

What’s happening?  First, the system is trusting the bit and telling us that the instance is valid.  This shouldn’t be too surprising: the bit was added so that we wouldn’t have to perform an expensive check every time we check the validity, and we’re using the optimization.

Second, the result of the STLength() operation is a bit odd.  It’s calculating the length used to draw the invalid object, not the length of the object itself.

Okay, you say: let’s make this instance valid and see if we can fix things:

declare @h geometry = @g.MakeValid()

select @h.STLength()  -- 20

select @h.ToString()  -- LINESTRING (0 0, 10 0, 5 0, 10 0)

Ack!  We’re trapped.  MakeValid() tries to preserve the input geometry whenever possible, and therefore refuses to touch a valid instance.  Our instance isn’t actually valid, but we said it was.

So if you’re creating a geometry from scratch, how are you to know how to set this bit?  That is, how on Earth do you figure out if an instance is valid?  Unless you’re absolutely sure, let the system do it for you: tell SQL that it isn’t valid and then run MakeValid().  Unlike setting the value true, there’s absolutely no harm in doing this—other than perf.  In fact, when MakeValid() sees an object that isn’t marked as valid, it will first check to see if the instance is valid.  If it is, MakeValid() won’t muck with the data, but will just flip the bit to true.

And keep in mind that although you can play with these structures, most of the time you shouldn’t need to.  Use the built-in methods, and use the builder/sink API if you need to do something fancy.  As always, there are plenty of examples on the SQL Server Spatial Tools CodePlex project.

Cheers,
-Isaac