Background

In the team I'm on these days, we're tasked with developing and operating end-to-end semi-realistic applications on our Azure platform with an eye to first-hand experiencing any pitfalls/pains/horrors that our customers face and channeling that to improving the platform. Our first marquee application we developed for that was a virtual online sports betting web site (with fake money of course). Our focus was to get it realistic enough that it hits most of the technical aspects we wanted and even convince bored co-workers to spend some time on it to get some real usage, but not spend years polishing every last bit of it. As such, we managed to show a few leagues in soccer and basketball with updated schedule and betting odds by using a few sources like Bing and scraping web sites, with automated web jobs that keep updating our DB as such.

And even though we did coerce some co-workers to use our web site, we knew that to drive realistic loads on it we'll either have to launch enormous viral marketing campaigns or fake the load using bots. For whatever reason we decided on the latter, so we wrote a few bots that incessantly executed end-to-end scenarios against our web site and recorded their happiness/unhappiness with every interaction. And being the ambitious over-achievers that we are, we deployed this identical setup across all our Azure regions with automated pipelines that automatically recorded incidents whenever there was consistent unhappiness anywhere. Then we went on to other projects while periodically checking on those incidents to light fires under the butts of any teams responsible for this unhappiness on our platform.

The incidents

So that's the background, which brings us to last week. We'd recently gone through a cleanup and were very proud of ourselves having zero uninvestigated incidents, but we looked one day to find many incidents filed on the same day: in a two-and-a-half hour period on 8/29 across the world one of our bots - the Casual Soccer Browser as we fondly call him - kept failing with timeouts while trying to view the events page for Soccer. This was very strange: all these sites were independent and if there was an issue in Azure that affected all regions we'd have definitely noticed it - people running around with their hair on fire tends to be pretty conspicuous after all. So what was going on?

The errant query

Looking a little bit deeper into our logs for a few of those failures, we found that what was causing the page load to be so slow was the database query for fetching the events to show. So I created a graph for how long that query took (we use Elasticsearch and Kibana for that part of our pipeline which gives us the blessed ability to create these ad-hoc visualizations pretty quickly) and the anomaly immediately jumped out - the query was executing blazingly fast until around 11 AM that day, then it started taking around 2 minutes per execution which is crazy, then around 1:30 PM it got back to blazing fast executions like nothing happened. So we have a culprit but are just as puzzled as before: why would this query suddenly slow down like that across the world then get back to normal like this?

The data dive

Our main suspicion was a data problem: something strange must have happened in the soccer world around that time. So we started querying our database. We did see some matches that started and ended around that time but it was just a few matches - nothing our SQL Server databases would struggle with. Looking at the code though, I noticed that the query also looked at our Lines table to determine betting odds for any events that were bettable (bettable = we have odds and the game hasn't started yet). The way our site works is that we store the odds for a game as a row in the Lines table, and if we find that the odds have changed, we add a new row and mark the previous row as not current; only showing the current line as the odds for the game when you see it. Typically the odds can change just a few times for any given game (the way we see it) but when I did a join + count for this investigation on the games during this period I found this one game that had 15K lines for it! Definitely suspicious…

The log dive

So we went back to look at our logs for the web jobs that scrape web sites for current odds, I finally found the logs that explain this whole mystery - these two logs that kept repeating every time the job ran:

Added "HomeTeam Real Madrid, AwayTeam Betis, HomeMoneyLine -1000, AwayMoneyLine 2500 DrawMoneyLine 900, Date 8/29/2015 12:00:00 AM" to "{ Id = 245628, HomeTeam = 38, AwayTeam = 848 }".
Added "HomeTeam Real Sociedad, AwayTeam Real Sporting, HomeMoneyLine -118, AwayMoneyLine 350 DrawMoneyLine 250, Date 8/29/2015 12:00:00 AM" to "{ Id = 245628, HomeTeam = 38, AwayTeam = 848 }".

The explanation

Here is what happened: when we were writing this web site, we had a problem where our source for game schedules would call a team by one name (say Seattle Seahawks) but the source for betting odds would call it something else (say Seahawks). So in the spirit of churning out something realistic-but-quick, we wrote some code that fuzzily matched team names in a way that if they matched perfectly then fine, but if they just matched in parts that would work too. Together with matching game times that worked really well in a way that never bit us, until that day in which these two games were happening at the same time:

1. Real Madrid vs. Real Betis
2. Real Sociedad vs. Real Sporting de Gijón

All these Real teams! And to make matters more fun, Real Sporting de Gijón is stored in our DB as Sporting de Gijón. So our web job while scraping for odds would first come across the odds of the first game and match it perfectly, but then it came across the second game and thought "Real Madrid is probably the same as Real Sociedad since they share part of the name, and Real Sporting de Gijón is probably the same as Real Betis, so that must be the odds for Real Madrid vs. Real Betis!" So it kept going like this:

OK, looks like the odds for this game are A:B, let’s store that.
Nope, looks like the odds now are X:Y, let’s invalidate the old record and store the new one.
Nope, looks like it’s A:B again! Let’s invalidate the old records and store the new one.
Whaddya know, it’s X:Y again! Let’s invalidate the old records and store the new one.

This led to the explosion of the Lines table and the query for those odds taking forever until mercifully the game started and wasn't bettable anymore and the site returned to normal operation across the world. Mystery solved.

TLDR; Two games with royally similar team names happened at the same time, confusing our toy online sports betting web site until the game was over.

In any software system, especially but certainly not just in the cloud, things can and will go horribly wrong in a variety of ways given enough time/opportunity. Some of those are unanticipated disasters: bugs, floods, hurricanes, etc. But a good number of those are just tradeoffs that are consciously made by the developers of the system in order to make other gains:

• a NoSQL system that anticipates some data inconsistency or even data loss to gain scale/performance,
• a cloud system that periodically reboots your VM's to gain automatic/unmanaged updates and security,
• ...

I'm not a purist, and I appreciate those tradeoffs and have made them myself many times, so I don't think these are necessarily bad or evil; but of course I'm writing this post to complain/rally about something, and it's this: if you have a platform/building block where you've made such a tradeoff and anticipate some way of things going wrong, you better give me a way to induce this failure mode to test the rest of the system with it. In other words: saying "my system occasionally decides to run around the room like a mad man to vent steam and you should design for that" is an appreciated but an almost useless statement if you don't give me a way to induce this running around mode in a controlled environment as I'm designing the room to make sure that when it happens it doesn't set the rest of the room on fire.

Doing right

For a decent example of doing this right, look at VM reboots in Microsoft Azure. When you have a Cloud Service in Azure, the system reserves the right to reboot or even reimage any of your role instances (VM's) at any time to deal with hardware failures/updates/etc. Your system should be resilient for that, usually by having redundant instances spread across fault domains. This is well-documented on multiple places, but even more importantly: Azure gives you REST API's, PowerShell commandlets and Portal UI buttons to reboot, reimage or rebuild any of your role instances which is a great way to test that your application is indeed resilient to this. So you can use/do something like WazMonkey to randomly disturb your staging (or hey, if you're brave enough even your production) environment and make sure you're all good, or you can do more targeted testing to e.g. make sure your startup scripts can survive being interrupted by a reboot without ill effects.

Note: I don't think this example is perfect. Ideally the Azure team would have a standard supported test kit that induces all known failure modes into your application (reboot, reimage with/without data loss, etc.) and stand behind it. But as far as things go, I feel good about putting this as my "good" example.

You done me wrong

So what are examples of getting this wrong? Alas, there are too many of those...

Let me pick on the SQL Azure team first (since I worked there so as much to blame as anyone): SQL Azure very much retains the right to drop your connection at any time for a variety of reasons and encourages good client behavior to tolerate this (with the various client SDK's doing a decent job of abstracting this resilience), and yet I don't know of any way to induce such connection drops in a reasonable way.

Another big example is the networking stack: everyone knows that TCP/IP and our networking infrastructure is notoriously noisy and error-prone, and yet it's surprisingly hard to induce all these failures (packet drops, latency, etc.).

Rallying cry

For the most part I've achieved my goal with this post, which is to cathartically rant about this so I can steel myself for the next smug "you should've designed for that" or "we don't support that" statement from a platform developer. But other than ranting, a side goal is to get anyone reading this to be less tolerant of such unsupported statements, and start demanding "OK where's my test kit" every time they hear it. If enough of us start demanding this especially from the most basic building blocks in our computer systems, I sincerely believe our systems will be significantly and noticeably more stable. In a way my attitude towards this is an extension of my attitude towards all software I write: I never say I've actually written my software to perform X until I see it performing X in practice, so I can never sincerely say I've designed/written my software to be resilient to failure Y until I see it actually handle Y in practice. So if I can't induce Y, I'll have to let my software just hang out in the wild until Y inevitably happens. Which is a crazy slow way to test software, and unfortunately the way many of us do it today.

If you've been following this blog recently, you'd have noticed that I'm having a blast trying different data products on Azure and playing with them. I recently managed to get Spark / Spark SQL (Shark's replacement) running on Azure in the same way, but rather than dedicate a post to them, I thought it'd be more fun to pit them all against each other, throwing Hive on HDInsight into the mix, in a friendly competition to analyze some data on my Azure blob storage. So without further ado, I present the ultimate showdown!

The scenario

When I thought this up, I didn't want to design or even steal a precise scientific benchmark to execute (like TPC-DS for example), since I wasn't interested in precise performance benchmarking and I wouldn't be able to tune each one sufficiently anyway. (If you want a more scientific benchmark that compares some of the products here, I found this awesome effort to be pretty informative.) Instead I wanted a simple scenario the likes of which I see a lot, but artificial enough that I'd be able to play with it/release the code for it so anyone can replicate it. So I came up with the following: we'll have a simulated Cloud application in which each VM logs operation messages to blob store in a simple comma-separated format. For this toy example, each log is either a) a start of an operation, b) a success or c) a failure. The business questions we want each of our contestants to answer are:

1. How many failures/successes did each VM get?
2. How long did the average successful operation take on each VM?

I came up with these two questions as a good starting set for this experiment because the first one can be answered with a single scan through the data, while the other would require a self-join to correlate the start and finish messages for each operation. Which gives me a good feel for how the different contestants at least handle this first level of complexity.

The data

Now that I had a scenario in mind, time to cook up some data! I created a simple Azure Compute service where each instance just churns out as fast as it can log messages to its own CSV log file on my Azure blob container. Each line in that file is of the format "{date},{instance name},{operation id},{message}", where date is in a Hive-friendly format, operation ID is a GUID and message is a free-form string so I can throw in a little string manipulation later.

I also cooked up the data so that every tenth role instance (0, 10, 20) would have a higher random operation failure rate, and every seventh instance (0, 7, 14, 21, 28) would have a higher operation duration. This way I can double-check myself and the contestants later to see if we can catch these anomalies.

With that in hand, I deployed it to Azure with 30 (small) instances. They were churning out data pretty fast so I shut it down after about 10 minutes when I had 35.1 GB of data to chew on: this seemed like enough data to make a decent first cut, but not so much that I'd have to worry about sharding across storage accounts to start with.

(Note: all the code used for this experiment is in the public github repo here.)

Enter our first contestant: Hive/HDInsight

So let's start the showdown with Hive/HDInsight: our supported and longest standing NoSQL product in this arena. I spun up an 8-node HDInsight cluster backed by a SQL Azure Hive metastore (so I can use it later with the other contestants), created an external Hive table pointing to my logs, and started issuing the queries using the awesome relatively new Hive portal on the HDInsight cluster.

I created two queries for the two business questions above: a simple COUNT() query for what I'll call the "Count Errors" scenario, and an aggregate over self-join query for what I'll call the "Average Duration" scenario. The average elapsed times for those scenarios on unadorned Hive were 280 seconds and 489 seconds respectively (and yes, the results did capture the planted anomalies so I have decent confidence in the correctness). When I created 16-node clusters, the times went down to 218 seconds and 434 seconds, which is an OK speedup but not exactly linear scaling.

So this was unadorned Hive on HDInsight 3.1. But one of the exciting changes that came with HDInsight 3.1 is the introduction of Tez to Hive, which enables a more sophisticated execution engine than plain-old Map-Reduce. It's not on by default, but it's definitely worth trying out. So I changed the execution engine to Tez (SET hive.execution.engine=tez) and tried it out. The results were surprisingly mixed: the best speedup was in the 16-node case in the Average Duration query which was executed in the blazingly fast time of 305 seconds (spoiler alert: this the fastest among all contestants for this scenario). Interestingly though in the 8-node case this scenario was slower, executing in an average of 574 seconds. For the Count Errors scenario the results are more straightforward: Tez sped up the scenario so it executes in about 2/3 the time as Hive-over-Map-Reduce for both 8-nodes and 16-nodes scenarios.

Here is the summary of the Hive/MR (default) and Hive/Tez results:

And in comes the youngest challenger: Presto

And next comes Presto: the youngest entrant in this showdown. Since Presto is a memory-hungry beast, in my basic configuration I had 8 A6 workers (which is the memory-intensive VM at 28GB RAM and 4 cores), a Large coordinator and 1 Small Hive Metastore. I used the same backing SQL Azure DB for the metastore as I did for Hive so I had the same table definition right away, then I used the Presto CLI from my box to issue the queries. I modified the Count Errors and Average Duration queries to match the more ANSI SQL dialect of Presto (I got the same results as Hive, so again I feel good about the correctness of my results).

The results were pretty impressive especially on the Count Errors scenario: on 8 nodes, Presto blew away 8-node HDInsight: 55 seconds vs. the best Hive result for 8 nodes which is 182 seconds. But wait: is it fair to compare the 8 A6 nodes Presto got vs. the 8 Large nodes HDInsight uses? I can't give Presto a fair chance with Large nodes, and I can't customize node sizes on HDInsight, so I'm stuck with this apples-to-oranges comparison here. They have the same number of cores, but A6 nodes are about twice the price of Large, so I think the fair comparison is to compare against 16-node HDInsight. Even with that allowance, Presto on the Count Errors scenario strongly wins since the best Hive result there is 133 seconds, still more than twice as slow as Presto's 8-node result.

On the Average Duration scenario, the results are good but less impressive. 8-node Presto executed the query in an average of 462 seconds, which beats 8-node Hive (489 seconds for Hive/MR, 574 seconds for Hive/Tez) but is much slower than 16-node Hive/Tez (305 seconds).

So here is the summary of the rumble of Presto vs. Hive/Tez if we take the same rough price point (8 A6 vs. 16 Large):

And here is the rumble if we take the same number of cores (8 A6 vs. 8 Large):

Note: the results get switched around a bit if it's 16 A6 vs. 16 Large, since at that scale Presto is a bit slower than Hive/Tez on the Average Duration scenario, though it beats it on the Count Errors one.

One more note on Presto: one of the things that don't show here that much that I like about it is that of all the three I'm trying here, it struck me as the most predictable in its perf at different scale points. You've seen the surprising scale results I got out of Hive/Tez and Hive/MR, but with Presto when I tried 4 cores vs. 8 cores vs. 16 cores, on both scenarios I got a nice almost (but not quite) linear scaling in the duration: 150->55->38 on Count Errors, and 66->462->366 on Average Duration. Also, if you look at individual query durations (instead of averages) for each scale point, Presto typically has much less variation at any one scale point.

And finally: our electric Shark!

And now comes my newest and most troublesome addition to Blue Coffee: Spark and Spark SQL/Shark (it's really Spark SQL now, but I just like calling it Shark too much). I have to admit: this took me by far the most effort to get up and running, and even now I'm pretty sure I don't have it running that well. My troubles got much less severe once I moved away from the master branch in Spark (which is way too active/unstable) and the standalone discontinued Shark repository, to the branch-1.1 branch with the packaged Spark SQL there that replaced Shark. However, even after that move I still encountered problems. I think my main performance problems that I haven't fully gotten to the bottom of is in the shuffling of data between nodes (BlockFetcher): I had to bump up a couple of connection timeouts to even get it working on the more complex Average Duration scenario, but it still worked too slow for what it is and I think it comes down to problems there.

For this post, I'm not using Spark's full power: I'm not loading the data first into RDD's, instead just querying them from cold storage directly. As follow-up I hope to experiment with that, but for now since I'm not worrying about loading things into memory I can get away with using Large instances for the workers (instead of A6), so I can compare against HDInsight directly and fairly.

So I provisioned a Spark cluster: 8 Large workers, 1 Small Master node, 1 Large node for the Thrift server and 1 Small Hive Metastore node. Since the Hive metastore is backed by the same Azure DB where my table definitions are already stored, and Spark SQL understads HiveQL, I just launched the same queries I used for Hive from BeeLine from within the cluster (launching them from outside fails because Azure Load Balancer kills my connection after one minute).

The results were pretty bad. On 8 workers Count Errors finishes in 271 seconds which is about the same as Hive/MR 8 nodes but much worse the Hive/Tez or Presto, and Average Duration finishes in 714 seconds which is about 25% slower than the slowest 8-node contender so far (Hive/Tez). The worse news is that on 16 nodes the times hover about the same (on average slight faster but within the variance). I dug into it a tiny bit and it seems to do with my mysterious problem with the performance of fetching blocks among the nodes: since that's slow for whatever reason, exchanging blocks among 16 nodes overwhelms any speed gains from distributing the processing among them. It also doesn't help that by default Spark SQL has spark.sql.shuffle.partitions set to 200 which leads to several stages (especially among the small ORDER BY parts I put in there) to be split among 200 tiny tasks, leading to pretty big unnecesary overheads.

Since this is the least tuned/understood contender for me, I'll refrain from putting summary graphs in here: I fully acknowledge that this needs more work to get a fair chance in the arena, so I don't want anyone skimming this post to get the wrong impression about it from a graph.

So... now what?

My hope is that what you get out of this post isn't the specific performance numbers - since again I didn't tune any of this, and it doesn't qualify as a fair performance benchmark. Instead, what I'm hoping you'll get is:

1. the feel for these different data analysis products and what they can do and
2. how awesome it is that you can right now choose among all those different options (and more) to analyze your data in Azure!

In that spirit, I'm hoping to go on to expand this to other possibilities, e.g.: how about instead of loading the logs into blobs, the workers directly inserted them into an Elastic Search cluster instead? Would that make the end-to-end scenario easier? Or what if the logs went into Event Hub, and from there were consumed by both a Storm cluster that calculated average durations on the fly, and a spooler that placed the data into Azure blob store? What if the data was placed in blob store in a more sophisticated format (e.g. ORC or Parquet) than CSV? The possibilities are endless and awesome!

Another avenue I'd like to explore is higher scale: go beyond my puny 35 GB data volume here and into the 10's or 100's of TB's scale. It'd be great to see this work, and what I'd need to shard/do to get this scenario working smoothly.

When you create an HDInsight cluster, whether through the Azure portal or programmatically through e.g. PowerShell, you get a chance to add extra storage accounts to the cluster. In PowerShell it looks something like this:

$clusterConfig = New-AzureHDInsightClusterConfig -ClusterSizeInNodes 4
$allAccounts = @('myfirstaccount','mysecondaccount','mythirdaccount')
$clusterConfig = Set-AzureHDInsightDefaultStorage $clusterConfig `
  -StorageAccountName $allAccounts[0] `
  -StorageAccountKey $(Get-AzureStorageKey $allAccounts[0]).Primary `
  -StorageContainerName 'mycontainer'
$allAccounts |
  Select-Object -Skip 1 |
  %{$clusterConfig = Add-AzureHDInsightStorage $clusterConfig `
  -StorageAccountName $_ `
  -StorageAccountKey $(Get-AzureStorageKey $_).Primary}

What that code snipppet does is setup the default storage account/container (where the cluster's data will go by default), but also configures additional storage accounts that the cluster can access in its jobs later.

A common question that we get though is: I already have a cluster configured, but I want to read some data from (or less frequently, write some data to) a storage account that I didn't setup in this way when I created my cluster - is that possible? The answer is: it's complicated, which is why I'm dedicating this post to a detailed explanation of that slippery answer.

So when you say "configured" - what do you mean exactly?

Before jumping into what's possible or not, it really helps to explain what exactly happens when a storage account is configured. Within the WASB driver that accesses data in Azure blob storage, when we try to access data within a container in a storage account, we go through a series of decisions based on our available configuration to determine how we'll try to access that:

1. If we have a Shared Access Signature for the container, we'll use that.
2. Else if we have a key for the whole account, we'll use that.
3. Else we'll try to get in with anonymous access, and hope the container is a public container.

The key phrase in the preceding part is "available configuration". When a cluster is created with some storage accounts like shown in the introduction, the HDInsight infrastructure does some plumbing work so that the account keys are placed (encrypted, more on that later) in the "core-site.xml" file on every node in the cluster, which is the configuration file loaded by default by almost all Hadoop ecosystem components, so that these keys are part of the available configuration for almost everything you do in HDInsight. But if you know how Hadoop configuration works, you'll know that this is not the only way things can get into the available configuration. For every job you launch, you're allowed to define whatever you want in that job's configuration, which means that you can sneak in additional storage accounts' keys in there!

Alright, show me that word count!

So let's try this out. The easiest way to try this is through a Map-Reduce job, and instead of writing a new job, I'll ask you to bear with me through yet another usage of the well-worn WordCount example. If you want to follow through this example, you'll need an Azure subscription, the Azure SDK installed with that subscription selected as default in PowerShell, an HDInsight cluster, and a storage account that wasn't added to that cluster when it was created. I'll assume the following input variables are defined in your PowerShell:

# Inputs
$clusterName = 'yourclustername'
$wantedAccountName = 'youraccount'
$wantedContainerName = 'yourcontainer'
$localPlayDirectory = 'C:\LocalPlay'

where $localPlayDirectory is just a directory we'll use for files we upload to/download from the cluster, and the rest are self-explanatory. With that out of the way, we need to get the account key, create a storage context we'll use for it, and make sure the container is there:

# Preliminaries: Get the key, create the context, create the container
$wantedAccountKey = $(Get-AzureStorageKey $wantedAccountName).Primary
$wantedAccountContext = New-AzureStorageContext `
  -StorageAccountName $wantedAccountName `
  -StorageAccountKey $wantedAccountKey
$container = Get-AzureStorageContainer $wantedContainerName `
 -Context $wantedAccountContext `
 -ErrorAction SilentlyContinue
If ($container -eq $null)
{
  $container = New-AzureStorageContainer -Context $wantedAccountContext `
    -Name $wantedContainerName
}

And we'll create a simple input blob in the container that we'll count the words for:

# Preliminaries: Create an input blob
$localPlayDirectory = md $localPlayDirectory -Force
$inputFile = "$localPlayDirectory\simple.txt"
'This is an interesting word count demo. It even has two of This' |
  Out-File $inputFile -Encoding ascii -Force
$inputBlobName = 'input/simple.txt'
$inputBlob = Set-AzureStorageBlobContent -Context $wantedAccountContext `
  -Container $wantedContainerName -Blob $inputBlobName -File $inputFile

I'll also define a function that I'll use to run jobs later:

function Run-PlayJob($jobDefinition, $outputDirectory = $null)
{
  If ($outputDirectory -ne $null)
  {
    # Delete the output directory if it exists from a previous run
    $outputBlobs = Get-AzureStorageBlob -Context $wantedAccountContext `
      -Container $wantedContainerName `
      -Prefix "$outputDirectory" | Remove-AzureStorageBlob
  }
  # Run the job and wait for it to finish
  $job = Start-AzureHDInsightJob -Cluster $clusterName `
    -JobDefinition $jobDefinition
  $job = Wait-AzureHDInsightJob -Job $job
  # Get the stderror to get the diagnostics output
  Get-AzureHDInsightJobOutput -Cluster $clusterName `
    -JobId $job.JobId -StandardError
  If ($outputDirectory -ne $null)
  {
    # Download all the output blobs
    $outputBlobs = Get-AzureStorageBlob -Context $wantedAccountContext `
      -Container $wantedContainerName `
      -Prefix "$outputDirectory/part-r" |
      Get-AzureStorageBlobContent -Context $wantedAccountContext `
      -Destination $localPlayDirectory -Force
    # Dump them into stdout
    Get-ChildItem "$localPlayDirectory\$outputDirectory" |
      Get-Content
  }
  Else
  {
    # No output directory, get the stdout of the job
    Get-AzureHDInsightJobOutput -Cluster $clusterName `
      -JobId $job.JobId -StandardOutput
  }
}

OK finally we finish the preliminaries and get to the crux of this. Here is how we define a Word Count example that reads the input from an alien storage account, and writes its output there as well:

# Map-Reduce example
$fullAccountName = "$wantedAccountName.blob.core.windows.net"
$wordCountDef = New-AzureHDInsightMapReduceJobDefinition `
  -JarFile '/example/jars/hadoop-mapreduce-examples.jar' `
  -ClassName 'wordcount' `
  -Arguments "wasb://$wantedContainerName@$fullAccountName/$inputBlobName", `
   "wasb://$wantedContainerName@$fullAccountName/mroutput" `
  -Defines @{"fs.azure.account.key.$fullAccountName"=$wantedAccountKey}
Run-PlayJob $wordCountDef 'mroutput'

If you've been following along, running the above should give you a nice word count for this file that wasn't in the cluster's configured accounts, which is awesome! A few things to note:

1. Note the "-Defines" section, where we defined the key for our wanted account. We could've defined even more keys for more accounts, or we could've also defined shared access signatures instead.
2. Note the usage of fully qualified WASB URI's for the input/output files so we can specify files outside the default account/container.

OK, but can Pigs fly?

Well if that's all there is to it, why did I say in the beginning the answer is complicated? Seems so far to be a simple "yes", right? Well, it would be if HDInsight only exposed pure Map-Reduce, but of course there are other things in there that you can run. So would this work for everything? Well, let's try pig!

Fortunately I can just reuse all the preliminaries I did in the Map-Reduce section, and jump right to the Pig job definition here:

# Pig example
$pigScript = "SET fs.azure.account.key.$fullAccountName '$wantedAccountKey';
LOADED = LOAD 'wasb://$wantedContainerName@$fullAccountName/$inputBlobName';
TOKENIZED = foreach LOADED generate flatten(TOKENIZE((chararray)`$0)) as word;
GROUPED = group TOKENIZED by word;
COUNTED = foreach GROUPED generate COUNT(TOKENIZED), group;
store COUNTED into 'wasb://$wantedContainerName@$fullAccountName/pigoutput';"
$pigDef = New-AzureHDInsightPigJobDefinition `
 -Query $pigScript.Replace("`n", "").Replace("`r", "")
Run-PlayJob $pigDef 'pigoutput'

OK first try it out if you're a brave person following hands-on. Back? You should've seen this succeed with flying colors as well, because Pig is indeed very amenable to this treatment. The code looks a bit different but it's essentially the same idea: we use SET instead of -Defines to pass the configuration value we want, and we still use the same fully qualified URI's. So far so good.

Hive, where our lucky streak runs out

Stil not seeing the "it's complicated" reasoning? Well, now we get to why it's not all butterflies and rainbows. Let's try this with Hive:

# Hive example - doesn't work unless metastore has the key
$hiveScript = "SET fs.azure.account.key.$fullAccountName=$wantedAccountKey;
DROP TABLE IF EXISTS FromMyAccount;
CREATE EXTERNAL TABLE FromMyAccount
(AllOfIt string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
LOCATION 'wasb://$wantedContainerName@$fullAccountName/input';
SELECT * FROM FromMyAccount;"
$hiveDef = New-AzureHDInsightHiveJobDefinition `
 -Query $hiveScript.Replace("`n", "").Replace("`r", "")
Run-PlayJob $hiveDef

If you try running this, you'll finally hit the dreaded error:

Failed with exception java.io.IOException:org.apache.hadoop.fs.azure.AzureException: org.apache.hadoop.fs.azure.AzureException: Container yourcontainer in account youraccount.blob.core.windows.net not found, and we can't create it using anoynomous credentials.

What this is saying really is that WASB didn't find the key for the account in its configuration (the reason it doesn't just say that should be more understandable if you follow its reasoning I outlined in the beginning: it's in the last part of that chain where it's trying to access the data using anonymous credentials). But why? We use SET to provide this configuration, what's it complaining about? The reason lies in the stateful nature of Hive. Unlike Pig (which is stateless), Hive keeps metadata in a service called metastore (backed by a SQL database). The problem is that the "CREATE TABLE" statement runs in the metastore service, which doesn't get the SET statement so doesn't get the configuration from there. There is to the best of my knowledge no supported way to pass this configuration value to the metastore service, so unfortunately the only supported way to get that scenario going is to drop and recreate your cluster with that account configured from the start.

Now, you'll notice I qualified that statement with "supported". If you really want to live on the edge and get this to work temporarily, you can still do it. I won't show detailed step-by-step, but roughly:

1. Enable Remote on the cluster (can only be done in the Azure portal.)
2. Log into the headnode. Make sure it's the active headnode not the passive one (you can check by which services are running, or by the "Hadoop Service Availability" page on the desktop). If it's the passive one, you need to log out, modify the RDP file you downloaded to point to the other headnode instance, and go there.
3. Modify the core-site.xml under the "C:\apps\dist\hadoop-xxx\etc\hadoop" directory and add the key configuration in there.
4. Restart-Service metastore
5. Rerun the script from above, it should work now.

This really isn't the best idea for the long run though: the minute this VM is reimaged by Azure, or for whatever other reason HDInsight switches to the other headnode, this will stop working.

Update (9/9/2014): Thanks to Ashit Gosalia from the HDInsight team for pointing me to this awesome article by Eric Hanson that describes an ingenious way around this problem: instead of hitting the metastore service that doesn't have this key configured, we can bypass it and just tell Hive to get the metadata from a SQL Azure instance directly! This does require you setup your own SQL Azure DB to back this up and configure it as a metastore, so putting it together it looks like this:

# More inputs
$sqlAdminUser = '{fill in}'
$sqlAdminPassword = '{fill in but avoid exclamation marks}'
$metastoreDatabaseName = 'ExternalMetastore'

# Preliminaries
$location = $(Get-AzureHDInsightCluster $clusterName).Location
$sqlScriptLocation = `
 'https://raw.githubusercontent.com/apache/hive/branch-0.13/' + `
 'metastore/scripts/upgrade/mssql/hive-schema-0.12.0.mssql.sql'
$sqlScript = $(Invoke-WebRequest $sqlScriptLocation).Content

# Create SQL Azure metastore DB
$sqlServer = New-AzureSqlDatabaseServer -Location $location `
 -AdministratorLogin $sqlAdminUser -AdministratorLoginPassword $sqlAdminPassword
$sqlDatabase = New-AzureSqlDatabase -ServerName $sqlServer.ServerName `
 -DatabaseName $metastoreDatabaseName -Collation 'SQL_Latin1_General_CP1_CI_AS' `
 -Edition Web
$AllowEveroneFirewallRule = New-AzureSqlDatabaseServerFirewallRule `
 -ServerName $sqlServer.ServerName -RuleName AllowEveryone `
 -StartIpAddress 0.0.0.0 -EndIpAddress 255.255.255.255
$AllowAzureFirewallRule = New-AzureSqlDatabaseServerFirewallRule `
 -ServerName $sqlServer.ServerName -RuleName AllowAzure -AllowAllAzureServices
Invoke-Sqlcmd -Query $sqlScript `
 -ServerInstance "$($sqlServer.ServerName).database.windows.net,1433" `
 -Username $sqlAdminUser -Password $sqlAdminPassword -Database $sqlDatabase.Name
				
# Hive script pointing to my blob
$hiveScript = "
DROP TABLE IF EXISTS FromMyAccount;
CREATE EXTERNAL TABLE FromMyAccount
(AllOfIt string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
LOCATION 'wasb://$wantedContainerName@$fullAccountName/input';
SELECT * FROM FromMyAccount;"

# Defines for Metastore redirect and key
$connectionUrl = `
 "jdbc:sqlserver://$($sqlServer.ServerName).database.windows.net;" + `
 "database=$($sqlDatabase.Name);" + `
 "encrypt=true;trustServerCertificate=true;create=false"
$defines = @{
  "fs.azure.account.key.$fullAccountName" = $wantedAccountKey;
  "hive.metastore.uris" = "";
  "javax.jdo.option.ConnectionURL" = $connectionUrl;
  "javax.jdo.option.ConnectionUserName" = "$sqlAdminUser@$($sqlServer.ServerName)";
  "javax.jdo.option.ConnectionPassword" = "$sqlAdminPassword";
}
# Full Hive job definition
$hiveDef = New-AzureHDInsightHiveJobDefinition `
 -Query $hiveScript.Replace("`n", "").Replace("`r", "") `
 -Defines $defines

# Run the job
Run-PlayJob $hiveDef

Security implications

So in this last section I want to discuss some of the security implications of what I explained here. By default HDInsight encrypts the storage keys for all the initial accounts using a certificate which is generated for this cluster. This is good because these keys give wide access to everything in those accounts, so the more carefully guarded they are the better. Unfortunately this certificate isn't available outside the cluster so we couldn't do the same encryption ourselves, which means that whenever we passed those keys around we passed them in plain-text (the REST API is over SSL so it's not really plain-text over the wire). Also, all the job configuration is stored into job history files in the default container for the cluster, so after you run any of the example jobs here then look at your default container, you'll find a blob called something like "mapred/history/done/.../job_xxx_conf.xml" which contains those keys in plain sight (to anyone with access to that container).

So if you're running your job in someone else's cluster, you really need to assume they'll be able to access your keys if you use the techniques described here. Also even if this is your cluster, this is fairly sloppy key management to have your keys lying around like that in plain text (though I've been guilty of that myself for not-so-sensitive accounts).

A decent mitigation is to use shared access signatures instead of account keys. The basics are the same, except the configuration knob you want is "fs.azure.sas.youraccount.blob.core.windows.net" instead of "fs.azure.account.key.youraccount.blob.core.windows.net". This way if they key is compromised it's at least a granular key to a single container instead of the whole account. Of course for really sensitive data, you are probably better off just configuring the cluster with that account from the start so the keys are encrypted from the beginning.

The Facebook team recently open-sourced a very cool distributed query engine that they concisely called Presto. Unlike say HBase, Presto doesn't store its own data, instead it can plug-in data from a variety of sources (e.g. Cassandra) and offers an ANSI SQL query engine that distributes the query processing to many nodes. Since I'm as knowledgable about Presto as I am about making musicals, I'll refrain from trying to explain more and instead redirect you to their home page or this short article for more information. Instead, let me describe how I got it on Azure Compute and querying data from blob store!

Presto getting a little blue...

When I saw Presto I really wanted to see it in action on Azure and play with it. Of course I could've spun up a few Linux VM's and setup Presto on those (and in fact a colleague of mine started doing just that), but I was looking for a bit more fun so I decided to get it on Blue Coffee instead. Since Presto is pretty much all pure Java, getting it running on Windows wasn't too hard: mainly extending an if statement that only allowed OS X and Linux in to let Windows through as well, and adding a Windows native library to presto-hadoop-apache2. After that it was a matter of slight reverse engineering of their scripts and configuration to get the logic right, and boom I had it running on Azure (of course it was that easy).

Snakey charm

As explained, Presto on its own is not that interesting, it needs data! When I first got it up, my first test catalog for it was Cassandra: I already had that running in Blue Coffee, and configuring it was relatively easy. So I added a catalog configuration class for it, and then tried it out with a simple Azure service that stood up a Cassandra cluster next to a Presto one, with Presto configured to be able to query the Cassandra cluster. It worked like a charm, so now it was on to more interesting things...

Going after the blobby prize

Presto can also query data from "Hive", but it would help if I first explain why I put those quotes there. If you're not familiar with it, Hive is a Hadoop-based service that stores metadata in a database fronted by a service called the Metastore service, stores the actual data in a Hadoop-compatible file system, and then has a server that can translate SQL (HQL) queries into Map-Reduce (or in the case of Tez pure Yarn DAGs) jobs on a Hadoop cluster to get you the results. Presto though, being the crack SQL engine that it is, doesn't use that last part (which is the bulk of Hive): it just queries the Metastore service for metadata, then goes and does its own query processing fetching the data directly from the Hadoop file system when it needs it.

So, since blob storage is already exposed as a Hadoop file system through WASB, this seemed very enticing: I could just spin up a measly Hive metastore service pointing to data on my Azure blob storage account, then I can use Presto as a distributed SQL query engine on that data! Brilliant! Of course, I can already do this with HDInsight and Hive, but the more cool ways I can analyze my data the merrrier right?

Hive wasn't on Blue Coffee, so I added it (at least enough to get the Metastore service running). After that, it was just a matter of adding another catalogue configuration logic into Presto to hook it up with Hive, some fun debugging, then I was able to spin up a Presto cluster that analyzes my blob store data. It was awesome - and since I'm a generous guy like that, in this post I want to guide you through your own way of spinning up such a cluster so you can experience the awesomeness yourself.

A quick guide to a DIY Presto-on-Azure cluster

WARNING: this is all bleeding-edge stuff so don't expect easy packaged solutions here.

As usual, you need Visual Studio and the Azure SDK. To make this tutorial easier, I'll also assume you have an Azure SQL Server database with Hive metadata pointing to your data already. This means: you create an Azure SQL database, create an HDInsight cluster with this database configured as the Hive metastore, create your Hive tables there (make sure you use fully qualified WASB URI's as the location for these tables), then you can drop this HDInsight cluster since we won't need it after that. It's also possible to make the Hive metastore service backed by a local Derby server and skip the preceding step, and if you're adventurous please try that.

With all this prep out of the way, let's build us a Presto service!

1. Open Visual Studio, and create a new Windows Azure Cloud Service project. Add 3 worker roles named "HiveMetastore", "PrestoCoordinator" and "PrestoWorker" (the naming is important) and make sure you have Azure SDK version 2.3 selected.
2. Right-click on the solution, manage nuget packages for solution, Online, Select “Include Prerelease” instead of “Stable Only” (because you’re hard-core like that), search for “Presto”, install “Microsoft.Experimental.Azure.Presto” to the two Presto roles.
3. Similarly, install “Microsoft.Experimental.Azure.Hive” to the HiveMetastore role.
4. Use the following code for the HiveMetastore's WorkerRole.cs:

   using Microsoft.Experimental.Azure.Hive; 
   namespace HiveMetastore
   {
   	public class WorkerRole : HiveMetastoreNodeBase
   	{
   		protected override HiveMetastoreConfig GetMetastoreConfig()
   		{
   			return new HiveSqlServerMetastoreConfig(
   				serverUri: "//{yourdbserver}.database.windows.net",
   				databaseName: "{yourdb}",
   				userName: "{user}@{yourdbserver}",
   				password: "{password}");
   		}
   	}
   }
   

   Filling in the appropriate values for your Azure SQL Server database.
5. Use the following code for the two Presto services' WorkerRole.cs:

   using Microsoft.Experimental.Azure.Presto;
   using Microsoft.WindowsAzure.ServiceRuntime;
   using System;
   using System.Collections.Generic;
   using System.Linq;
   namespace PrestoCoordinator
   {
   	public class WorkerRole : PrestoNodeBase
   	{
   		protected override IEnumerable<PrestoCatalogConfig> ConfigurePrestoCatalogs()
   		{
   			var hiveNode = RoleEnvironment.Roles["HiveMetastore"].Instances
   				.Select(GetIPAddress)
   				.First();
   			var hiveCatalogConfig = new PrestoHiveCatalogConfig(
   				metastoreUri: String.Format("thrift://{0}:9083", hiveNode),
   				hiveConfigurationProperties: new Dictionary<string, string>()
   				{
   					{ "fs.azure.skip.metrics", "true" },
   					{ "fs.azure.check.block.md5", "false" },
   					{ "fs.azure.account.key.{youraccount}.blob.core.windows.net", "{yourkey}" },
   				});
   			return new PrestoCatalogConfig[] { hiveCatalogConfig };
   		}
   		protected override bool IsCoordinator
   		{
   			get { return {true for Coordinator, false for Worker}; }
   		}
   	}
   }
   

   Filling in the information for your Azure storage account, and fixing the IsCoordinator property for the coordinator/worker roles appropriately.
6. Use the following for your ServiceDefinition.cs:

   <?xml version="1.0" encoding="utf-8"?>
   <ServiceDefinition name="<your service>" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition" schemaVersion="2014-01.2.3">
     <WorkerRole name="PrestoCoordinator" vmsize="Large">
       <Imports>
         <Import moduleName="Diagnostics" />
       </Imports>
       <Endpoints>
         <InputEndpoint name="HTTP" protocol="tcp" port="8080" localPort="8080" />
         <InternalEndpoint name="Dummy" protocol="tcp" port="8020" />
       </Endpoints>
       <LocalResources>
         <LocalStorage name="DataDirectory" cleanOnRoleRecycle="false" sizeInMB="102400" />
         <LocalStorage name="InstallDirectory" cleanOnRoleRecycle="true" sizeInMB="256" />
       </LocalResources>
     </WorkerRole>
     <WorkerRole name="PrestoWorker" vmsize="A6">
       <Imports>
         <Import moduleName="Diagnostics" />
       </Imports>
       <Endpoints>
         <InternalEndpoint name="HTTP" protocol="tcp" port="8081" />
       </Endpoints>
       <LocalResources>
         <LocalStorage name="DataDirectory" cleanOnRoleRecycle="false" sizeInMB="102400" />
         <LocalStorage name="InstallDirectory" cleanOnRoleRecycle="true" sizeInMB="256" />
       </LocalResources>
     </WorkerRole>
     <WorkerRole name="HiveMetastore" vmsize="Small">
       <Imports>
         <Import moduleName="Diagnostics" />
       </Imports>
       <Endpoints>
         <InternalEndpoint name="Thrift" protocol="tcp" port="9083" />
       </Endpoints>
       <LocalResources>
         <LocalStorage name="DataDirectory" cleanOnRoleRecycle="false" sizeInMB="1024" />
         <LocalStorage name="InstallDirectory" cleanOnRoleRecycle="true" sizeInMB="256" />
       </LocalResources>
     </WorkerRole>
   </ServiceDefinition>
   

   Notice that I use A6 for the PrestoWorker role: Presto is very memory-hungry, so I really got best results by using A6.

And that should be that. This defines a service that exposes the Presto port that you can interact with using the Presto CLI (IMPORTANT: this is naively unsecured, so if you have sensitive data on your Azure account please don't deploy this). You should be able to try this out in your Azure emulator, and then interact with it using the Presto CLI:

1. Download the CLI jar from here .
2. Run it like this:

   java -jar presto-cli-0.74-SNAPSHOT-executable.jar --server http://localhost:8080
   

3. Try it out:

   use catalog hive;
   show tables;
   select * from mytable limit 10;
   

4. And if all goes well, you can deploy it (again, heed security warning above) and interact with it from your machine the same way except using {yourservice}.cloudapp.net instead of localhost.

Parting thoughts

I really like Presto so far, and I think it's a very promising technology (I have no ties to the team that developed it, and would love to chat with them at some point). This is very early days for both Presto itself at least in the open world outside its promising start within Facebook, and for my experimentation with it on Azure. I can already see very fast query performance for certain types of queries (faster than I could get with Hive on HDInsight), and hopefully I'll get to play with it more and see what I can do with it.

In the first part of this N-part series (where N may or may not equal 2), I thoroughly convinced you that while testing against mock systems is wise and all, it's always cool to also live dangerously and write/run unit tests against live systems at least every once in a while. In this post I'd like to continue this with another pattern that has proved useful to me in this thrilling world.

The secret wild side pattern

So you're a hip start-up programmer writing a super-cool machine learning system to figure out the best coffee blends you can make for any time of day (and you're of course reading this in 2018 when Microsoft has become cool with start-ups again - isn't it great?). C-Blenda relies on a web service that always gives the available coffee beans and their prices/characteretics in any city in the world:

IEnumerable<BeanInfo> GetAllCoffee(Location location)

Now as a conscientious programmer, you abstract this out into an abstract base class (not an interface - a pet peeve of mine that I may expand on some day), with a concrete implementation that actually calls into the web service, and various mock implementations that you use in your tests. You then code up your brilliant implementation with awesome tests to go with it, for example this test that checks that you always give at most three recommendations for the morning (can't tax that caffeine- deprived brain too much in the morning):

[TestMethod]
public void GoodNumberOfMorningRecommendations()
{
  var coffeeInfoProvider = new MockInfo();
  var recommender = new Recommender(coffeeInfoProvider);
  var ideas = recommender.TastiestBlends(Seattle(), Morning());
  Assert.IsLessThan(4, ideas.Count());
}

Your tests pass, you do the obligatory dance-around-the-office celebration, but now you crave more: your tests should also pass against real data. You want your tests to be prim & proper and run against mock implementations in the everyday world, but have the occasional wild night against live implementations.

Introducing two-face

Here is how I usually give a wild side to my tests: first I make the test class an abstract base class, with an abstract method to get the concrete implementation of the dependent system:

protected abstract CoffeeInfoProvider NewCoffeeInfoProvider();

[TestMethod]
public void GoodNumberOfMorningRecommendations()
{
  var coffeeInfoProvider = NewCoffeeInfoProvider();
  var recommender = new Recommender(coffeeInfoProvider);
  var ideas = recommender.TastiestBlends(Seattle(), Morning());
  Assert.IsLessThan(4, ideas.Count());
}

I keep all of the actual tests in the base class, and I then create two thin derived classes that just implement this abstract method differently: the mock one that just returns a mock implementation:

public class MockCBlendaTests : BaseCBlendaTests
{
  protected override CoffeeInfoProvider NewCoffeeInfoProvider()
  {
    return new MockCoffeeInfoProvider();
  }
}

And a live one that either creates a real implementation based on configured information, or skips the tests if not configured for that:

public class LiveCBlendaTests : BaseCBlendaTests
{
  protected override CoffeeInfoProvider NewCoffeeInfoProvider()
  {
    var credentials = TestConfiguration.GetCoffeeInfoProviderCredentials();
    if (credentials == null)
      Assert.Inconclusive("Skipping live tests because we're not configured for it.");
    return new CoffeeInfoProvider(credentials);
  }
}

(See part 1 for my discussion of this TestConfiguration). And voila: I just need to enter my credentials to the web service in a config file whenver I want to test against a live system, the mock tests always run, and I can keep all the tests that don't care about which implementation they go against in one clear place.

Luckily all the test systems I've tried this against (NUnit, JUnit and MSTest) all have the necessary ingredients to make this work: they don't choke on abstract base test classes, and they have a way of skipping the test in the running of it.

This pattern has for the most part served me well. There are a couple of drawbacks though:

1. It doesn't enforce running the live tests regularly. Ideally it'd be coupled with a continuous build system that does run the tests with live credentials at regular intervals, otherwise we're at the mercy of my and my team's discpline to actually run them when needed (which of course is always an excellent idea to depend on my discipline...).
2. Visual Studio at least doesn't differentiate in its UI that much between the two versions of the tests: you sort of see two results for a test called BaseCBlendaTests.GoodNumberOfMorningRecommendations, which can be confusing on which one is the mock and which is the live. It's a bit annoying but it never hurt me too much.

There's an explosion of awesome OSS projects happening in the big data analysis space now. A big chunk of them follow a similar pattern: they're released as Apache projects under the Apache Software Foundation, they are typically written in Java or at least a JVM language like Scala, and even though the JVM is platform-independent they are typically developed and tested on Linux with helper scripts written in bash and/or Python. Over in Redmond, a bunch of us in Microsoft thought about all the ways our customers can use these tools on Azure, and as a result we have a few ways already including:

1. You can use HDInsight, which offers a bunch of these tools in a convenient managed service: Hadoop (Yarn, MapReduce), Hive, Pig, HBase, ...
2. You can provision a linux VM in Azure (yes, we'll still love you), or a bunch of them, and deploy your favorite services in there. You can even use your favorite automation tools such as Chef to automate that deployment.

These are all excellent and supported avenues, and if they serve your needs you should definitely pursue them. However, some of us started thinking about how it would look like to run these tools in an Azure cloud service. I started experimenting with code to deploy and run some of these projects as worker roles in Azure services, and in the spirit of openness I put these experiments up on GitHub and as NuGet packages. This is very much still in the early experimentation phase, but in this post I wanted to show you how you can use these packages to deploy a Cassandra cluster in Azure in your own cloud service. My hope is that those of you who love living on the bleeding edge will try it and give us thoughts/feedback on how useful this seems, general approach and whatever else you have.

Adventurous gulps (tutorial)

So let's walk through creating a web site that uses our own Cassandra cluster as a backing data store. To follow along, you'll need Visual Studio (I'm using VS 2013 Ultimate because I work at Microsoft and they don't mind giving us our own software, but I think Express should work), the Azure SDK, and ideally an Azure subscription, though you can just debug it on the Azure Emulator locally.

1. First, create a new Cloud project. Add a worker role named CassandraNode, and an ASP.Net web role called FrontEnd. To simplify things I chose a simple Web Forms template with no authentication.
2. Install the Azure.Cassandra NuGet package. In the Package Manager Console window:

   Install-Package Microsoft.Experimental.Azure.Cassandra -IncludePrerelease -ProjectName CassandraNode

3. The CassandraNode role will need some local storage resources. Right-click on the CassandraNode role in the cloud project, Properties, Local Storage, and add two resources: an InstallDir sized 256 MB, and a DataDir sized 1024 MB.
4. It will also need to expose some TCP ports for communication. In the same properties page, in the Endpoints tab, expose three Internal endpoints: Storage at port 7000, RPC at port 9160 and NativeTransport at port 9042.
5. Put the following code as the definition of your WorkerRole class for the CassandraNode role:

   public class WorkerRole : RoleEntryPoint
   {
     private CassandraNodeRunner _runner;
     public override void Run()
     {
       _runner.Run();
     }
     public override bool OnStart()
     {
       var installDir = RoleEnvironment.GetLocalResource("InstallDir").RootPath;
       var dataDir = RoleEnvironment.GetLocalResource("DataDir").RootPath;
       var javaInstaller = new JavaInstaller(Path.Combine(installDir, "Java"));
       javaInstaller.Setup();
       var nodes =
         RoleEnvironment.CurrentRoleInstance.Role.Instances.Select(
           i => i.InstanceEndpoints.First().Value.IPEndpoint.Address.ToString());
       var config = new CassandraConfig(
         clusterName: "AzureCluster",
         clusterNodes: nodes,
         dataDirectories: new[] { Path.Combine(dataDir, "Data") },
         commitLogDirectory: Path.Combine(dataDir, "CommitLog"),
         savedCachesDirectory: Path.Combine(dataDir, "SavedCaches")
       );
       _runner = new CassandraNodeRunner(
         jarsDirectory: Path.Combine(installDir, "Jars"),
         javaHome: javaInstaller.JavaHome,
         logsDirctory: Path.Combine(dataDir, "Logs"),
         configDirectory: Path.Combine(installDir, "conf"),
         config: config);
       _runner.Setup();
       return base.OnStart();
     }
   }
   

   This code should setup Cassandra on startup, placing the OpenJDK and the Cassandra jar files in the Install directory, and configuring Cassandra to connect with the rest of the role nodes in a cluster and putting its data in the data directory.
6. Now that we're done with the Cassandra role, let's play with it! Let's create a web site to store all the myths about Cassandra. First, we need to interface with Cassandra. To do that, we'll use the NuGet package CassandraCSharpDriver to work with it:

   Install-Package CassandraCSharpDriver -ProjectName FrontEnd

7. Now add the following two classes to store/retrieve our data from Cassandra:

   public class Myth
   {
     public string Name { get; set; }
     public string Description { get; set; }
   }
   
   public static class Myths
   {
     private static Lazy<ISession> _session =
       new Lazy<ISession>(InitSession);
     private static Lazy<PreparedStatement> _insertStatement =
       new Lazy<PreparedStatement>(PrepareInsertStatement);
   
     public static void AddMyth(Myth myth)
     {
       var bound = _insertStatement.Value.Bind(myth.Name, myth.Description);
       _session.Value.Execute(bound);
     }
   
     public static IEnumerable<Myth> GetAllMyths()
     {
       return _session.Value.Execute("SELECT * FROM allmyths")
         .Select(r => new Myth()
         {
           Name = (string)r["name"],
           Description = (string)r["description"]
         });
     }
   
     private static PreparedStatement PrepareInsertStatement()
     {
       return _session.Value.Prepare(
         "INSERT INTO allmyths(name, description) VALUES (?, ?)");
     }
   
     private static ISession InitSession()
     {
       var cassandraRole = RoleEnvironment.Roles["CassandraNode"];
       var cassandraHosts = cassandraRole.Instances.Select(
         i =>
           i.InstanceEndpoints.First().Value.IPEndpoint.Address).ToArray();
       var builder = Cluster.Builder()
         .AddContactPoints(cassandraHosts)
         .WithPort(9042)
         .WithDefaultKeyspace("myths");
       var cluster = builder.Build();
       var session = cluster.ConnectAndCreateDefaultKeyspaceIfNotExists();
       session.Execute(
         "CREATE TABLE IF NOT EXISTS allmyths(" +
         "name text PRIMARY KEY, description text)");
       return session;
     }
   }
   

   This code sets up a session with Cassandra (after discovering its nodes), creates a keyspace and table for the data, and inserts/retrieves data from there.
8. Finally just add some UI. Since I'm not exactly the world's foremost expert on web UI, I'll be light on details here: Add a web form called "AddMyth.aspx" with a couple of TextBox controls, and the submit button with this simple logic:

   Myths.AddMyth(new Myth()
   {
     Name = NameText.Text,
     Description = DescriptionText.Text
   });
   Response.Redirect("Default.aspx");
   

   And then add a web form called "ShowMyths.aspx", with the following simple GridView:

   <asp:GridView id="AllMyths" runat="server"
     DataSourceID="MythsDataSource" />
   <asp:ObjectDataSource ID="MythsDataSource"
     runat="server"
     SelectMethod="GetAllMyths" TypeName="FrontEnd.Myths"
   />
   

   And then link to both forms from Default.aspx.
9. If you're going to test it in the cloud, you need to make Cassandra a multi-node cluster. So either increase the node count in the UI properties for the Cloud configuration for the CassandraNode role, or just edit this in the ServiceConfiguration.Cloud.cscfg file directly:

   <role name="CassandraNode">
     <instances count="3" />
   

And we're done! You can launch this web site in the debugger on your local Compute emulator and things should work, and/or deploy it to the cloud and see Cassandra operating in Azure. If it doesn't work, you can set the Diagnostics level to "All information" in the properties page for the CassandraNode role, and check out the logs from Cassandra in the WADLogsTable in your storage account to see what went wrong.

So... what other flavors have you got?

Hope this gave you a taste of how to run Cassandra as a cloud service in Azure using this (though I have to emphasize again: this is experimental, and if you want a supported way please just install a supported distribution of Cassandra on Azure VM's). Other than Cassandra, I also have packages to help deploy Apache Kafka and Apache ZooKeeper. I'm also hoping to start putting other packages in there, as well as testing and making sure this approach is resilient to all the failure modes that happen in the cloud. If this all sounds particularly interesting to you, I'd love to hear your thoughts so please leave a comment on this post.

In this day & age of SaaS and everything cloud, a lot of us rarely write code in isolation anymore. Our code is always getting a live twitter feed, using Skynet's machine learning service to analyze it for dangerous human rebellions, and reacting on the results by calling into the appropriate crowd control REST API's. And if we want to test our code to make sure it always terminates the appropriate targets, conventional wisdom tells us to introduce a mocking layer for all the services we depend on, and then test our code against fake versions of those services.

I agree with conventional wisdom for the most part: not only does this save us from having to kill a bunch of people everytime we test the rebellion scneario, it also saves money (Skynet is pretty expensive), helps make our tests fast and reliable (they don't fail everytime twitter is down or our internet is spotty), and helps us cover edge cases and error conditions in ways that would be very difficult to do if we're testing against real services. There are many mocking frameworks to help you do this (I'm partial to Moq for .NET and Mockito for Java), and of course just good patterns of having abstract base classes and concrete implementations that either go to the live service or a fake one (though be careful of putting too much code in the live service class: this is code that won't be tested and it's a bad code smell).

But, and of course you knew this was coming, I also think that testing our code against live services is immensely useful, either in conjunction with testing against mocks or even sometimes instead of it. In this short series I want to discuss some patterns and practices I picked up along the way to do this, and some of my learnings so far.

What I want

There are various things one wants when living on the edge like that to get the thrill benefits of testing against live services while still getting most of the convenience, safety and cost-effectiveness of testing with mocking:

1. Secrets should never go into source control. This applies whether you're in a public GitHub repo or in your company's super-safe code vault. Ideally, whatever patterns we come up with should make it hard to accidentally check-in a secret (account key, password, etc.) in the normal course of testing, even for a junior member in our team who's just learning the ropes.
2. Tests should run in isolation. Even if we're all using the company's credit card to pay for our live services, ideally we should all be able to run our tests on separate machines without having to yell out over our cubicles: "I'M STARTING A RUN NOW - NOBODY DARE START ONE WHILE I'M RUNNING MINE."
3. Tests should clean up resources. Especially for expensive resources like VM's in the cloud or pizza orders, no test run should end up with artifacts around consuming money (unless I guess you're working for an evil corporation - in which case by all means stick it to them). The challenge here is that almost by definition these tests will run with buggy code, so our cleanup mechanisms ideally should be as resilient as possible to these pesky bugs.
4. Live tests should be easy to run. Nobody wants to run your tests, including you. So if running your test involves secret incantations and arcane rituals nobody will run them.

The test.conf file pattern

This is one of my favorite patterns, and one I've seen in many projects so I'm most definitely not inventing it (I'd credit original inventor but most likely this is just one of those organically developed things). I've personally seen it more ingrained in the Java and C++ worlds than the C# world for some reason: that's just my very subjective observation, but since this is my blog I'll base on it the decision to target my example to the C# crowd.

The basic idea of the pattern is to put all the information about the live services you want to use for testing (including secrets) into a file or a set of files that are read by the tests and used to connect to the services. The files are never checked into source control, and whoever wants to run the tests needs to fill in the information into those files before running them. There are other ways I've seen of passing this info around for tests: e.g. environment variables, registry settings, command-line parameters and of course the dreaded hard-coded value in the unit test that you need to edit. I personally have come to prefer the conf file pattern since having these things in a file makes them so much easier to save, copy, and generally manage (in a secure way of course) than any of the other options I've seen.

Getting the test.conf file

The first hurdle you have to pass when implementing this pattern is where to get this file. I've seen several options:

1. Expecting the file in the same directory as the binaries: this is cumbersome because you'd have to leave your IDE and put the file there (and this directory can be different e.g. if you're running Debug vs. Retail binaries), and also because you nuke this file whenever you nuke your binaries directory for a really clean build.
2. Expecting the file in a well-known location such as the user's profile: this is not a bad option, but a) it clutters my machine and b) different source enlistments all use the same file, which can be a bad thing (but can also be a good thing).

The last option as I said isn't bad, but my current personal favorite (for C# projects) is to have the file in a special directory in the enlistment (I usually call it TestConf) and embed this file as a resource. So my enlisment is usually organized something like:

|- Source
   |- EmergencyPizzaProcurement
   |- EmergencyPizzaProcurement.Tests
|- TestConf

(Note that TestConf is not checked in). And then in my project file (.csproj) for the test project, I'd have a link to it like:

<ItemGroup>
  <EmbeddedResource Include="..\..\TestConf\PizzaVendors.txt">
    <Link>PizzaVendors.txt</Link>
  </EmbeddedResource>
</ItemGroup>

Of course this means that in a new enlistment people would have to manually create this directory and file in order to build that, which contradicts my easy-to-run desirable. To circumvent that, I add an MSBuild step to create this directory and file if they don't exist:

<Target Name="BeforeBuild" Outputs="..\..\TestConf\PizzaVendors.txt">
  <MakeDir Condition="!Exists('..\..\TestConf')"
    Directories="..\..\TestConf" />
  <WriteLinesToFile
    Condition="!Exists('..\..\TestConf\PizzaVendors.txt')"
    File="..\..\TestConf\PizzaVendors.txt"
    Lines="# List of URL's for pizza vendors to use (one per line)"
    Encoding="Ascii" />
</Target>

This nicely doubles as a place to put simple instructions for the format of the file. Finally, in my test code I'd have a helper method to get me the configured information:

public static IEnumerable<string> ReadConfigurationFile(string name,
  bool skipCommentLines = true, bool skipEmptyLines = true)
{
  using (Stream resourceStream =
    typeof(TestConfiguration).Assembly.GetManifestResourceStream(
    "EmergencyPizzaProcurement.Tests." + name + ".txt"))
  {
    StreamReader reader = new StreamReader(resourceStream);
    string currentLine;
    while ((currentLine = reader.ReadLine()) != null)
    {
      currentLine = currentLine.Trim();
      if (skipCommentLines && currentLine.StartsWith("#")) // Comment
      {
        continue;
      }
      if (skipEmptyLines && currentLine == "")
      {
        continue;
      }
      yield return currentLine;
    }
    reader.Close();
  }
}

Avoiding these files in source control

As I hope I've stressed enough by now (among many others who've stressed it before me), these files should never go into source control. I'm usually using git these days, and luckily this makes it easier to avoid having these files sneak in there: I just add a line in my .gitignore file to always ignore these files when using git add:

# Test configuration
TestConf/

Format of the file

My overriding concern when writing these files is that they should be human-readable and editable. Since these are typically targeted at myself and my coworkers, I don't care about making them super-robust or generic. So I typically avoid XML or JSON, and just opt for simple text files and create several of them instead of worrying about sections. This also works because they're usually simple and just contain enough information to connect to the live services (see below).

I thought you hated configurable code...

I definitely do. And in general I hate seeing this mechanism (or similar ones) used for evil by making the tests configurable, e.g. having the test read how many files it should create to test with from a configuration file definitely veers into the dark side as far as I'm concerned. The test configuration here should just specify enough info to be able to connect to live services, no more no less. I'm only calling it test configuration because I'm not very imaginative with names, but I'm hoping to find a better name and start calling it that in the future.

And that's all I have to say about this pattern. In the next post I'll talk about other patterns for writing such edgy tests, but I'd love your feedback on this as usual (I'm still learning best practices here so I'd love to hear what works for you).

Starting I think a few years ago, I began to heavily drink the immutability kool-aid. I started getting oddly satisfied whenver I saw a class like this:

public sealed class WeatherState
{
   private readonly float _temperature; // In Celsius of course
   private readonly float _windSpeed; // km/h
   ...
}

With this kind of class, if I have to write a SatisfactionMeter class that calculates anyone's happiness for this weather, I can simply write

public sealed class SatisfactionMeter
{
   private readonly WeatherState _weather;

   public SatisfactionMeter(WeatherState weather)
   {
      _weather = weather;
   }
   ...
}

and not worry about having to clone the WeatherState input object in case someone decides the weather is hotter now in the middle of my calculation. If in this class I have this method:

   public int GetSatisfaction(Person subject)
   {
      return _allCritera
        .Sum(criterion => criterion.EvaluateFor(subject, _weather));
   }

I can speed up my calculation on my multi-core machine by simply adding an AsParallel in there (assuming Person is nicely immutable as well):

      return _allCritera
        .AsParallel()
        .Sum(criterion => criterion.EvaluateFor(subject, _weather));

and not worry if somewhere deep in the analysis code some clever code decides to do what-if analysis by changing the temperature up and down thus messing up other threads' calculations in weird and wonderful ways.

Anyway: people much smarter than me have extolled the virtues of immutability for a while now so go read their explanations if you want to learn more, but this is just to say that I was smitten and wanted this goodness whenever I can get it.

Love in C# land

While C# is no Haskell, a subset of said smart people mentioned above have been steadily making it much easier to reap the benefits of this style of programming:

1. You can't mark a class as immutable, but you can mark fields as readonly and most (all?) primitive and basic library types are immutable so in practice I didn't miss that too much.
2. The Immutable Collections library has been heavenly, and even though it's still a beta nuget package I've personally been using it in almost all my projects since I found it and have had no problems at all with it.
3. Even auto-properties are going to get immutability-favoring features in C# 6, thus hopefully correcting my pet peeve with everyone using auto-properties with private setters as if that was an equivalent way of having readonly fields (it's not - methods in the class can still mutate those properties).
4. And of course all the parallelism goodness that came after around .Net 3.5 (PLINQ, TPL, ...) has made it much more lucrative to program in an immutable style and very easily get the parallelism advantages out of it.

So overall my love affair with immutability has been mostly untroubled in C#, except for our ever-going fight with complex data structures...

Brewing troubles

My first seeds of doubt that maybe immutability is a bit of a phase for me, and like everyone going through a phase I'm annoying the people around me by taking it too far, were sown when I wrote a relatively big library/toolset that revolved around a complex data structure (a description of a test run, including what tests to run, how to set up their environment, what parameters they take, etc.). Since I started writing it mostly on my own I indulged in my new-found love and made it all immutable, which proved more complex than I thought. To illustrate the problems, let me take the weather example from above a little further. Let's say I want to write a weather predictor, and I realize that the easiest way to predict tomorrow's weather without those pesky weather models is to take today's weather and just add/subtract a few degrees. So to make the job of my predictor even easier, I decide to add a new method to the WeatherState class that just returns the same weather but with a different temperature:

  public Weather ChangeTemperature(int newTemperature)
  {
    return new Weather(newTemperature, _windSpeed);
  }

And right away there are already a few problems:

1. The method name: If you're reading this method in an article about immutability, you may not have trouble recognizing that ChangeTemperature returns a new object with a new temperature. But if you're trying to use a new library and discover a method called ChangeTemperature on the object you're using, it's not a far stretch to think that this method actually changes the temperature in your object so you just use it like that (as if it's a setter). Which is exactly what happened when people started using my library: it became a fairly common bug for people to ignore the new returned value from the ChangeX() methods and just assume that the method changes the existing object. Now with the benefit of hindsight I don't think I would have named them ChangeX(), but to this day I don't have a great name pattern for this - maybe WithNewX? (e.g. WithNewTemperature()). Also in my defence even some base library methods suffer from this confusion: I'm sure many people fell into the trap of calling String.Replace() on a string thinking it replaced it in-place, instead of taking the return object as the new string.
2. The method implementation: The implementation is a one-liner, but it's not exactly as simple as implementing a setter. You have to pass in all the members of the class to the new constructor except what you're changing, which is definitely more complex and error-prone than { _temperature = value; } (and it's the reason I have this method in the first place, instead of having client code just use the constructor directly to achieve the same effect). And it gets progressively worse as the class gets bigger.

Now, partially to address the second problem, I began to aggressively nest my classes to keep them from having too many members. So in this example let's say we wanted to add the wind direction to the weather state: instead of adding it as a new member, I would bundle it up with wind speed into a new WindVelocity class, and add that to the WeatherState class. Now the WeatherState class still only has two members (temperature and wind velocity), and everything is tidy and awesome.

Except of course that's when my real trouble began: let's say we want to predict tomorrow's weather as the same as today except with stronger wind. So I want to add a method to change wind speed. Now there are two ways I can do this:

1. I can add a WeatherState.ChangeWindSpeed() method which creates a new velocity object with the new wind speed and returns the new weather state object or
2. I can add a WindVelocity.ChangeSpeed() method and a WeatherState.ChangeWindVelocity() method, and leave it to the client code to first construct the new wind velocity object then use that to create the new weather state object.

Neither option is pretty. The first one means that my highest level class will have way too many methods, and is also a weird violation of abstraction levels (why should the weather state object understand the speed concept in the velocity object? it's an internal detail of the workings of velocity). The second one leads to pretty verbose client code:

  var tomorrowsWeather = todaysWeather.ChangeWindVelocity(
    todaysWeather.WindVelocity.ChangeSpeed(
      todaysWeather.WindVelocity.Speed + 5));

In the end I settled on the second option, mainly because I'm a sadistic monster who doesn't care about the suffering of the users of my library. But even my cruel heart was moved a little by their plight...

Uneasy truce

Annoyingly, I don't have any real conclusions to this post. The whole experience left me with a vague feeling that surely languages can and should evolve to improve the situation here: after all this is almost purely a syntactic complexity not a conceptual one, so the language should help me out here. But I have no idea how I would do it if I were a language designer, which I guess is why I'm not a language designer.

And until such time as languages make these problems moot - will I still make my data structures immutable? Probably. I still love all the goodness that comes with immutability, and am not willing to give that up. I'll try to name my methods better to avoid confusion, but honestly I still don't have good solutions to the problem of changing a deeply nested member. So if you have any thoughts I'd love to hear them.

The problem

Azure Blob Storage, like Amazon S3 and Google Cloud Storage and I'm sure other data storage systems in the cloud and on Earth, takes a nice simple do-one-thing-well approach: give me a bunch of bits and a name within a namespace, and I'll associate these bits with that name for you to ask for it later. In the case of Azure, the bunch of bits is called a blob and the namespace is called a container (within a storage account). The bits could be a video stream or an XML document, the name could be "Plan for next billion $.docx" or "My consciousness.vhd" and it all looks the same as far as storage is concerned. In particular though: the name could be "I\am\a\directory\with\a\file" or it could be "Psht/this/is/what/a/directory/looks/like" and it would still be treated as a plain old name in blob storage, since in the storage service itself these "\" and "/" characters have no special meaning and are just like "a" or "$". This is not true of the SDK and some of the tools on top of the service, but they are all providing their own illusions on top of a simple flat name-value store, just like we needed to provide our own illusion...

The problem is that Hadoop wants its backing file system to be a hierarchical file system, with the directory structure we've all grown familiar with and fond of. So we needed to code up our WASB FileSystem class (this is the code that presents the Azure Storage service as a FileSystem implementation in Hadoop) to bridge the two worlds in a way that:

1. Presents the hierarchical system abstractions as faithfully as possible with (as much as possible) no cracks through which our abstraction leaks. This is important not just for theoretical purity, but as we painfully learned through trial and error because any leaks in our abstraction can cause various Hadoop ecosystem components to break in strange and wondrous ways...
2. Enables customers to analyze data in blob store that they obtained through common tools outside HDInsight.

The approach

Given the requirements, the most obvious approach is to do what the SDK and most tools over Azure storage do: if Hadoop asks us to store the file "Trip Durations.csv" in the directory "Buses" within the top-level directory "Seattle", then we just store it as the blob "Seattle/Buses/Trip Durations.csv". Then when Hadoop asks us to list all the files in the directory "Seattle/Buses", we looks for all blobs whose names start with the prefix "Seattle/Buses/" (and whose names don't contain "/" after that) and we return those, taking everything after the last "/" as the file name. And if it asks us to list all directories under "Seattle", we'll look for all blobs whose names start with the prefix "Seattle/" and whose names do contain "/" after that, find all the distinct directory names (sandwiched between the "/" characters) and return those: e.g. if we find "Seattle/Buses/Trip Durations.csv", "Seattle/Buses/Ridership.csv" and "Seattle/Bikes/Trip Durations.csv" then the two directories we return are "Buses" and "Bikes". Easy (sort-of).

But of course life can't be that easy. The first major problem comes when Hadoop does something like this:

Tricky case 1

Create directory "Cars" under "Seattle"
List all directories under "Seattle"

The trickiness here comes from the fact that we want to create (and see) an empty directory. The scheme above doesn't allow for that: we typically would know that there is a directory called "Cars" once we see a blob called "Seattle/Cars/Pollution.csv" but we have no blob called that yet. We liked the basic approach, so we considered two ways to supplement and salvage it:

1. Create a special catalog blob per container that records such facts as the existence of empty directories.
2. Create a special marker blob per directory that we create.

We decided against the first approach mainly because of contention problems when many machines are creating/deleting directories all over the place and want to record this in the one blob. So we turned to the second, and now our problem became how do we mark these special blobs. Our first attempt (which some people saw in our very early previews) was to create empty blobs with a special suffix "_$folder$", which worked OK but its main drawback (aside from its ugliness when viewed outside HDInsight) was that it led to the following situation being possible:

Tricky case 2

(In Hadoop) Create directory "Thoughts" and put file "Brilliance" in there with very important thoughts
(Oustide Hadoop) Create blob "Thoughts" with very important thoughts in there
(In Hadoop) Tell me whether "Thoughts" is a directory or a file

With our implementation so far, the third step would be stumped: we have three blobs in there, "Thoughts_$folder$", "Thoughts/Brilliance" and "Thoughts", so the concept "Thoughts" is both a file and a directory! Of course we could error out, but we weren't very fond of that because a well-meaning customer could find themselves in that situation, with data in there to analyze and no simple remedy. Ideally we would've stopped the customer in the second step before the data is placed so they could place it in a consistent manner in the first place. So for this and other reasons, we decided to move to our current implementation: we create the marker blob as an empty blob with the exact name of the directory and no ugly suffixes, and mark it instead using attached metadata called "hdi_isfolder". So now the second step in case 2 above becomes nicely blocked: the customer would get an obvious conflict and would have to stubbornly delete the marker blob before creating the new "Thoughts" blob and shooting herself in the foot.

Other wrinkles

With the approach above of honoring the standard "this/is/a/directory/with/a/file" naming convention and using special metadata-marked blobs to create empty directories we were mostly OK. We hit some interesting situations though during implementation and testing, so for fun here is my favorite:

Tricky case 3

(Outside Hadoop) Create the blob "Songs/Get Lucky"
(In Hadoop) Check if the directory "Songs" exists
(In Hadoop) Delete the file "Songs/Get Lucky"
(In Hadoop) Check if the directory "Songs" exists

The correct behavior is to answer "Yes" both times we check if the directory exists: the first time because we decided to honor the naming convention and say that the existence of the blob "Songs/Get Lucky" implies the directory "Songs", and the second time because we already said the directory exists and didn't delete it so it should still exist (normal file systems don't delete a directory just because the last file is deleted). But the naïve implementation of our approach would not yield this behavior: our implementation would dutifully perform the complex "Check for prefix or marker blob" check in the second step and correctly answer "Yes", then in the third step delete the blob "Songs/Get Lucky", then in the fourth step say "No" since now the container is completely empty.

To deal with this (and other quirks) we codified the concepts of "materialized" and "implicit" directories: a materialized directory is one where we have the special marker blob, and an implicit one is one we infer from the existence of blobs that have it as a prefix (i.e. an implicit directory "Songs" exists because of the existence of the blob "Songs/Get Lucky"). Then whenever we delete a file, we check if its parent directory is implicit or materialized: if it's implicit, we materialize it first before we delete the file then the situation above would be resolved.

Note that we could've dealt with this situation in the second step instead: when we check for directory existence, we always materialize it if necessary before answering "Yes". The main reason we didn't do that is that we honor read-only operations: Azure Storage allows customers to give read-only permissions on a container to an HDInsight cluster (through the use of Shared Access Signature keys), so if we're doing a read-only operation like checking for directory existence, we avoid doing any write operations such as materializing a directory. When deleting a file though we assume we have write-access so we give ourselves leeway to do that.