by Felix Rieseberg, Partner Catalyst Team

This case study describes the first open-source collaboration between TED and Y Combinator, enabling applications developed with SailsJS to be deployed directly to Azure.

Customer Problem

SailsJS is an open-source real-time MVC framework for Node.js. It used widely by Node.js developers to develop enterprise-grade server solutions. In the Node world, SailsJS is to Node.js what Ruby on Rails is to Ruby. Its ensemble of small modules work together to provide simplicity, maintainability, and structural conventions to Node.js apps.

The company behind SailsJS, Treeline.io, was a part of Y Combinator’s Summer 2015 class.  During the first meetings between Treeline.io and TED, it became obvious that there’s great interest in SailsJS and equally great interest in Azure by companies using SailsJS, yet a reliable deployment path was not defined. The core of the issue was simple: How can Microsoft and Treeline.io best enable customers to deploy SailsJS apps as Azure Web Apps?

Overview of the Solution

We began our investigation into a solution by looking at other cloud providers – many offered tutorials, but most of them seemed cumbersome. SailsJS already comes with a command line interface and we decided that the best user experience would explain itself: If the command [sails run]starts SailsJS, wouldn’t the best deployment experience be a single command like [sails deploy azure]?

The solution provides exactly that: from within the SailsJS CLI, users can deploy any Sails app within a few seconds. The solution consists of three Node Modules – a Node Machine implementing Azure APIs, an npm module extending the SailsJS CLI with a [sails deploy] command, and an npm module enabling [sails deploy] to directly deploy to Azure.

Implementation 

SailsJS makes heavy use of Node Machines, an open standard for JavaScript functions. They are essentially meta-data wrappers for APIs, allowing other applications to consume the packages in a standardized way. We started by building a Node Machine for Azure, which is capable of setting up the local Azure account, creating and updating websites and uploading files. In total, the machine pack implements 13 APIs:

·       Checking for an active Azure subscription

·       Listing azure subscriptions

·       Registering an azure subscription

·       Setting a new active azure subscription

·       Checking if a website already exists for a given account

·       Creating a new website

·       Setting deployment credentials for a website

·       Uploading files to a website

·       Uploading webjobs to a website

·       Triggering webjobs

·       Fetching meta data information for a deployed webjob

·       Fetching stdout and stderr for a webjob

·       Listing available VM images for an account

The machine pack is available on GitHub, node-machine.org and NPM independently of this solution, as it might be useful in other scenarios.

For the purpose of this solution, we consumed the Node Machine to implement a deployment strategy for SailsJS. Two Node modules had to be created to enable the scenario: The SailsJS CLI needed to be extended with a [sails deploy] command. This proved to be surprisingly easy and only took about 50 lines of code. Then, the [sails deploy]command needed a descriptive strategy for deploying apps to Azure, which we encapsulated in the Node Module sails-deploy-azure.

The module sails-deploy-azure in detail implements the following flow.  (Note: The code is too long to include here, but each section is followed by links to relevant code locations.)

1)      The command sails deploy accepts a website name and deployment username/password as optional parameters. If the command is run without parameters, a new website has to be created. We start by checking the local environment for an Azure account. Users who have the Node-powered Azure-CLI installed won't have to do anything, while users without it are asked to authenticate the local environment (they won't have to install the CLI though, since we wrap and abstract the CLI as a dependency).  If the command is run with parameters, we don’t have to create the Azure Website, and skip to step #3. [Code: L22-L51 in sails-deploy-azure/index.js, consuming machinepack-azure

2)      Once we're authenticated, we check if a website for the current SailsJS app already exists in the current account by comparing namespaces. If so, we could just deploy to it - if not, we create one. [Code: L76-L152 in sails-deploy-azure/index.js, consuming machinepack-azure]

3)      We're ready to ship things up, so we zip the local folder, excluding unnecessary files and folders. We could have used Git to deploy, but Treeline.io preferred to not take a Git dependency. We use a Node-native ZIP implementation to compress the whole app into a single file. Next, we upload it to a temporary folder on the web app’s virtual machine. [Code: L169-187 and L263-L340 in sails-deploy-azure/index.js, consuming machinepack-azure]

4)      Now that the package is uploaded to Azure, we finish execution of local commands and commence remote execution. We upload a script containing commands to clean the site, unzip the package and run [npm install] to Azure, where it is automatically assigned a REST endpoint with API - allowing us to call the script, fetch its status and stderr/stdout output. Once the script is uploaded, we simply call its API, pipe through the output to the local machine and watch as Azure is setting up the SailsJS website. [Code: L188-246 in sails-deploy-azure/index.js, consuming machinepack-azure; sailsdeploy.ps1]

Running the Code

To try this out, you'll need Node.js and NPM installed on your local machine. To try the actual deployment, you can either go the brave path of also setting up a SailsJS app or use a provided test app (http://1drv.ms/1A7r3hS). If you want to go with a provided test app, skip to step 5.

1) Install the latest version of Sails by running the following command. At the time of writing this tutorial, commit 8747a77273c949455a8a89d79abfd36383d10e73 was used.

npm install -g balderdashy/sails

2) Create a new folder for your brand new SailsJS app and open up PowerShell/Terminal in said folder. Run the following command to scaffold the app:

sails new .

3) Update the configuration to use the correct port by opening up config/env/production.js and updating it to look like this:

module.exports = {

  port: process.env.port,

}

4) It's a good idea to disable Grunt for your website - it's extremely useful while in development mode, but shouldn't be part of a deployment. This step is not specific to Azure, but a good asset management practice for SailsJS. Open up package.json and remove everything that begins with "grunt-" except for Grunt itself.

5) Log into the Azure Portal and create a new web app. Ideally, the website should have some power to run the Sails installation process without trouble. To ensure enough resources, create the website either in a "Basic" or a "Standard" plan. Make also sure to set deployment credentials for your website.

6) Go back to your Sails app and create a file called ".sailsrc" in its root folder. Fill it with the following JSON. Make sure to set the sitename, deployment username, and deployment password to the right setting.

{

    "commands": {

        "deploy": {

            "module": "sails-deploy-azure"

        }

    },

    "azure": {

        "sitename": "YOUR_SITENAME"

        "username": "YOUR_USER",

        "password": "YOUR_PASSWORD"

    }

}

7) You are ready for deployment! Run the following command from your app's root folder to let your local Sails app deploy to Azure:

node ./node_modules/sails/bin/sails.js deploy

Challenges

Azure does not allow non-organizational accounts to set deployment credentials via command line and instead forces users to set those credentials via website. For users with organizational accounts, the user experience is therefore better, since we can automate the setting of credentials – whereas our only option for ‘pay as you go’ customers is to point to the Azure Management Portal.

Opportunities for Reuse

The same code can be reused in large portions to enable deployments of similar frameworks – it is obvious that the implemented flow is not specific to SailsJS. Practically speaking, the ZIP package being deployed to Azure could be filled with any framework or application. The script being uploaded to Azure to finish the installation is equally agnostic, but could also be extended to include more framework-specific commands. The Node Machine can be reused by any Node application implementing Node machines.

Since SailsJS itself as well as all three modules developed during this customer engagement are open source and licensed with the MIT license, other developers are free to reuse the created code.

Felix Rieseberg is an Open Source Engineer in Microsoft's Partner Catalyst team where he spends his time coding on the cutting edge of technology, trying to improve developer's lives by releasing quality code on GitHub and touring the world presenting the lessons we've learned.  Read his blog at http://www.felixrieseberg.com/, follow him on twitter @felixrieseberg, and check out his work on github at https://github.com/felixrieseberg.

Overview

By Shawn Cicoria, Partner Catalyst Team

This case study focuses on migrating existing multi-tiered applications to Azure, using Docker containers and nginx to expedite rehydration and reduce the overall surface area for open ports for the main application. After reviewing this case study you’ll have a basic understanding on a use case for nginx and dealing with a common scenario for on-premises applications and migrating them to the cloud – specifically, handling non-standard HTTP ports that are artifacts of developer choices and masked when on-premises.

The Problem

The problems presented and addressed are two areas that often happen when certain implementation choices are made and lack of communication between an organization’s Development and Infrastructure teams exists.  As teams progress towards a DevOps structure and culture, these issues are generally recognized and addressed early.

The example presented here is abstracted from a more complex solution and simplified to highlight the basic processes and core technologies that are applied to address some common issues.

Rapid Environment Setup and Deployment of Tiered Solution

Existing multi-tiered solutions are often hard to rehydrate in new environments and may take days, even weeks to get the right people focused on addressing the connections and linkage of all the moving parts.

In a good, well established DevOps culture, the team can easily and quickly establish a full environment for development, test, staging, disaster recovery, even production with minimal intervention. Often however, if teams are disconnected or lack concern, motivation, or just lack focus until its tool late, these proper processes and approaches are not followed, and migration is slower and more expensive than it needs to be.

Use of Non-Standard Ports

One primary issue is that developers often implement solutions that make use of non-standard TCP ports (e.g. HTTP(s) over port 8000 or 44300 – not 80 or 443). Standard or “well known ports” are listed here: http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers and here http://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml.

When solutions are deployed internally on-premises, these issues generally go unnoticed, since traffic within the corporate network is usually routed without concern as ports are not generally locked down on routers, switches, and internal bridges. Other than additional configuration that may happen on the server to open up these additional ports (which may be open by default – no firewall), the use of non-standard ports goes undetected or pushed to a low priority fix. However, moving to the cloud requires that we have more explicit control and knowledge of port usage.

Existing On-Premise Solution Overview

The architecture that is being used is shown in the diagram in this section (Figure 1 - On-Premises and Non-Standard Ports). The core aspects of the solution are:

1. Browser based rich client
   1. HTML & JavaScript
2. Tomcat used for static content – port 80/443
   1. HTML / CSS / JavaScript / images
   2. No direct calls from this tier to any other tier
   3. “serverConfig.js” containing variable that identified the Spring Framework DNS name and Port – as example:

var baseAddress = 'http://apiserver:8080';

1. Spring Framework / Java – port 8080/44300
   1. Used for REST services
2. Mongo DB – port 27017
   1. Document Storage / Database

NOTE: While static content was deployed to an IaaS instance running Docker, further determination should be done to identify if CDN is applicable and other application remediation approaches; further review of nginx modules, CDN requirements, and cache-ability of content is required.

Non-Standard Ports

Non-standard ports are an issue as we move the solution to the cloud. Generally edge gateways and proxy servers expect most traffic or permit only traffic to known ports. While getting an exception to be implemented in a firewall rule for specific applications is an option, this creates governance overhead, management, and support issues as the number of solutions in the enterprise increase with these exceptions.

Rapid Environment Setup

As new developers and testers ramp onto a team and a solution, it’s important to be able to rapidly stand up a working environment so they can be productive right away. In addition, as new releases of the solution are implemented, testing the solution in a clean environments helps to diagnose situations where old code or remnants of prior solutions may remain. While production deployments may be different, using clean well known configurations and applying a release on that narrows the complexity and helps mitigate issues that occur as machines run over time and various configuration or environment changes are made and potentially not tracked.

Overview of the Solution

# Dockerizing MongoDB: Dockerfile for building MongoDB images
 # Based on ubuntu:latest, installs MongoDB following the instructions from:
 # http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/
 
 FROM ubuntu:latest
 MAINTAINER Shawn Cicoria shawn.cicoria@microsoft.com
 
 ADD mongorun.sh .
 # Installation:
 # Import MongoDB public GPG key AND create a MongoDB list file
 RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
 RUN echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | tee /etc/apt/sources.list.d/10gen.list
 
 # Update apt-get sources AND install MongoDB
 RUN apt-get update && apt-get install -y mongodb-org
 
 # Create the MongoDB data directory
 RUN mkdir -p -m 777 /data/db
 
 RUN mkdir -p -m 777 /tmp
 
 COPY mongodb.catalog.json /tmp/mongodb.catalog.json
 COPY mongodb.dealers.json /tmp/mongodb.dealers.json
 COPY mongodb.quotes.json /tmp/mongodb.quotes.json
 COPY mongorun.sh /tmp/mongorun.sh
 
 WORKDIR /tmp
 
 RUN chmod +x ./mongorun.sh
 
 # Expose port #27017 from the container to the host
 EXPOSE 27017
 
 VOLUME /data/db
 
 ENTRYPOINT ["./mongorun.sh"]
 

#!/bin/bash
 
 # Initialize a mongo data folder and logfile
 sudo rm -r /data/db 1>/dev/null 2>/dev/null
 mkdir -p -m 777 /data/db 
 touch /data/db/mongodb.log
 echo step 1
 # Start mongodb with logging
 # --logpath Without this mongod will output all log information to the standard output.
 # --logappend Ensure mongod appends new entries to the end of the logfile. We create it first so that the below tail always finds something
 /usr/bin/mongod --smallfiles --quiet --logpath /data/db/mongodb.log --logappend &
 MONGO_PID=$!
 echo step 2
 # Wait until mongo logs that it's ready (or timeout after 60s)
 COUNTER=0
 grep -q 'waiting for connections on port' /data/db/mongodb.log
 while [[ $? -ne 0 && $COUNTER -lt 90 ]] ; do
 sleep 2
 let COUNTER+=2
 echo "Waiting for mongo to initialize... ($COUNTER seconds so far)"
 grep -q 'waiting for connections on port' /data/db/mongodb.log
 done
 
 # Now we know mongo is ready and can continue with other commands
 echo now populate
 #some point do something to chedk if already run; but for this demo just do it.
 /usr/bin/mongoimport -d ordering -c catalog < /tmp/mongodb.catalog.json
 /usr/bin/mongoimport -d ordering -c dealers < /tmp/mongodb.dealers.json
 /usr/bin/mongoimport -d ordering -c quotes < /tmp/mongodb.quotes.json
 
 wait $MONGO_PID

FROM java:8-jre
 MAINTAINER Shawn Cicoria shawn.cicoria@microsoft.com
 
 ENV APP_HOME /usr/local/app
 ENV PATH $APP_HOME:$PATH
 RUN mkdir -p "$APP_HOME"
 
 WORKDIR $APP_HOME
 
 ADD ordering-service-0.1.0.jar $APP_HOME/
 ADD startService.sh $APP_HOME/
 
 RUN chmod +x startService.sh 
 
 EXPOSE 8080
 #CMD ["java", "-jar", "ordering-service-0.1.0.jar"]
 CMD ["./startService.sh"]
 
 

#!/bin/bash
 
 while ! curl http://mongodb:27017/
 do
 echo "$(date) - still trying"
 sleep 1
 done
 echo "$(date) - connected successfully"
 
 java -jar ordering-service-0.1.0.jar

FROM nginx:1.7.10
 MAINTAINER Shawn Cicoria shawn.cicoria@microsoft.com
 
 ENV WEB_HOME /usr/local/web
 
 ADD Web.tar $WEB_HOME/
 
 COPY nginx.conf /etc/nginx/nginx.conf
 
 EXPOSE 8000
 CMD ["nginx", "-g", "daemon off;"]

worker_processes 1;
 
 events {
 worker_connections 1024;
 }
 
 http {
 include mime.types;
 default_type application/octet-stream;
 
 sendfile on;
 keepalive_timeout 65;
 
 gzip on;
 
 server {
 listen 8000;
 server_name localhost;
 
 location / {
 root /usr/local/web/Web;
 index index.html index.htm;
 }
 
 error_page 500 502 503 504 /50x.html;
 
 location = /50x.html {
 root html;
 }
 
 location /catalog {
 proxy_pass http://orderservice:8080;
 }
 

 location /quotes {
 proxy_pass http://orderservice:8080;
 }
 location /shipments {
 proxy_pass http://orderservice:8080;
 }
 }
 }
 

#!/bin/sh
 cd mongoseed
 docker build -t scicoria/mongoseed:0.1 .
 cd ../orderService
 docker build -t scicoria/orderservice:0.1 .
 cd ../staticsite
 docker build -t scicoria/staticsite:0.1 .
 cd ..

#!/bin/sh
 docker run -d -p 27017:27017 --name mongodb -v /data/db:/data/db scicoria/mongoseed:0.1
 docker run -d -p 8080:8080 --name orderservice --link mongodb:mongodb scicoria/orderservice:0.1
 docker run -d -p 8000:8000 --link orderservice:orderservice scicoria/staticsite:0.1

var baseAddress = window.location.protocol + '//' + window.location.hostname;

By Felix Rieseberg, Partner Catalyst Team, with input from Domenic Denicola (Google, member of the ECMAScript/JavaScript technical committee).

This case study describes a solution that enables the use of io.js with Azure Web Apps (formerly known as Azure Websites).  Any developer interested in using the io.js fork of Node.js on Azure can learn how in this case study.  A more concise tutorial is available on Azure.com.

Customer Problem

The Node.js fork io.js gained significant attention and popularity, since many Node.js core contributors declared it a more agile, forward-thinking and community-driven implementation of JavaScript I/O.  Within days after the initial release, many popular leading companies announced that they were planning to switch to io.js for better performance.  As a result, cloud providers hurried to support  io.js with their Node.js infrastructure.  Several JavaScript, Node.js and startup community members contacted Microsoft, indicating interest in running io.js on Azure Websites.    Based on that input, we set to work ensuring that io.js runs well on Azure Web Apps.

 Overview of the Solution

Using io.js on Azure Web apps is enabled through a deployment script that a) downloads the latest available stable version of io.js for Windows during deployment and b) configures the runtime to use the downloaded binary as the system-wide Node executable.

The resulting solution is available in three flavors:

• One-click installation via Azure Deploy (also known as Azure Resource Manager) on GitHub
• Inclusion of the deployment payload with any project
• Manual inclusion of an io.js binary, paired with the runtime configuration.

Implementation

Kudu, a Microsoft-sponsored open source project, powers Azure Web App deployments. It allows for custom deployment scripts and can execute Bash, PowerShell and scripts in other languages during the deployment phase. IISNode, the IIS plugin that enables the use of Node.js, can be configured to use a specific binary as Node executable.

Using Kudu and IISNode together allows for an elegant implementation.  A script describing the deployment of the project creates a ‘bin’ folder outside wwwroot and downloads the latest available stable version of io.js. While the script also handles other required deployment tasks, it’s merely 150 lines long. In detail, it starts by accepting the deployment command, copying the new deployment payload to the respective wwwroot folder. During this phase, the script also ensures that Node and npm are installed and accessible. Once deployed, the script moves on to download iojs (see below). Once that is done, Azure Website’s default Node installation routine is executed – meaning that npm modules are downloaded and the environment is configured for IISNode.

:: 2. Dowloading io.js

if not exist "D:\home\site\bin" mkdir D:\home\site\bin

if not exist "D:\home\site\bin\iojs" mkdir D:\home\site\bin\iojs

call :ExecuteCmd curl -L -o D:\home\site\bin\iojs\iojs.exe https://iojs.org/dist/latest/win-x64/iojs.exe

call :ExecuteCmd curl -L -o D:\home\site\bin\iojs\iojs.lib https://iojs.org/dist/latest/win-x64/iojs.lib

Configration of IISNode itself is simple: A file named iisnode.yml in the wwwroot of a website is parsed by IISNode, allowing a text-based configuration of environment variables. One of those variables is ‘nodeProcessCommandLine’, allowing the specification of a binary to be used for Node calls.

nodeProcessCommandLine: "D:\home\site\wwwroot\bin\iojs\iojs.exe"

io.js is npm-compatible, so no changes are required to enable package management.  To get this running in your Azure Website, see the instructions at http://azure.microsoft.com/en-us/documentation/articles/web-sites-nodejs-iojs/.

To run this code yourself, see the full tutorial on Azure.com or directly to GitHub, where clicking the ‘Deploy to Azure’ button automates the process.

Opportunities for Reuse

The code is available on GitHub and referenced as the primary solution for io.js on Azure Web Apps on Azure.com. It has been referenced in various blog posts and "cool technology on Azure" demos.

The quick support for io.js was received well by the community. Simon Bisson (ZDNet, Infoworld) wrote: “It’s good to see Microsoft simplifying the process of working with new open Web technologies on Azure. Scripts like this, while not supported directly, make it possible for developers to experiment with new technologies without affecting production systems. Sharing scripts through Github is an important development, as it gives developers the option of modifying and updating scripts – and for the io.js team to fork the script and adopt it as part of their distribution tools.”

The code can be easily reused for different branches or forks of Node.js and is also the basis for Node 0.12 on Azure.

Felix Rieseberg is an Open Source Engineer in Microsoft's Partner Catalyst team where he spends his time coding on the cutting edge of technology, trying to improve developer's lives by releasing quality code on GitHub and touring the world presenting the lessons we've learned.  Read his blog at http://www.felixrieseberg.com/, follow him on twitter @felixrieseberg, and check out his work on github at https://github.com/felixrieseberg.

By Beat Schwegler, Partner Catalyst Team

In this case study, our customer needed to predict the end location of a trip so that they could effectively manage their car-sharing fleet.  We used Azure ML and Storm to build a model that predicts the end location of individual trips, which customer can then aggregate to most effectively reposition cars as needed to meet expected demand.  If you are a developer interested in location hot spots or techniques for hierarchical geofencing, read this case study to understand both model and implementation of these techniques.

Customer Problem

Our customer runs large scale car sharing fleets across multiple cities worldwide. They need to identify locations with high demand to ensure that enough cars are available for pick-up. To plan the fleet repositioning, it is important to predict possible drop-off locations. Together with their demand prediction, this allows the customer to take pre-emptive actions, such as repositioning cars (e.g. through limited time special offers). In this paper we discuss and implement an approach to predict drop-off locations based on one, real-time transmitted pick-up location.

Overview of the Solution

The overall architecture of the solution leverages a real-time data processing pipeline.

Figure 1: real-time data processing pipeline

The car (or any other device) periodically sends its location which is ingested into the pipeline for further processing. In this solution, we use Storm (http://azure.microsoft.com/en-us/documentation/articles/hdinsight-storm-overview/) to analyze the input stream and generate a summary for each trip, which we store in table storage. Each trip summary contains the following information:

start location, end location, start time, trip duration, driver id

Our approach is based on the assumption that drivers follow certain patterns, such as driving from the train station to the office, or bringing kids to school each weekday morning. The idea is to identify the driver’s most likely drop-off locations based on the current pick-up location.

To implement the location clustering, we created a hierarchical model and computed the cluster that contains the highest number of locations. In such a model, each cluster is divided into four child clusters until we arrive at the required cluster size, the so-called leaf cluster. Each leaf cluster contains its list of locations. To limit the memory footprint and improve performance, we decided not to pre-create the clusters and leaves but to create them only when a location is added to the respective cluster/leaf. Figure 2 to Figure 4 show a representation of the clustering, and a visualization that only clusters containing actual locations are created. The red clusters in Figure 4 highlight the five top leaves (the leaves containing the most locations).

 Figure 2: Visualization of clusters (actual locations are in the Seattle area)  

Figure 4: Zoomed visualization of the 5 top locations (the leaves are marked in red)

One issue with a fixed hierarchical cluster is that the actual top location might be located in the intersection of leaves, and therefore won’t be identified by the model (Figure 5).

Figure 5: Top cluster is in the intersection of three leaves

To address this issue, we simply created a second model which is shifted by half of the leaf size, both vertically and horizontally. We then query the two models for their top locations and remove intersecting leaves by only taking the one with the highest number of locations (Figure 6).

Figure 6: Using a second but shifted model returns the actual intersecting top cluster (the red square)

For the drop-off predictions based on the start location, we perform the following steps:

1. We load all trips for the particular driver into the model and retrieve all trips that started in the same cluster. For improved accuracy, we can filter trips based on day or time before loading them into the model. The solution could also be extended to run filter queries as part of the top location retrieval, so basically passing a LINQ expression as an argument. 

2. We retrieve all trips that started in the same leaf. In the case no previous trip started in the current leaf, we iteratively query the parent cluster, until we find a trip or reach a cluster level where the prediction becomes irrelevant.

3. We now iterate over all the trips that started in the same cluster and create a list of leaves where these trips ended. We then count how many of the trips end in a particular leaf and calculate the probability of the prediction.

Limitations: It is clear that this route prediction only works for drivers who follow a specific pattern, such as frequently driving from the train station to the office. Figure 7 shows the performance of a driver which follows a certain pattern: The prediction accuracy progressively increases. After ~40 trips, the model is able to predict around ~60% of end locations with an accuracy of 200 meters. 

Figure 7: Performance for a driver who follows a regular trip pattern

This is comparison to the second driver (see Figure 8): After only 10 trips, we’re able to predict ~50% of end locations with an accuracy of 200 meters. After around 100 trips, it looks like the driving pattern changes and the accuracy decreases. 

Figure 8: Performance for a driver with changing trip pattern

Implementation

The solution consist of two algorithms: First the hierarchical clustering and second the actual route prediction algorithm (which leverages the clustering for its predictions).

The code for the two algorithms can be found on GitHub: https://github.com/cloudbeatsch/RoutePrediction.

Hierarchical clustering algorithm  

The hierarchical clustering algorithm is based on the composite pattern (Figure 9).

Figure 9: Class diagram - hierarchical clustering algorithm

·        LocationClustering represents the service façade and implements the algorithm to query the top location

·       Each model consists of a single root cluster of type ClusterRoot

·       Each hierarchy level represents a different zoom level and contains one to four child clusters (which are either of type ClusterHierarchy or ClusterLeaf in the case that the minimal cluster size has been reached) (Figure 8)

·       Each leaf contains a collection of LocationInstance objects (of which each contains a location object of type T)

·       The cluster hierarchy can be traversed using a visitor which implements IClusterVisitor

Figure 10: Sample cluster hierarchy

Clustering locations to identify top locations

Using location clustering to identify top locations is straightforward:

1.        1. Create an instance of LocationClustering<T> by typifying the location object and specifying the size of the leaf in meters

       var locations = new LocationClustering<string>(200);  

2. Add locations using the AddLocation method

       locations.AddLocation(latitude, longitude, “<tripId>”);
 
Locations can be added at any time. This allows the model to learn even after the first top leaves have been queried.

3.      3. Once the locations have been added, just query the top clusters using the GetSortedLeafs method. This returns an enumerable with the requested number of top leaves (the top clusters).

foreach (var leaf in locations.GetSortedLeafs(maxNrTopClusters)) {
   foreach (var location in leaf.GetLocations())
   {
      var tripId = location.LocationObject;
   }
}

Predict end locations based on start location

To predict end locations, we build a single cluster per driver using previous trips. We add the start locations and store the trip’s end location as the LocationObject. In the example below, we store objects of type TripSummary (which contains latitude and longitude of the end location):

var locations = new LocationClustering<TripSummary>(200);

locations.AddLocation(startLat, startLon, tripSummary);

Once all the trips have been loaded, the model can be used to predict end locations based on start latitude and longitude.   In its essence, this requires the following steps: 

1. Retrieve the leaf for the start location. In the case the retrieved leaf contains no trip, we iterate over the parent cluster until we have a cluster which contains at least one trip.

2. For each trip in the cluster, we retrieve its end location leaf and count how many trips end in the same leaf.

3. To calculate the probability of our prediction, we divide the number of locations in a leaf by the total number of locations.

public List<RoutePredictionItem> PredictEndLocation(double startLat, double startLon)
{
   var cluster = startLocations.GetLeaf(startLat, startlon);

   int clusterLevel = 0;
   // check how many start points we have within a cluster

   // and iterate through the cluster hierarchy to get at least 1 location
   while (cluster.GetNrOfLocations() == 0){

      var parent = cluster.GetParentItem();

      if (parent != null){

         if (parent is ClusterItem<LocationInstance<TripSummary>>){

            cluster = parent as ClusterItem<LocationInstance<TripSummary>>;

            clusterLevel++;

         }

      }

   }

   Dictionary<ClusterLeaf<TripSummary>, RoutePredictionItem> leafs =     

new Dictionary<ClusterLeaf<TripSummary>, RoutePredictionItem>();

   // create a list of endLocation leafs and count how many times an end location
   // is part of the leaf

   foreach (var location in cluster.GetLocations()){

      ClusterLeaf<TripSummary> leaf =
         location.LocationObject.GetParentItem() as ClusterLeaf<TripSummary>;

      if (!leafs.ContainsKey(leaf)){

         leafs[leaf] = new RoutePredictionItem() {

            Lat = leaf.CenterLat,

            Lon = leaf.CenterLon,

            NrOfEndPoints = 0,

            Probability = 0.0 };

      }

      leafs[leaf].NrOfEndPoints++;
      leafs[leaf].Probability = leafs[leaf].NrOfEndPoints /cluster. GetNrOfLocations();

   }

   var predictions = leafs.Values.ToList();
   // sort the predictions on NrOfEndPoints

   predictions.Sort();

   return predictions;

}

To test the model and measure its accuracy, we split the dataset into a training and a validation set. Now we only load the training data into the model and validate our algorithm by predicting the end locations of our validation dataset. We now simply compare the predictions with the actual values and calculate the distance between them.

Challenges

One challenge we encountered was the creation of trip summaries, basically the extraction of the start and end location from the transmitted data stream. Since GPS data isn’t always accurate, we have to clean the data using different filters to ensure accuracy. To do so, we built a data pipeline and applied different filters (for example, a filter to remove jittering or a filter to recognize one-off measurements). This work will be published in a separate paper.

Another challenge presented was the decimal degrees format of latitude and longitude: Scaling and traversing across the different hierarchy levels is much more intuitive and less error prone when using a projection of metric X and Y axes (e.g. in meters). That’s why decimal degrees are only used at the service boundary, all internal operations are using the projection in meters. The transformation between the different projections have been implemented in a separate library. This geo transformation library can be found on GitHub: https://github.com/juergenpf/Geodesy.

An alternative approach to the proposed solution could be to use SQL Servers’ spatial data capabilities to count start and end locations in predefined geo-fences.

Opportunities for Reuse

The hierarchical location clustering algorithm can be re-used for any solution that needs to identify location hot-spots or requires a hierarchical geo-fencing. The implemented Visitor allows to traverse the hierarchy and enables additional scenarios such as cluster based aggregation or visualization.

The route prediction algorithm represents a simple but yet effective way to predict end points of frequently done trips, and can be used in a variety of different scenarios. The GitHub repository contains a test project which demonstrates route prediction and validation using sample location data. 

Beat Schwegler is a Principal Architect on the Partner Catalyst Team based in Switzerland. Beat enjoys working with organizations of all sizes to develop innovative solutions. He calls himself “head in the cloud – feet on the ground” and loves to share his experiences at conferences worldwide. He blogs at www.cloudbeatsch.com.

By Felix Rieseberg, Partner Catalyst Team

Ember is one of the most popular MVC web frameworks available today - Yahoo, Vine, Groupon, LinkedIn, ZenDesk are only few of the websites built with it. This case study describes the development and usage of ember-cli-windows, a small command-line utility greatly improving the developer experience by reducing build times for Ember users running Windows 10, 8.1, 8, or 7. The tool has now become the official way of running Ember-CLI on Windows.  This technique can be more broadly applied to optimize any scenario involving lots of disk read/write operations.

Customer Problem

A core component of Ember is its powerful tooling: Ember CLI is a command-line client that makes it easier to scaffold, test, develop and build complex web applications. One of the main features is the build process.  Ember-CLI transpiles modern JavaScript code (ECMAScript 6) or JavaScript-like languages (TypeScript, CoffeeScript, etc.) and dependencies into optimized, backwards-compatible ECMAScript 5 JavaScript code. This process is run during development whenever a file change occurs (to render the live preview of the web app), during testing, and when the app is built for production.

On Windows, the build process can be magnitudes slower than on Linux or Mac OS X. Since the build process is executed in response to any local file change during development, Windows developers quickly find themselves staring at a slow “Building…” multiple times in an hour, wondering why the same build only takes seconds on OS X and Linux

Why does this happen?   It turns out that two services, Windows Defender and the Windows Search Index monitor new files and writes to existing files.   These services are valuable and important – Windows Defender protects against viruses, and Windows Search Index ensures that you can find what you’re looking for on your machine.  But this good and valuable work has unintended consequences.  During the build process, many temporary files get created by the various code transpilers. As this happens, Windows Defender and Windows Search Index examine and lock these temporary files. While the two services check thousands of small files, the developer sits and waits.

Overview of the Solution

A solution exists: Since the temporary files are simply copies of existing code, the user can safely disable both the Windows Search Index and Windows Defender for the local tmp folder in an Ember application source folder. 

Ember-CLI-Windows was developed in close collaboration with the Ember team and open source volunteers to automate the full configuration (including the disabling step mentioned above), enabling users to simply run ember-cli-windows in any source folder to reduce build times by up to 70%.

The same practice can be used to optimize the performance of any scenario involving lots of disk read/write operations.  See http://social.technet.microsoft.com/wiki/contents/articles/953.microsoft-anti-virus-exclusion-list.aspx for more details.

Implementation

At the core, Ember-CLI-Windows is a Node.js command-line tool that programmatically invokes PowerShell scripts to determine the current version of Windows and the current user rights.  Then it appropriately configures both Windows Defender and the Windows Search Index.

function runPowershell(scriptPath, name, cb) {

    var child = spawn('powershell.exe', ['-NoProfile', '-NoLogo', "& {& '"+ scriptPath + "' -

path '" + dir + "'}"]), // jshint ignore:line

        output = [];

    child.stdout.on('data', function (data) {

        console.log(data.toString());

        output.push(data.toString());

    });

    child.stderr.on('data', function (data) {

        console.log('Configuration Error: ', data.toString());

        output.push(data.toString());

    });

    child.on('exit', function () {

        console.log(name + ' configured!');

        cb(output);

    });

    child.stdin.end();

}

Snippet 1: Running PowerShell from Node.js (backwards-compatible to Windows 7 

The two PowerShell scripts (setup-defender.ps14 and setup-search.ps15) were initially only written to support Windows 8.1 and later versions, but the open source community quickly stepped in to provide backwards compatibility for Windows 7 (Thanks to bschuedzig, kellyselden, Jackbennet and petetnt!).

Installing and using Ember-CLI-Windows

Ember-CLI-Windows comes in two forms: a standalone command line client for users who are developing multiple applications, and a non-global addon to Ember-CLI. In both cases, Node.js and NPM are required (as they are for Ember-CLI).

To install and execute the standalone version, simply run: 

npm install ember-cli-windows -g

ember-cli-windows

Should you get a PSSecurityException, allow your PowerShell to execute the script. Run the following command as Administrator: 

Set-ExecutionPolicy Unrestricted -scope Process

ember-cli-windows

Installing and using Ember-CLI-Windows as an Ember Addon

Ember-CLI-Windows can be invoked from any Ember-CLI installation. If you’re already in a scaffolded Ember-CLI application, simply install the addon by running the following command:

npm install --save-dev ember-cli-windows-addon

To execute Ember-CLI-Windows, just run:

ember windows

Performance Gained

Mileage varies, but some users were kind enough to share their results. In the case of this Dutch company, the build time went from 41 seconds to 8 seconds.

Challenges

Automating the configuration on Windows 8.1 and later versions was quite easy, while achieving the same goal on Windows 7 was much more difficult. The advances Windows has made in command line automation since Windows 8 were very clear. Before Windows 8.1, edge-cases like special user groups, group policies, folder names with special characters, etc. required some PowerShell magic. In the case of Windows Search, we bundled an Interop DLL to shim features missing in PowerShell v1.

Opportunities for Reuse

The same challenge exists whenever code generators and other tools create and destroy a large number of temporary files in a non OS-level temp folder. While the use case sounds unique, it is common for open-source tools to not rely on the operating system’s “temp” folder so that the tool will operate in a platform-agnostic way. 

Felix Rieseberg is an Open Source Engineer in Microsoft's Partner Catalyst team where he spends his time coding on the cutting edge of technology, trying to improve developer's lives by releasing quality code on GitHub and touring the world presenting the lessons we've learned.  Read his blog at http://www.felixrieseberg.com/, follow him on twitter @felixrieseberg, and check out his work on github at https://github.com/felixrieseberg.

When building applications which require data encrypted at rest and on-the-wire there may be significant complexity added to the software development process.  Organizations building these types of applications must not only invest in the additional resources to write the code, but they must also ensure the implementation is actually secure.

In this document you will learn how you can leverage the Azure Encryption Extensions library to easily encrypt data you store in Azure Blob Storage using Microsoft’s proven Cryptographic Service Providers.  This solution also allows you to upgrade existing Azure Blob Storage code with minimal changes.

The Solution

Azure Encryption Extensions is a simple library designed to streamline the work required to encrypt data stored in Azure Blob Storage. Data is encrypted on-the-fly as it is uploaded to Azure, and decrypted as it is downloaded. Unencrypted data never leaves your machine and you can manage your keys however you'd like.

The library provides a set of wrappers over the public Azure Storage SDK which closely mirror the existing storage methods.  This allows developers to access blob storage in a standard way, and reference existing documentation, while still achieving industry-standard security.

Choosing the Encryption Method

Two core types of encryption are supported.  Symmetric and Asymmetric.  Here we will provide a high level description of how they work, but for more details on the implementation see the links at the bottom of this document.

The asymmetric mode provides encryption when you prefer to use a Public-Private Key pair, such as an X509 certificate. This provider allows you to encrypt data with the public key, and decrypt data with the private key. It is useful in scenarios where you want to let systems encrypt data but not give them access to decrypt it. You can share the public (encryption) key broadly while maintaining the security of your data by protecting the private (decryption) key.

Asymmetric encryption is valued for the flexibility in managing encryption and decryption keys separately, however it can be very slow for larger datasets. 

Azure Encryption Extensions solves this by implementing an industry standard pattern of first encrypting your data with a random symmetric key (very fast) and only using asymmetric encryption to protect the random key.  This random key is then encrypted asymmetrically using your key-pair. The encrypted random key is prepended to the encrypted stream so that it can be recovered as part of the decryption process. This whole process happens on-the-fly during reads and writes with no buffering or disk access required.

The symmetric mode is designed for scenarios where you prefer to use the same key for encryption and decryption. This is often more simple to manage, and performs better than the asymmetric option. The provider performs industry standard AES encryption.  

Storing your keys

Azure Encryption Extensions is intentionally non-opinionated about key storage and permits a wide variety of options for key generation and use. 

For all encryption types default constructors will generate cryptographically random and secure keys, which you can then export easily as a JSON string, or write to a file, and store as appropriate.

Alternatively you can provide your own keys in a variety of formats, including X509 Certificates, binary arrays, or JSON strings.

It is essential that you have an appropriate key management solution for your application, and this should be one of your first design decisions.  Some scenarios may benefit from working with Azure Key Vault while others may have custom key management strategies.

Using the Library

Here we present a simple example of encrypting and uploading a file to Azure Blob Storage, and then downloading and decrypting it.  In this example a typical X509 certificate is used for asymmetric encryption.

 CloudBlockBlob blob = container.GetBlockBlobReference("TestBlob"); 
 
 // Create an Asymmetric provider from an X509Certificate2
 var provider = new AsymmetricBlobCryptoProvider(certificate); 
 
 // Encrypt and upload the file to Azure, passing in our provider 
 blob.UploadFromFileEncrypted(provider, path, FileMode.Open);
 
 // Download and decrypt the file
 blob.DownloadToFileEncrypted(provider, destinationPath, FileMode.Create);

Note that the Upload and Download methods have signatures which are very similar to those from the underlying Azure Storage SDK.  Simply append the word “Encrypted” to the end of the methods to leverage the encrypted version and include one additional parameter to reference the encryption provider.  Defaults for all optional parameters will match the original methods.

Code Artifacts

Azure Encryption Extensions is available as a library released under a MIT license.  It is fully open source on GitHub and compatible with .Net 3.5 or later.

You can install the library via NuGet:

PM> Install-Package AzureEncryptionExtensions

Or get the latest version by syncing the repository directly:

https://github.com/stefangordon/azure-encryption-extensions

Extensive documentation on the encryption implementation and library usage is available in the readme.

Summary

Azure Encryption Extensions is an easy way for teams to add encryption to their application while leveraging Azure Blob Storage.  It helps ensure you implement your security in an industry standard way and permits easy retrofitting of legacy storage applications.  These extensions can be used in client applications ranging from line-of-business desktop solutions to Xamarin based mobile apps.  Additionally, the extensions can be used in server side solutions running in Azure or your own datacenter.

The AllJoyn protocol is an emerging leader in the device communications space. Think of it as “Internet of Things” on your local LAN — connecting your TV, lights, stereo and dishwasher to each other. See www.allseenalliance.org for details about the protocol and the alliance behind it.

With active support from most major electronics manufacturers, devices are starting to be released to the public.  All new LG smart TVs have AllJoyn support, and LifX, an exciting new Kickstarter-funded smart lightbulb is quickly gaining traction.  Qualcomm is building AllJoyn-enabled chipsets to power a wide assortment of upcoming consumer devices.  

Unfortunately, despite significant support from manufacturers, software to actually manage these devices and leverage AllJoyn is almost nonexistent.  OEMs still prefer propriety communications protocols for their applications and third party developers have mostly avoided participating. 

Several barriers have prevented significant software investment in the space and limited consumer adoption:

• No direct cross-platform support in existing AllJoyn code
  • It is possible to leverage the existing code on a variety of devices, but each must be built independently and with little prior art.
• Lack of community involvement
  • The existing code and documentation are extremely complex and inaccessible to most developers.
  • Third-party samples and frameworks are rare.
• AllJoyn Routers are uncommon on most LANs.
  • AllJoyn requires a router or gateway on the local network to facilitate communications.  These have been challenging to setup and out of reach of most consumers.

Overview of the Solution

To bring AllJoyn to a broader audience it is critical to unleash a broad developer community that can quickly build applications and frameworks to support the protocol.  To do this we must lower the barriers to entry in the space by providing solutions that work with common tools and efficient workflows.

Microsoft Technical Evangelism & Development (TED) took the approach of building a fully cross-platform solution that enables developers to rapidly build AllJoyn applications with modern web technologies which can be deployed to iOS, Android, Windows Phone and Windows Desktop from a single code base.

The core of this solution is the AllJoyn Plugin for Cordova.  Apache Cordova is arguably the most robust option for cross platform mobile development in JavaScript and HTML.  It powers PhoneGap, Ionic, AppGyver and many other popular platforms.  This AllJoyn plugin can be used in any Cordova based framework to enable authoring of AllJoyn applications in JavaScript to run across all devices.

The AllJoyn Plugin for Cordova seamlessly ports and compiles the native C AllJoyn SDK for all platforms.  It then wraps the SDK with platform-specific shimming layers, and finally unifies it across all platforms with a simplified JavaScript API.  The plugin provides an abstraction layer that enables developers to simply start writing JavaScript and trust that their code will run across platforms, without worrying about challenges with native code. 

The second facet of our solution is to help spur development by clearly demonstrating approaches to common scenarios.   Three samples have been built using the plugin to illustrate approaches for AllJoyn lighting systems, televisions, and custom peer-to-peer implementations.

Finally, a major contribution from Windows is the inclusion of an AllJoyn Router in Windows 10 – ensuring that any LAN can trivially include the required router functionality by simply connecting a Windows PC or Xbox.

These tools, released in March 2015 at Mobile World Congress, are having an immediate impact on the AllJoyn landscape.  Several partners are currently in development with them and plan to release applications this year to control and manage their devices.

Additional Information

Find the Code

The core plugin lives on Github
https://github.com/AllJoyn-Cordova/cordova-plugin-alljoyn

It can be installed from the Cordova plugin repository, or directly from the repo:

cordova plugin add org.allseen.alljoyn

or

cordova plugin add https://github.com/AllJoyn-Cordova/cordova-plugin-alljoyn.git
  

Learn

We have put together several samples to help you get started.  Each is a Cordova application based on the plugin.  You can clone them; add the plugin; and build/deploy them right away.

Controlling a TV

https://github.com/AllJoyn-Cordova/cordova-tv-alljoyn

See the blog post covering the sample.

As well as an in depth look.

Controlling a Lightbulb

https://github.com/AllJoyn-Cordova/cordova-lighting-alljoyn

See the blog post about the sample.

and a more in depth overview.

Peer to Peer chat

https://github.com/AllJoyn-Cordova/cordova-chat-alljoyn

Stefan Gordon is a Principal Software Engineer on the Partner Catalyst Team where he uses his distributed services background to help build innovative open source tools and platforms.  You can follow his other work at stefangordon.com

Overview

Diagnosing distributed system failures and deadlock conditions can be difficult. This case study provides insight on how to use existing tools from the Microsoft Azure platform. In addition utilities to assist in diagnosing and identification of root cause. With the information provided and the tools as a foundation, readers have additional guidance for identification of root cause, diagnosing and recommending solutions when complex issues arise on distributed solutions running in Microsoft Azure.

By Shawn Cicoria, Partner Catalyst Team

Customer Challenge

The customer in this case was Jet.com which is utilizing a 3rd party storage engine that is an Open Source (OSS) platform providing persistence based upon an event-sourced application pattern. The provider exposes interfaces via REST and JSON in addition to providing client libraries for both .NET and Java; the .NET version is available through NuGet.

Jet.com is an e-commerce startup based in Hoboken, NJ. Jet.com offers a price club model for membership and provides optimization of pricing of shopping carts that looks to maximize savings for members by minimization of shipping and other relate charges. They are in beta period (as of May 2015) with limited insider release and have significant funding and backing.

What follows is an overview and a quick walkthrough of some of the Microsoft Azure tools that assist in diagnostics, along with an overview of a tool utilized to verify in a parallel path whether transient errors are occurring over time.

Underlying Platform Transient Error

During the diagnostics, it was apparent and confirm that there are small outages of a few seconds with Azure Web Sites and the current VNET support. The current implementation of VNET support in Azure WebSites is a Custom VPN layer developed by the Azure networking and WebSites team.

The custom VPN layer used by Azure WebSites, in this case, emitted a particular, and rare, Win32 exception. This error is WSAEADDRNOTAVAIL (“The requested address is not available in its context”) – an error message emitted by the VPN layer when a network becomes invalid. While this error is a defined Win32 error, it is emitted by the custom VPN layer that Azure Web Apps utilizes for VPN to customer VNETs.

For some background, Azure WebSites provide management, command line access, and file browsing on Azure Web Apps, a shared infrastructure component called Kudu is available, running in the browser. Kudu (https://github.com/projectkudu) connects to the web site through a custom implementation of virtual networking using a VPN software solution. Kudu can be accessed using a URL of the form:

http://{yourwebsitename}.scm.azurewebsites.net

Its user interface includes file browsing and a choice of CMD or PowerShell command line access:

Figure 1. Kudu UI

Cloud and Transient Network Errors

Distributed systems and especially those targeting cloud need to be resilient to transient network conditions that occasionally occur between tiers. As Virtual Networks (VNETs) in Azure help bridge varying instances of cloud services, Virtual Machines, and even Azure Web Apps, they are not as reliable as hardwired physical networks using switches.

VNETs can be thought of as software-defined networks that exist as virtual tunnels within physical networks. These virtual tunnels are created to isolate tenants and prevent intermingling of ‘clear’ traffic. Ultimately, they translate to physical layer traffic that is all commingled, but at the IP layer, it is a secure encrypted tunnel. Thus, developers should expect failure at some point and need to design their code to be resilient to network failures.

Provider Client Errors

The symptom of the issue that occurred was that the provider’s client library would eventually hang. Since all interactions with the Website were done through a singleton connection – which was hanging –the entire website stopped working. One recommendation is to provide a “pool” of objects; if one goes into error state, it is ejected from the pool and another is used in its place. However, the implementation didn’t currently do this; additionally, while it may have minimized down time, it would have just masked a core issue in the provider library, not fixed it.

After creating a small repro MVC site and WebJobs[1] for testing the VNET and the Provider’s client, all preliminary indications pointed to a deadlock condition in the provider’s client library .NET code.

Overview of the Solution

Finding the Problem

In the end, the solution was for the Provider to deal with a deadlock situation in their client library managed code. However, given the obscurity of the error code, isolating this issue was initially challenging.

Initial Diagnostics and Tools

The first step was to establish a simple repro of the error. That was challenging to do as the VPN layer used in Azure Web Sites is a custom implementation. However, this was the first step.

A “TCPPing” diagnostic tool was created (see blog post) and the associated GitHub repo that contains the code:

http://blogs.msdn.com/b/scicoria/archive/2015/02/15/troubleshooting-tool-azure-webjob-tcp-ping.aspx

https://github.com/cicorias/webjobtcpping

Diagnostics as a Service

On Standard or Basic mode Azure Web Sites (not Shared), a feature called “Diagnostics as a Service” (DaaS) is available. This can be reached via the Web Site Kudu page:

http://{yourwebsitename}.scm.azurewebsites.net/DaaS

From the page shown below, you can initiate a Memory Dump (other things too). We will then be able to leverage the Windows Debugger, or as done below using the “automatic analyze” feature of DaaS.

Figure 2. Diagnostics as a Service

When running, it’s easiest to choose the ‘Schedule Analysis’ and select Live. To have the Analysis done, ensure you select it:

Figure 3. Creating a Diagnostics Session

Dump Analysis

The analysis is provided as a single MHT file, readable in Internet Explorer. The first page indicates 2 errors, which are insignificant; however, the first warning, highlighted below is an indication of a blocked thread.

Figure 4. Error Analysis

Blocked Threads / Call Stack

Generally a blocked thread with a Monitor.Wait occurs in multi-threading handling when the owner thread has not released an object. Examining the Stack Trace we see that the client code was awaiting error handling from a SocketError condition.

Figure 5. Stack Trace

Debugging Provider’s Client Code

In this case, the Provider’s code is open sourced, so it was relatively easy to download the source and start debugging their code. While not finding the specific issue immediately the clues provided in the DaaS report were quick indications of areas to look int he code. After reviewing the code, we were able to identify an area of complex code that handles and wraps TCP Sockets.

Collaboration with Engineering

Instrumental in validation was working with one of the developers of Azure Web Sites VNET layer at Microsoft, we then acquired a development instance of Kudu and Azure Web Sites running on a dedicated PaaS instance. With this we were able to consistently reproduce the error condition. By working together, engineers from Azure, the Provider, and Jet.com were able to find and diagnose the problem, fix it, test the fix, and release a new build in a timeframe that enabled Jet.com to continue making progress on their upcoming release.

Vendor Updates

The Provider issued a GitHub update to their library, which was then tested and validated in our development environment.

Code Artifacts

Azure Web Sites – Web Job Diagnostic Jobs

This tool is a continuous WebJob that will TCP Ping (open a socket) to a remote address and report back via diagnostic logging if any failures are encountered. This was key in validating that the network was actually still working even when the customer’s client code was not.

https://github.com/cicorias/webjobtcpping 

Web Jobs and Logging

Azure Web Apps offers diagnostic logging – but it’s important to understand where and how it works. Note that Azure Web Sites also will limit logging. Azure Web Sites limit log files to 100MB. Kudu’s real-time console logging is limited based upon the # of lines, which appears to be around 270 lines (but isn’t clearly defined in the documentation).

More information about setting up Diagnostics capabilities and limitations are in the following Azure articles.

• Monitor Web Apps in Azure App Service[2]
• Troubleshoot a web app in Azure App Service using Visual Studio[3]
• Enable diagnostics logging for web apps in Azure App Service [4]

The Kudu console (Admin site) for Azure Web Sites gives access to the folders and directories of the site. It can be used to find and analyze the extensive pre-built logs that provide details regarding the operation of the site.

Logging Helper Class

In any distributed system logging is essential. There’s nothing more helpful than logging that can be turned up for diagnosing what a running system is doing. Given we had limitations with running against a customer instance and initially running remote debugging sessions against the Kudu Development environment, logging provided much of what was needed.

While the Kudu Diagnostics Tool was helpful in identifying the deadlock, logging still provides key information that additionally identified and confirmed that no issues were encountered by the test tool when proper object handling during TCP Socket failures in code. The custom logging was applied within the test tool and test app. The recommendation here is to understand how Web Apps handle logging via Trace or even System.Console (standard output / error) and where that goes inside of Web Apps diagnostics (log files) and the Kudu file structure and limitations.

Here is an example of how to do custom logging with a helper class:

public class Helper
{
 
 public static void LogException(Exception ex)
 {
 if (ex is AggregateException)
 ((AggregateException)ex).Flatten();
 
 var msg1 = string.Format("Exception Occurred: {0}", ex.Message);
 var msg2 = string.Format("EX type: {0}", ex.GetType().ToString());
 
 Console.Error.WriteLine("ERROR: " + msg1);
 Console.Error.WriteLine("ERROR: " + msg2);
 Console.WriteLine("ERROR: " + msg1);
 Console.WriteLine("ERROR: " + msg2);
 Trace.TraceError(msg1);
 Trace.TraceError(msg2);
 }
 
 public static void LogMessage(string message)
 {
 var msg1 = string.Format("NOT AN ERROR: {0}", message);
 
 Console.WriteLine(msg1);
 Trace.TraceError(msg1);
 }
 }

Opportunities for Reuse

Azure Web Sites provide many powerful tools for analyzing and monitoring a site’s use and operations. In a system that uses a VPN, transient network conditions are to be expected, and can make isolating and fixing problems especially difficult. In this case study, we provided both an examination of the built-in tools as well as some tools we built to isolate problems caused by transient network issues. These approaches for identifying core underlying issues can be used in other systems that rely on VPNs, or in any situation where a more detailed understanding of the network status is required.

Additional References and Links

• [1] Azure Web Apps – WebJobs - http://azure.microsoft.com/en-us/documentation/articles/web-sites-create-web-jobs/
• [2] Monitor Web Apps in Azure App Service - http://azure.microsoft.com/en-us/documentation/articles/web-sites-monitor/
• [3] Troubleshoot a web app in Azure App Service using Visual Studio - http://azure.microsoft.com/en-us/documentation/articles/web-sites-dotnet-troubleshoot-visual-studio/
• [4] Enable diagnostics logging for web apps in Azure App Service - http://azure.microsoft.com/en-us/documentation/articles/web-sites-enable-diagnostic-log/

by Beat Schwegler and Ivan Judson, Partner Catalyst Team

The Problem

Our customer uses an on premise Cloudera Cluster to archive and mine customer related data. They wanted to use HDInsight for additional compute capacity and Microsoft Azure Machine Learning to create predictive models. However, their data contains PII (Personal Identifiable Information) and their company policy prevented them from uploading such data to a public cloud.

One possible approach would be to simply remove all PII related data from the dataset. However this is not feasible because building the predictive models and its predictions are related to individual customers. Therefore we explored an approach in which we anonymize data before it is sent to the cloud and de-anonymize it, once the data is back on premise.

Overview of the Solution

We wanted to make the task of anonymizing/de-anonymizing feel like a native Hadoop capability. That’s why we decided use Hadoop’s Pig and extend its capabilities by implementing two user defined functions: ANONYMIZE and DEANONYMIZE. This allows us to embed anonymizing and de-anonymizing as part of an on premise PIG data transformation job, ensuring no PII goes off-premise:

Here an example anonymizing data (in this case the ownerId) as part of a Pig script:

A = LOAD 'data/xyz_device' using PigStorage(';') AS (

ownerId: chararray,
specNr: chararray,

senderId: chararray,

deviceData: chararray);

B = FOREACH A GENERATE jaglion.ANONYMIZE(ownerId), specNr, senderId, deviceData;

jaglion.WASBSTORE B INTO 'data/devices/anonym/xyz_device' USING PigStorage (';');

And here an example of de-anonymizing data:

A = jaglion.WASBLOAD 'data/results/anonym/xyz_result' using PigStorage(';') AS (

ownerId: chararray,
resultData: chararray);

B = FOREACH A GENERATE jaglion.DEANONYMIZE(ownerId), resultData;

STORE B INTO 'data/xyz_result' USING PigStorage (';');

The anonymizing function uses Hadoop’s HBASE as the persistent key/value store to retrieve and store the PII data correlation. This HBASE instance is part of the on premise Hadoop cluster.

We implemented two different correlation modes for different level of privacy:

High privacy mode

Each time the ANONYMIZE function is called, it returns a unique id. In this mode, even if the customerId is the same, the function will return two different anonymized ids. This mode ensures that off premise data can’t be correlated across the anonymized dimension.

ANONYMIZE(customerId) != ANONYMIZE(customerId)

High privacy mode can be used for independent jobs such as pattern recognition, large scale data transformation or model calculations. However it won’t be useful for jobs which gain insight by correlating multiple dependent data items.

Medium privacy mode

The ANONYMIZE function returns always the same anonymized id for the same request. This mode allows for off premise correlations across the same anonymized ids.

ANONYMIZE(customerId) == ANONYMIZE(customerId)

Code Artifacts

The repository for the two user define functions (UDFs) can be found here: https://github.com/irjudson/jaglion.

To use the UDFs, the java package and its dependencies needs to be registered first:

REGISTER /usr/lib/zookeeper/zookeeper.jar;

REGISTER /usr/lib/hbase/hbase-client.jar;

REGISTER /usr/lib/hbase/hbase-common.jar;

REGISTER /usr/lib/hbase/hbase-protocol.jar;

REGISTER /usr/lib/hbase/hbase-hadoop-compat.jar;

REGISTER /usr/lib/hbase/lib/htrace-core.jar;

REGISTER bin/jaglion.jar;

Now, two UDFs can be used within Pig. Assume we loaded the data into A using a statement similar to

A = LOAD 'testdata';

Once A is loaded, the statement below anonymizes the first column in A using medium privacy (the same value will always generate the same anonymized value):

B = FOREACH A GENERATE jaglion.ANONYMIZE($0, 0);

The following statement will generate a unique anonymized value for each value in A, regardless if the source values are the same:

C = FOREACH A GENERATE jaglion.ANONYMIZE($0, 1);

To de-anonymize, we simply call the DEANONYMIZE function:

D = FOREACH B GENERATE jaglion.DEANONYMIZE($0);

E = FOREACH C GENERATE jaglion.DEANONYMIZE($0);

Opportunities for Reuse

The anonymizing and de-anonymizing functions can be used in any Hadoop project that needs to remove PII data from a dataset, while still being able to correlate the anonymized data back to its original PII data.

It can be leverage to enable hybrid cloud scenario or allow a dataset to be shared more broadly, either within the organization or among external partners.

Ivan R. Judson is a software engineer on the Partner Catalyst Team where he leverages deep R&D and startup experience to help partners (including startups!) solve their toughest problems. Passion for data and strong empathy for users are a common theme in his work, which he documents through talks at conferences and his blog at irjudson.org.

Beat Schwegler is a Principal Architect on the Partner Catalyst Team based in Switzerland. Beat enjoys working with organizations of all sizes to develop innovative solutions. He calls himself “head in the cloud – feet on the ground” and loves to share his experiences at conferences worldwide. He blogs at www.cloudbeatsch.com.

DemocracyOS (YC-W15) and Deis on Azure: A TED Case Study

Steven Edouard, Tim Park

The Problem 

Who is Democracy OS?

DemocracyOS (DOS) is a Y-Combinator Winter 2015 cohort non-profit startup that aims to redefine how organizations make decisions, from government to academic organizations. DOS aims to redistribute decision making in organizations by making it easy for members to participate in policy-making, encouraging democracy within their group.

DOS's product is completely open source and takes the form of an Express/Node.js application, backed by MongoDB for persistent data. You can think of the Democracy OS app similar to WordPress in that it's free and open to use; however, to monetize DOS intends to host instances of the application for organizations and charge a subscription fee.

The Problem

The high level problem that Democracy OS has is that they want to deploy their application on Azure with a highly productive developer workflow (like one that Heroku provides and similar to our Azure Websites workflow) on top of a Linux based infrastructure. This paper describes the work we did with them and the Deis open source project on Azure to enable this very common pattern we see across startup companies using open platforms on Azure.

In their solution, Democracy OS must provision a new instance of the application as well as a new database to back the instance. Each application is hosted on a URL with the form:

<organization_name>.democracyos.com

Azure Websites didn't solve this problem for 2 reasons:

1)     Azure Websites runs on Windows and we hit Windows-specific issues such as native modules not building on Windows or paths being too long which was not seen in their OSX & Linux development environments.  This is a common problem with open stack oriented companies with our Azure Websites platform.

2)     DOS wanted to skip building their application on customer deployment and deploy only the prebuilt application bits per instance. At the same time they needed to easily push built versions of the application for each release and make those updates readily available for each instance. Out-of-the-box this wasn't as easy with Azure websites.

Overview of the Solution

What Democracy OS needed was infrastructure that would allow their customers to dynamically create Democracy OS instances quickly. It turns out that a Docker-based solution could solve their problem by distributing a prebuilt application via the Docker hub and using Dockerfiles to deploy their applications.

However, DOS also needed container orchestration and HTTP traffic routing. Because they were already using Heroku, they loved the idea of Deis, an open source cloud which aims to mimic Heroku-style deployments. Powered by Docker containers and the NginX web server, Deis allows easy implementation of the required deployment URL scheme and each customer instance can easily be scaled by any number of Docker containers making each instance horizontally scalable.

The figure below shows the application deployment workflow for a new customer DemocracyOS instance:

The Democracy OS management application uses Deis controller API to automate the deployment of new DemocracyOS.com hosted DemocracyOS instances. They use the Docker hub to deploy the prebuilt version of their app and each customer app pulls the base image and is customized using deployment-specific configuration variables set through the controller api.

Deis runs on CoreOS, a new Linux distribution which is very small, automatically patches itself with minimal downtime and most importantly, runs in a warehouse computing style over an entire cluster of virtual machines. A CoreOS cluster can power an entire Azure Cloud Service however, the Deis platform itself has had a few issues deploying to CoreOS on Azure and documentation & support was still in its infancy.

In particular, they ran into a number of clock skew issues with CoreOS on Azure that caused the distributed config system (etcd) to think the cluster had fatally lost quorum and fail. We engaged the CoreOS team and partnered on a solution that resulted in a bug fix and pull requests to solve the NTP issue in CoreOS build 626. We verified the fix with CoreOS 626, load tested it, and chaos monkeyed the cluster and feel confident in its performance now for future startups. As an aside, and as part of this work, we also migrated the nitrogen.io cluster we are using internally to Deis to get additional operational experience with the Deis/CoreOS/Docker stack for future customers and partners.

How to Programmatically Deploy a 'Wordpress' Style Application on Deis

Democracy OS used the Deis controller api to deploy a the democracy OS base image with deployment-specific variables to control which monogdb database backed the instance as well as other things such as the organization name.

We created an example application which is what the actual Democracy OS instance manager implementation (found here) uses. This walk-through, with more detail can be found here.

We want to use the Deis Controller API because we want the ability to plug the power of deis-scalable apps to any existing infrastructure we have.

For example - if you have a company that hosts an app for people such as Wordpress.com, Drupal or DemocracyOS this would be super useful to plug into your user-facing application.

This repository has an example application using the node.js Deis controller wrapper.

First you need to instantiate a client object and login to the cluster:

var DeisAPI =require('deis-api');

var client =newDeisAPI({

    controller : nconf.get('deis_host'),

    secure     :false,  // Optional

    username   : nconf.get('deis_user'),

    password   : nconf.get('deis_password')

});

We'll use the Inquirer module to extract a few command-line parameters like the Organization Name, Background Color and cpu & memory limits of the application. In a real-world integration you may replace this with a configuration from your database for the specific user or organization you're hosting the app for.

inquirer.prompt([{

  type:'input',

  name:'appName',

  message:'Please specify a name for the application instance to deploy'

},

{

  type:'input',

  name:'orgName',

  message:'Please specify a name for the organization'

},

{

  type:'input',

  name:'bgColor',

  message:'Please specify a background color'

},

{

  type:'input',

  name:'memory',

  message:'Please specify the maximum memory allocated to the instance'

},

{

  type:'input',

  name:'cpu',

  message:'Please specify the maximum cpu time allocated to the instance'

}]

Calling the API

Now that we have all of our information needed to call the api we'll go ahead and call the api wrapper functions, remember this uses a plain old REST API hosted by your deis cluster so you can do this from any language:

Authenticate

Authentication is easy and this is the equivalent as doingdeis login on the cluster:

  console.log('creating app: '+ answers.appName);

  var appName = answers.appName;

  // authenticate

  client.auth.login()

Create the App

This registers the App on deis and allocates a virtual host for the app in the form of<app_name>.<your_domain>.com:

  .then(function() {

    console.log('sucessfully logged into deis');

    return client.apps.create(appName);

  })

However this step just allocates the app space on Deis but doesn't deploy anything. We'll do that in a later step.

Setting Deployment-specific Variables

Now we'll use the questions from the user of the example deployer application to set environment variables for this specific deployment:

  // setup instance-specific environment variables

  // this could be things such as the organization's name

  .then(function(results) {

    console.dir(results);

    console.log('successfully created app: '+ answers.appName);

    console.log('setting deployment specific variables');

    return client.config.set(results.id, {

        ORGANIZATION_NAME: answers.orgName,

        BG_COLOR: answers.bgColor

      },

      {

        memory: { cmd: answers.memory },

        cpu: { cmd:parseInt(answers.cpu) }

      });

  })

Notice how we're setting the environment variables ORGANIZATION_NAME as well as BG_COLOR to the variables collected from the user of this example app. This data however can come from a database, a json file, or anywhere else for your specific use case.

Deploying the Docker Image to the Application

Finally to deploy the image, (in our case, our registry is the Docker hub) we simply specify the image name and Deis will automatically deploy the application. Deis knows when your Docker image exposes an endpoint and will automatically route port 80 or 443 to the virtual host name:

  .then(function(){

    console.log('successfully set deployment-specific variables');

    return client.builds.create(answers.appName, nconf.get('docker_hub_username') +'/simple-node:latest');

  })

  .then(function(results) {

    console.dir(results);

    returnconsole.log('successfully deployed node.js application');

  })

Now after a couple minutes (this application specifically deploys its npm packages on startup) you should see the customized app which is setup to change the background color and display the organization name set in the app settings.

Code Artifacts

The primary artifacts are contributions to external open source projects. We made a contribution of tooling and documentation to the Deis project about how to spin up a Deis cluster on Azure, continue to make enhancements to our CoreOS Linux agent, and helped diagnose issues with both on Azure.

We also created a generic example on how to use the Deis controller api to do 'Wordpress' style deployments on Deis using Docker hub.

Democracy OS also open-sourced their instance manager which handles deploying to Deis via controller apis.

Opportunities for Reuse

There are many opportunities for reuse by any company interested in moving workloads from Heroku (Salesforce) to Azure. Deis also provides a solid devops foundation for running open source workloads from scratch on Azure through its developer friendly 'git push' development model. Developers can learn more about the Deis platform and how to deploy it on Azure at http://deis.io.

Steven Edouard is a full-time hacker & Software Engineer on the Partner Catalyst Team in San Francisco. Most of what he does is open-sourced on github and you can learn more about him at stevenedouard.com.

Tim Park is a Principal Engineer and Silicon Valley site lead for Microsoft.  Tim works in the  Partner Catalyst Team, which works with customers, partners, and open source communities to get the platforms they use to running on Azure.