Poly-what?

While there are many new features and long overdue updates one of the most interesting is the introduction of PolyBase to the SQL Server platform. For those who have not heard or worked with PolyBase, it is not a totally new feature as it has existed on the PDW…err APS appliance for sometime now. So what is it? In short, PolyBase allows you to use your vast array of T-SQL ninja skills to query, join and manipulate unstructured or semi-structure data stored either in HDFS or Azure Blob Storage directly from the SQL Server relational environment all without learning or understanding the first thing about Java, Map/Reduce, Hive or any of the other big data technologies you’ve heard buzzing around. So let’s see how this works.

Hands-On

A a pre-requisite this post assumes you have an Azure account. If you don’t have one….shame on you…just kidding, you can sign-up for an account and even receive a free $200 trial credit. If you have an MSDN membership you may be eligible for a MSDN Azure account that provides $150 a month in Azure credits. With that out of the way let’s build out a SQL Server 2016 sandbox. Begin by signing into the Azure management portal. From the portal home page, click the ‘+ New’ button, then select Compute, Virtual Machine and From Gallery.

The virtual machine gallery consist of all sorts of pre-built VM images that you can select from to simplify the set-up/install process. In our case, we are interested in the SQL Server 2016 CPT2 image found under the SQL Server group.

During the set-up I recommend using a minimum VM size of A4 (8 cores, 16Gb memory) although this can vary depending on the size of the sample data you will be playing. After making you configuration selections, fight back the anticipation as your VM will be spun up and ready to go within a few minutes. Once your VM shows as ready on the Virtual Machine dashboard, you can select it and navigate to the dashboard. From the dashboard, you can use the Connect button to initiate a remote desktop session using the credential you specified during set-up.

Because you use started with a pre-defined image, SQL Server 2016 is already set-up and ready to go sans PolyBase. To add the PolyBase functionality we much run through the installer, but before we can do that, you much update the Java JRE to a version of 7u55 or higher (https://www.oracle.com/technetwork/java/javase/downloads/index.html). After that is complete, launch the SQL Server 2016 installer which is found at: C:\SQLServer_13.0_Full. Click the Next prompts through the process to Add a Feature to an Existing Installation and then simply select PolyBase. The install may take several minutes and then you are really, really ready to go.

Open SQL Server Management Studio (SSMS) and connect to the local, SQL 2016 instance. By default, PolyBase integration is turned off. To enable it use the following command and then restart the SQL Server instance.

EXEC sp_configure 'hadoop connectivity', 4; 
GO 
RECONFIGURE; 
GO

For this demo, we use the 4 option which indicates that we will be using HDInsight or Azure Blob Storage for PolyBase. There are other options for both Cloudera and Hortonworks Hadoop distributes available. After ensuring you have restarted the instance, you are ready to use PolyBase.

Data Source, File Formats and External Tables

These are three parts of PolyBase that you will need to familiarize yourself with and the MSDN documentation is a great reference as you move beyond the simplistic example given in this post. First we will create a data source. Since, we are using an Azure Blob Storage account, we must first create a credential that will be used to pass the access key to our storage account. The script below will create the database credential with the secret key required to access the blob storage account.

DBCC TRACEON(4631,-1);

CREATE MASTER KEY ENCRYPTION BY PASSWORD='MyP@ssword123';

CREATE CREDENTIAL wasb_bluewaterdata 
ON DATABASE WITH
 IDENTITY = 'bluewaterdata', 
SECRET='<MY SECRET KEY />';

Once the credential is in place, we can define a data source which references a specific container on a blob storage account using the above credential like so:

CREATE EXTERNAL DATA SOURCE taxidata
WITH (
    TYPE = HADOOP,
    LOCATION = 'wasbs://<CONTAINER />@<ACCOUNT />.blob.core.windows.net/',
    CREDENTIAL = wasb_bluewaterdata
);

Next, we need to create the file format to tell PolyBase how to read the contents of the blobs. In this example, we are simply reading comma-delimited CSV files so our file format will be as follows.

CREATE EXTERNAL FILE FORMAT CSV 
WITH ( 
    FORMAT_TYPE = DELIMITEDTEXT, 
    FORMAT_OPTIONS ( 
        FIELD_TERMINATOR = ','
    ) 
);

With the two foundational pieces out of the way, we are ready to create a table over top of our “big” data. The example I am using for this post is the NYC Taxi data I used in a previous post. This data as mention before is NYC Taxi trips and fare data in CSV format and includes a header row. I can define an external table that matches the file format as seen below, noting that I’ve referenced my data source, file format and the file folder that contains the data. I also want to point out reject type and value, which in this case accommodates the header row found in the file.

CREATE EXTERNAL TABLE trips 
( 
    medallion VARCHAR(255),
    hack_license VARCHAR(255),
    vendor_id VARCHAR(255),
    rate_code VARCHAR(255),
    store_and_fwd_flag VARCHAR(255),
    pickup_datetime VARCHAR(255),
    dropoff_datetime VARCHAR(255),
    passenger_count INT,
    trip_time_in_secs INT,
    trip_distance FLOAT,
    pickup_longitude VARCHAR(255),
    pickup_latitude VARCHAR(255),
    dropoff_longitude VARCHAR(255),
    dropoff_latitude VARCHAR(255)
) 
WITH 
( 
    LOCATION = '/trips_small', 
    DATA_SOURCE = taxidata, 
    FILE_FORMAT = CSV,
    REJECT_TYPE = value, 
    REJECT_VALUE=1 
)

With my data source, file format and external table created, I can find all three new objects in the SSMS Object Explorer as seen in the screenshot above. I can also query, join or manipulate the data referenced by the new external table like any other SQL Server table all without the data ever residing directly in the SQL Server database. I included a few sample queries that were run directly against the objects created above to round out the demo.

Wrap-Up

PolyBase is one more tool to help you bridge the gap between your relational and non-relational data stores through the use of your existing SQL skill set. I hope this post served as a general introduction to this technologies capabilities and why it is such an important piece to the SQL Server 2016 release. I also want to point out that as this entire post was written for SQL Server 2016 CTP 2, it is all subject to change before go live. Till next time!

Chris

Set-Up

To begin, there are two JAR files needed:

• ersi-geometry-api.jar – contains all the base/foundational classes
• spatial-sdk-hadoop.jar – contains UDF and serde implementations

Since you can download these jar files already compiled from the ESRI GitHub site, I’ll forgo the particulars on building each from its source and get right into the steps required to run these on a HDInsight cluster.

1. Upload both jar files to a container in your blob storage account. Make note of whatever folder/directory structure you use.
2. Use the following to format the path to your the jar files:
   • wasb://<container>@<account>.blob.core.windows.net/<directory>/esri-geometry-api.jar
   • wasb://<container>@<account>.blob.core.windows.net/<directory>/spatial-sdk-hadoop.jar
3. Using the Hive CLI or the HDInsight Query Console, declare both jar files or alternatively this can be done at the time of you spin up your cluster.

   add jar wasb://<container>@<account>.blob.core.windows.net/<directory>/esri-geometry-api.jar;
   add jar wasb://<container>@<account>.blob.core.windows.net/<directory>/spatial-sdk-hadoop.jar;

4. For each function that you intend to use, you declare a temporary function as seen in the example. Note that Hive v0.13 supports permanent functions if you will be making wide spread use of these functions.

   create temporary function ST_Point as 'com.esri.hadoop.hive.ST_Point';
   create temporary function ST_Intersects as ‘com.esri.hadoop.hive.ST_Intersects';

Now your ready to test. To test these functions without a spatial data set we can use two fixed points defined by their latitude and longitude. Since I spend an inordinate amount of time on an airplane, I’ve picked a point near La Guardia International (LGA) airport in New York and a second point near Los Angeles International (LAX) in California. From these two points, we can build a line string which will then allow us to measure the geodesic distance between the two points. The query including the function definitions discussed above looks like this:

set hive.execution.engine=tez;

add jar wasb://gis@bluewaterdata.blob.core.windows.net/lib/esri-geometry-api.jar;
add jar wasb://gis@bluewaterdata.blob.core.windows.net/lib/spatial-sdk-hadoop.jar;

create temporary function ST_Point as 'com.esri.hadoop.hive.ST_Point';
create temporary function ST_LineString as 'com.esri.hadoop.hive.ST_LineString'; 
create temporary function ST_GeodesicLengthWGS84 as 'com.esri.hadoop.hive.ST_GeodesicLengthWGS84'; 
create temporary function ST_SetSRID as 'com.esri.hadoop.hive.ST_SetSRID'; 

SELECT
    ST_GeodesicLengthWGS84(
        ST_SetSRID(
            ST_Linestring(40.773773, -73.872307, 33.944115, -118.40377),
            4326))/1000.0 * 0.62137119 AS DistanceInMiles;

Building a Hive Geospatial Demo

Now that I’ve whet your appetite for the subject matter, let’s turn our attention to building a more real-world demo beginning with the data. There are a number of examples floating around the interwebs that use a mythical set of Uber data. I say mythical because there are bigfoot like spotting of it but it appears as though the data set has been removed from the public domain and is no longer available. That being said a suitable substitute is the NYC Taxi Trip data made available by NYC Taxi Commission and Chris Whong via a FOIL request (read more about it and get the data here). This data set consists of two parts one with trip information (pick-up/drop-off date time and locations) and the second with fare and payment information. To build/run the demo below you will need to download the data from one of its many free sources and then upload it a blob storage account connected to your HDInsight cluster. Now spin up an appropriate sized cluster and let’ go. First let’s create two external tables over top of the data file:

DROP TABLE IF EXISTS TABLE fares;
CREATE EXTERNAL TABLE fares (
medallion STRING,
hack_license STRING,
vendor_id STRING,
pickup_datetime STRING,
payment_type STRING,
fare_amount FLOAT,
surcharge FLOAT,
mta_tax FLOAT,
tip_amount FLOAT,
tolls_amount FLOAT,
total_amount FLOAT
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
STORED AS TEXTFILE
LOCATION 'wasb://<container>@<account>.blob.core.windows.net/fares/'
TBLPROPERTIES ("skip.header.line.count"="1");

DROP TABLE IF EXISTS trips;
CREATE EXTERNAL TABLE trips (
medallion STRING,
hack_license STRING,
vendor_id STRING,
rate_code STRING,
store_and_fwd_flag STRING,
pickup_datetime STRING,
dropoff_datetime STRING,
passenger_count INT,
trip_time_in_secs INT,
trip_distance FLOAT,
pickup_longitude STRING,
pickup_latitude STRING,
dropoff_longitude STRING,
dropoff_latitude STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
STORED AS TEXTFILE
LOCATION 'wasb://<container>@<account>.blob.core.windows.net/trips/'
TBLPROPERTIES ("skip.header.line.count"="1");

Once the tables are in place, verify by issuing a few select statements using the LIMIT clause to verify that everything is primed and ready to go. With the set-up out of the way let’s turn our attention to a use-case. The narrative for this demo is straight-forward: The Contoso Taxi company is exploring new business opportunities and would like to analyze trips that either originated or terminated at La Guardia International Airport during the last year.The primarily challenge as far as this post is concerned is indentifying which trips are in scope for the purpose of analysis. To start, the data for this time period is moderately substantial with about 2gb of data generated each month which makes using Hive beneficial since we can leverage it distributed nature for the purpose of processing. Complicating matters however it the trip data that only contains a latitude/longitude for both pickup and drop-off locations. So how can we go about solving this problem in Hive using the ESRI Geospatial UDFs? First, I used one of the many free mapping tools found on the interwebs to define a polygon that encapsulates La Guardia. This polygon is defined by a series of lat/long points and is represented in well-known text (WKT) format as seen below.

POLYGON ((40.773506 -73.889508, 40.774081 -73.884028, 40.779855 -73.884919, 
40.780605 -73.879043, 40.781755 -73.879373, 40.782805 -73.878614, 
40.78143 -73.87528, 40.786004 -73.871153, 40.785379 -73.869668, 
40.785079 -73.869734, 40.783705 -73.869304, 40.78233 -73.87046, 
40.782255 -73.871648, 40.78063 -73.872804, 40.772881 -73.856991, 
40.772355 -73.856661, 40.770381 -73.858081, 40.76753 -73.852666, 
40.765155 -73.851214, 40.764905 -73.851676, 40.767105 -73.853062, 
40.768781 -73.856694, 40.76703 -73.858707, 40.766805 -73.862768, 
40.770356 -73.867753, 40.771481 -73.870228, 40.771931 -73.872143, 
40.772031 -73.873893, 40.771755 -73.876236, 40.771256 -73.877887, 
40.770505 -73.87967, 40.769606 -73.881321, 40.76853 -73.8834, 
40.767756 -73.885513, 40.76723 -73.887428, 40.767105 -73.888055, 
40.773506 -73.889508))

Using the shape data, I then can create a polygon object using the ST_Polygon UDF. Next, I use the ST_Point UDF to create a point for each trip’s pickup and drop-off location. Finally, I use ST_Contains as a predicate to find trips that either begin or end within the polygon. The resulting abbreviated query looks as follows:

SELECT *
FROM trips
WHERE 
    ST_Contains(ST_Polygon("<OMITTED FROM BREVITY>"), ST_Point(pickup_longitude, pickup_latitude))
OR 
    ST_Contains(ST_Polygon("<OMITTED FROM BREVITY>"), ST_Point(dropoff_longitude, dropoff_latitude))

Since we are interested in doing an analysis only on the trips identified, we would most like load these into a new table making whatever data transformations are needed in the process. The resulting full example including a new partitioned internal table and declarations is provided below.

DROP TABLE IF EXISTS nyc_taxi_analysis;

CREATE TABLE nyc_taxi_analysis (
    medallion STRING,
    hack_license STRING,
    vendor_id STRING,
    rate_code STRING,
    store_and_fwd_flag STRING,
    passenger_count INT,
    trip_time_in_secs INT,
    trip_distance FLOAT,
    pickup_longitude STRING,
    pickup_latitude STRING,
    dropoff_longitude STRING,
    dropoff_latitude STRING,
    trip_year INT, 
    trip_month INT, 
    trip_day INT
    trip_hour INT,
    trip_week INT,
    payment_type STRING,
    fare_amount FLOAT,
    surcharge FLOAT,
    mta_tax FLOAT,
    tip_amount FLOAT,
    tolls_amount FLOAT,
    total_amount FLOAT
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
STORED AS ORC;

set hive.execution.engine=tez;

add jar wasb://gis@bluewaterdata.blob.core.windows.net/lib/esri-geometry-api.jar;
add jar wasb://gis@bluewaterdata.blob.core.windows.net/lib/spatial-sdk-hadoop.jar;

create temporary function ST_Point as 'com.esri.hadoop.hive.ST_Point';
create temporary function ST_Contains as 'com.esri.hadoop.hive.ST_Contains';
create temporary function ST_Polygon as 'com.esri.hadoop.hive.ST_Polygon';

INSERT OVERWRITE TABLE nyc_taxi_analysis PARTITION (year, month, day) 
SELECT
    t.medallion,
    t.hack_license,
    t.vendor_id,
    t.rate_code,
    t.store_and_fwd_flag,
    t.passenger_count,
    t.trip_time_in_secs,
    t.trip_distance,
    t.pickup_longitude,
    t.pickup_latitude,
    t.dropoff_longitude,
    t.dropoff_latitude,
    YEAR(t.pickup_datetime) AS trip_year,
    MONTH(t.pickup_datetime) AS trip_month,
    DAY(t.pickup_datetime) AS trip_day,
    HOUR(t.pickup_datetime) AS trip_hour,
    WEEKOFYEAR(t.pickup_datetime) AS trip_week,
    payment_type,
    fare_amount,
    surcharge,
    mta_tax,
    tip_amount,
    tolls_amount,
    total_amount
FROM Trips t
JOIN Fares f ON t.medallion = f.medallion
AND t.hack_license = f.hack_license
AND t.vendor_id = f.vendor_id
AND t.pickup_datetime = f.pickup_datetime
WHERE 
    ST_Contains(ST_Polygon("POLYGON ((40.773506 -73.889508, 40.774081 -73.884028, 40.779855 -73.884919, 40.780605 -73.879043, 40.781755 -73.879373, 40.782805 -73.878614, 40.78143 -73.87528, 40.786004 -73.871153, 40.785379 -73.869668, 40.785079 -73.869734, 40.783705 -73.869304, 40.78233 -73.87046, 40.782255 -73.871648, 40.78063 -73.872804, 40.772881 -73.856991, 40.772355 -73.856661, 40.770381 -73.858081, 40.76753 -73.852666, 40.765155 -73.851214, 40.764905 -73.851676, 40.767105 -73.853062, 40.768781 -73.856694, 40.76703 -73.858707, 40.766805 -73.862768, 40.770356 -73.867753, 40.771481 -73.870228, 40.771931 -73.872143, 40.772031 -73.873893, 40.771755 -73.876236, 40.771256 -73.877887, 40.770505 -73.87967, 40.769606 -73.881321, 40.76853 -73.8834, 40.767756 -73.885513, 40.76723 -73.887428, 40.767105 -73.888055, 40.773506 -73.889508))"), 
ST_Point(pickup_longitude, pickup_latitude))
OR 
    ST_Contains(ST_Polygon("POLYGON ((40.773506 -73.889508, 40.774081 -73.884028, 40.779855 -73.884919, 40.780605 -73.879043, 40.781755 -73.879373, 40.782805 -73.878614, 40.78143 -73.87528, 40.786004 -73.871153, 40.785379 -73.869668, 40.785079 -73.869734, 40.783705 -73.869304, 40.78233 -73.87046, 40.782255 -73.871648, 40.78063 -73.872804, 40.772881 -73.856991, 40.772355 -73.856661, 40.770381 -73.858081, 40.76753 -73.852666, 40.765155 -73.851214, 40.764905 -73.851676, 40.767105 -73.853062, 40.768781 -73.856694, 40.76703 -73.858707, 40.766805 -73.862768, 40.770356 -73.867753, 40.771481 -73.870228, 40.771931 -73.872143, 40.772031 -73.873893, 40.771755 -73.876236, 40.771256 -73.877887, 40.770505 -73.87967, 40.769606 -73.881321, 40.76853 -73.8834, 40.767756 -73.885513, 40.76723 -73.887428, 40.767105 -73.888055, 40.773506 -73.889508))"), 
ST_Point(dropoff_longitude, dropoff_latitude));

Running the above script on a 16-node cluster and it completed in 638 seconds (your results may vary). Query the results to sample/review and ensure everything is in order.

Wrap-Up

In this post I introduced the ESRI Geospatial UDF library and the steps needed to run in on a HDInsight cluster. I discussed building a demo on real-world data to highlight a portion of the capabilities these functions provide. I hope you were able to take away at least a high level overview of the geospatial capabilities available within Hive and that should the need arise in a future project you can leverage these capabilities as appropriate. Till next time!

Chris

$AzureModule = Get-Module -Name Azure -ListAvailable
$LatestRelease = Invoke-RestMethod 
    -Uri https://api.github.com/repos/Azure/azure-powershell/releases/latest -Method GET

If ($AzureModule.Version.ToString() -ne $LatestRelease.name)
{
    $OutFile = "{0}\{1}" -f $env:TEMP, $LatestRelease.assets.name
    Invoke-WebRequest -Uri $LatestRelease.assets.browser_download_url -OutFile $Outfile -Method Get
    Start-Process -FilePath msiexec.exe -ArgumentList @("/i $OutFile /qb") -Wait -PassThru
}
else
{
    Write-Host "Azure Powershell module is up to date."

}

Till next time! Chris

My vision of the project is pretty simple and is roughly modeled after the built-in Execute SQL and Web Service Tasks. First, since the Azure ML batch execution service requires the input data to be hosted in an Azure Blob Storage account, the task would be capable of either uploading the required input data to an Azure Blob Storage account or simply using an URI to an existing blob. If the input source is not an existing blob, that data could come from a variable, a file connection or via direct input.

Next, the task would execute and monitor the batch prediction job via the end point provided. The task would also periodically log the status for the purpose of visibility. When the job completes, the task would allow the results to be downloaded to either a string variable or file connection.

To start, I re-purposed the Azure Storage Connection Manager found in SSIS Azure Blob Storage sample project on MSDN. The project itself consists of just a single task which I called ExecuteAzureMLBatch and its supporting classes. While a deep dive into the code is outside the scope of this post, those that are interested can download the project directly from my Github project: https://github.com/bluewatersql/AzureMLSSIS.

Building the ExecureAzureMLBatch Task

1. Download the code from https://github.com/bluewatersql/AzureMLSSIS

2. The solution consists of two projects: one for the connection manager and one for the task itself. The build events for both projects handle all the heavy lifting for using the components on your location machine. Ensure the path to GacUtil.exe as well as your SQL Server installation found in the Post-Build Events is correct for your machine. Note that you need to do this for both projects.

3. Build the project. The post-build events discussed in step 2 take care of registering the required assemblies in the GAC and copying the files to the required directories. If you get any squirrely errors, you may need to run Visual Studio as an Administrator to successful run GacUtil.

Using the ExecuteAzureMLBatch Task

After the solution has been successful built, your ready to use the task in an SSIS package. To use the task successfully, use the following steps:

1. Add a new Windows Azure Storage connection manager to your package. This connection does not have a design-time UI for configuration so you will need to use the properties window to set the following properties for where experiment sample data will live:
• UseDevelopmentStorage = false
• StorageName = <STORAGE_ACCOUNT />
• AccessKey = <STORAGE_ACCOUNT_ACCESS_KEY />
• Path = <CONTAINER_NAME /
2. Add the ExecuteAzureML task to the control flow design surface and configure.
   • Set the Azure ML Batch URL Endpoint (ex: https://ussouthcentral.services.azureml.net/workspaces/<….>/services/<…>/jobs)
   • Set the Azure ML Key
   • The experiment input source can come from a variable, a file connection or by direct input.
   • The experiment destination can be none, variable or a file connection and determines where the results are loaded to. Note that when none is specified the results are not downloaded.
   • 
3. Run your package

Once the rask runs, you can process the results as required. I hope you find this as interesting as I do and feel free to let me know if you have suggestions, feedback or would like to actively contribute.

Till next time!

Chris

Please note that I will not be getting into line-by-line code examples outside of those that are relevant to the Storm deployment. Likewise, I will not get too deep into any of the peripheral technologies involved. I will leave those for either a later blog post or an exercise for you the reader. The complete project and source will be posted to GitHub (HERE).

The rationale for this project is pretty straight-forward. As a father to young kids, I became acutely aware of swear words and foul language when my kids reach the parroting stage. For those without kids, this is the stage where a two or three year old hears a word and randomly begins inserting it into their daily vocabulary (typically at the most inopportune times, like when mom is around). This offers many humorous and potentially embarrassing moments depending on the environment.

To play off of this and since Twitter is useful for all sorts of things beyond letting people know what you are doing minute by minute, I decided to do some seemingly simple swear word analysis on people’s tweets. This analysis would hopefully allow me to answer questions like:

• Which parts of the world have the worst potty mouths?
• In what context (sentiment) do people use swear words?
• Can I identify topics or hashtags that people curse about the most?
• Who are the top 10 worst potty mouths on Twitter?

Luckily, all of these question can be easily answered and since it would be nice to do it in real-time, Apache Storm is perfectly suited to the task given the volume and velocity of data in the Twitter stream.

Project Overview

The bulk of this demonstration project, which I will refer to as Twitter Tattletale revolves around common ETL types of activities you would expect to find in a non-streaming, traditional BI/batch processing solution. In fact for this pseudo phase one implementation I will ignore streaming analytics entirely (and a fancy, pretty UI) so we can even break it down into ETL steps. Please understand, that even though I am comparing this to a more traditional ETL process, these steps occur in an unbounded manner, meaning that instead of a fixed beginning and end like that of a SSIS of Hadoop job they run continuously until terminated.

Extract

The entry point for this project is an Azure Service Bus queue. An external application handles consuming the Twitter stream and publishing brokered messages to the queue so our solution will simply need to ingest these messages.

Transform

There are multiple cleaning, filtering, enrichment and transformation steps that occur within this project. The process starts by filtering tweets so that only those in English are processed. Next, external services are used to determine the tweet’s sentiment and to reverse geocode the location based on the latitude and longitude coordinates if provided. Then, the tweet text is split into words for cleaning and indexing. Noise words, special characters, user mentions and urls are all stripped away before determining if the tweet is vulgar by counting the instance/number of swear words.

Load

Rather than loading a dimensional data warehouse, the tattletale project stores data in multiple formats each with its own purpose. Raw tweet data is stored so that it can be process later in a more traditional batch process such as with HDInsight (Hadoop). Aggregated data from the stream is persisted for both the tweet and extracted topics in a database (HBase). Now that the work has been defined, let’s move on.

Technologies and Considerations

There are some miscellaneous pieces that are ultimately external to Apache Storm but are important nonetheless. First, is the event producer which is responsible for capturing and storing tweets to the Service Bus queue. The application uses the Tweetinvi API to capture streaming tweets. For the demo I have implemented this as an Azure Worker Role, though it couple be easily run as a console based application on your desktop. The important thing to remember is that this event producer could ultimately be anything and the source of the stream could come from virtually anywhere. Next is Apache HBase. HBase is a distributed NoSQL database built on top of HDFS and based largely on Google Big Table. It is a columnar database, optimized for high volume, high velocity real-time random red/write access. Whew…that was a mouth full. Please note that the database you choose does not matter to Apache Storm. Outside of demo-land your choice should be driven by the requirements of whatever it is you are trying to build and implement.

SCP.Net

Now on to the good stuff. Before discussing the project specific topology, it is necessary to introduce SCP.Net (Stream Computing Platform). Storm, like most Apache projects is largely Java centric. Most of the documentation and/or examples that you’ll run across are written in either Java or Python. This abstraction allows you to build a Storm topology and deploy it a HDInsight Storm cluster either purely in .Net or in a hybrid deployment using a mix of Java and .Net. The link referenced above does a good job of explaining SCP.Net in greater detail and you can get started using it by installing HDInsight tools for Visual Studio in the authors follow-up post.

The Twitter Tattletale Topology

Recalling from the prior post that a Storm Topology consists of streams, spouts and bolts. Let’s start with a picture (below) and then take a look at each piece of the topology for the Twitter Tattletale demo.

1. Topology – the topology is a simple non-transactional topology, meaning that we do not necessarily care that data is processed in the order it was received.
2. Twitter Spout - reads brokered messages from the Azure Service Bus queue. The brokered messages consist on tweets serialized as JSON. The spout extracts the unique tweet id and tweet language and then filters out non-English tweets before emitting a tuple into the data stream. The tuple consists of the unique id and tweet JSON. Note that in an ideal (read non-demo) situation, the spout would be more generic than it is. The business of filtering should be done in a bolt, so that the spout could be potentially reused by other topologies.
3. Blob Writer Bolt – as the name implies, this bolt runs independent of the primary processing stream to write raw tweets (JSON) to a configured container in a Azure Blob Storage account. The bolt writes one blob per tweet using blob delimiters to annotate year, month and day of the tweet so the data set can be easily partitioned in Hive for later processing. I intentionally did not perform any aggregation meaning you would potentially wind up with millions of 1k files. This is not ideal for Hadoop batch processing. A batch process could be added to handle this and any compression (gzip/lzo).
4. Sentiment Bolt – calls an external REST-based service to determine the tweet’s sentiment. Currently the bolt uses the Sentiment140 API to classify each tweet as positive, negative or neutral. In the future (i.e. when I have time), I will leverage an AzureML service. The sentiment bolt emits a new tuple that consist of the unique id, serialized tweet and sentiment.
5. Geography Bolt – calls the Bing Maps API location service to reverse geo-code a tweets location when a latitude and longitude are provided. The Bing Maps API returns AdminDistrict, AdminDistrict2, CountryRegion, Locality and PostalCode all of which are emitted with the unique tweet id in a new data stream.
6. Merge Bolt – unions or joins the split processing workflow back into a single unified workflow. Since the sentiment and geography bolts both depend on external services, the decision was made to perform both operations in parallel. The merge bolt use field grouping on the unique tweet id to ensure that tuples with same id are routed to same worker.
7. Swear Word Bolt – handles most of the heavy lifting for this streaming process. This bolt prepares tweets by stripping out special characters, emojiis, urls and user mentions before splitting the tweet into unigrams and bigrams. Noise words are removed and a swear word dictionary identifies and counts all swear words. Two data streams are emitted from this bolt: tweets and topics.
8. HBase Tweet Bolt – writes enriched tweets to a HBase tweets table. Internally it uses an queue to implement micro-batching that is trigger by either batch size or a system tick tuple that is fired by Storm at a configured interval. The decision to micro-batch writes was made due to the fact that the C# HBase API is implement using the HBASE REST API (formerly Stargate).
9. HBase Topic Bolt – writes tweet topics with both sentiment, geography and a flag to indicate if swear words where found within the tweet context. Like the HBase Tweet Bolt this bolt also implements the micro-batching pattern, rather than individual writes.

Running the Demo

Since the demo consist of multiple pieces, there are prerequisites that have to be satisfied before running the Storm topology.

1. Azure Storage Accounts need to be created for the various Azure services used.
2. An Azure Service Bus namespace needs to be defined and a Queue to hold the tweets created.
3. A Cloud Service needs to be provisioned and the application which captures the Twitter stream should be deployed as a Worker Role.
4. Provision a HDInsight HBase cluster and Remote Desktop needs to be enabled if you want to use the HBase Shell to issue queries.
5. Provision a HDInsight Storm cluster

You probably noticed that I left out compute or sizing specifications for some of the prerequisites. This is intentional since how much “horsepower” you need depends on what you are trying to do with the demo and we will discuss scaling in a later post. After the prerequisites are in place, your topology can be deployed directly from Visual Studio using HDInsight Tools (‘Submit to Storm on HDInsight’ found on the project context menu). The add-in connects to your Azure account, finds your Storm cluster, builds the topology spec file (based on the TopologyDescriptor class marked as active), zips up the required files and finally deploys everything to the cluster.

One the topology has been submitted and is running, you can monitor it from the Storm Dashboard found in the Azure Portalor by using the Storm Topology Viewer in Visual Studio.

To see what Twitter Tattletale is actually doing, you can query HBase using the HBase shell, an HBase API or by using Hive. Creating a Hive table over HBase is by far the easiest approach (although least performant approach). The script below will create two Hive tables: one for tweets and one for topics.

DROP TABLE IF EXISTS hbase_tweets;
CREATE EXTERNAL TABLE hbase_tweets(
rowkey string, screenname string, country string, created_date string, sentiment string, swearwords int)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
'hbase.columns.mapping' = ':key,user:screenname,loc:country_region,tweet:created_dt, tweet:sentiment,tweet:swear_word_count')
TBLPROPERTIES ('hbase.table.name' = 'tweets');

DROP TABLE IF EXISTS hbase_topics;
CREATE EXTERNAL TABLE hbase_topics(
rowkey string, country string, created_date string, sentiment string, vulgar_tweet boolean)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
'hbase.columns.mapping' = ':key,tp:country_region,tp:created_dt,tp:sentiment,tp:vulgar_tweet')
TBLPROPERTIES ('hbase.table.name' = 'topics');

We can now query the tables to browse the data being collected, scrubbed and enriched using simple Hive queries.

--Summarize tweets by country SELECT COUNT(*) AS TotalTweets, SUM(IF(swearwords > 0, 1, 0)) AS VulgarTweets, SUM(swearwords) AS TotalSwearWords, country FROM hbase_tweets WHERE country IS NOT NULL GROUP BY country ORDER BY VulgarTweets DESC;

--Top 15 negative bigrams by country SELECT SPLIT(rowkey,'[_]')[0] AS key, country, COUNT(*) AS TopicCount FROM hbase_topics WHERE sentiment='Negative' AND SIZE(SPLIT(SPLIT(rowkey,'[_]')[0],'[ ]')) > 1 GROUP BY SPLIT(rowkey,'[_]')[0], country ORDER BY TopicCount DESC;

--XBox tweets by country SELECT country, COUNT(*) AS TopicCount FROM hbase_topics WHERE SPLIT(rowkey,'[_]')[0] = 'xbox' GROUP BY country ORDER BY TopicCount DESC LIMIT 15;

Credits

Before I wrap-up, I need to give credit where credit is due: Analyzing real-time Twitter sentiment with HBase in HDInsight

Wrap-Up

In this post, I introduce the basis and topology for the my Storm demonstration project: Twitter Tattletale. Although the subject matter of the demo is meant to be light-hearted and humorous the goal of this post (and the next few) is to introduce you to this very powerful and flexible Azure offering that presents new options for handle real-time streaming data in a distributed processing environment .In my next post, we will dig into the code, talk testing and finally deployment. Till next time!  

Chris

History & Storm Use Cases

Apache Storm was originally developed by Nathan Marz and the BackType team. BackType was later acquired by Twitter, who open-sourced the project. Storm is designed to be fast (benchmarked at 1 million 100 byte messages per second per node on commodity hardware), scalable and reliable with unbounded streaming data. It is also easy to use as it supports a number of different programming languages (Java, Python, Ruby, JavaScript and C#). These characteristic make it a perfect platform for handling distributed real-time event processing. Some of the most common real-time processing scenarios currently in place are:

1. Financial fraud detection
2. Network outage and/or intrusion detection
3. Preventative maintenance in manufacturing and transportation
4. Application monitoring
5. Customization or personalization of web content

Architecture

An Apache Storm architecture follows a master/slave architecture in a manner similar to Hadoop. The architecture consists of three different types of nodes who all function in different roles to deliver at-least once guaranteed message processing (exactly once message processing is also support through an abstract called Trident).

These nodes are:

• Nimbus (the master node) – a single daemon processes responsible for deploying, managing, coordinating and monitoring topologies running on the cluster. The master node assigns or reassigns all work when failures occur and is also responsible for distributing any required files to the supervisor nodes
• ZooKeepers – In a distributed environment, centralized event and information coordinate is critical to processes coordination. Apache ZooKeeper functions in this role within the architecture, storing state information such as work assignments and job status between Nimbus and Supervisor nodes.
• Supervisors (the worker nodes) – a daemon processes that spawn new task and monitor the status of workers. When Nimbus assigns a task to a Supervisor, the Supervisor spawns a new JVM process to execute each job.

Terminology

Getting started with Storm requires that you familiarize yourself with the terminology you’ll encounter as you begin building a distributed streaming processing solution.

Topologies

Analogous to a job, such as those found in more traditional or batch processing systems. Topologies are made up of data streams, spouts and bolts and are deployed to and run by an Apache Storm cluster. Topologies can be either defined as transactional (strictly defined order of events) or non-transactional and run indefinitely.

Data Streams

An unbound stream of tuples (a collection of key/value pairs), which serve as the most basic data structure within storm.

Spouts

The entry point into a Storm topology. A spout connects to a source of data, such as a database or a service bus event queue, transforms the data into a tuple and then emits the tuple for consumption and processing by one or more bolts.

Bolts

Responsible for transforming and processing data within your topology. Bolts can receive one or more data streams as inputs from spouts or even other bolts and can produce zero, one or more data streams as output. Common activities within bolts are calculations, filtering, aggregations, joins and writes to external sources such as a traditional relational or no-sql databse or even trigger a connected application event through web sockets or SignalR. When taken together these pieces allow you to easily implement a range of real-time processing systems. For example, suppose we captured tweets or Twitter messages and stored them on a Service Bus queue. A simple topology would start with a spout reading the tweets from the queue and emitting them for processing as tuples into the data stream. Subsequent bolts could be used to store the raw tweet to Azure Blob Storage for later consumption by a Hadoop cluster, while a second bolt could add value to the tweets by determining the tweet’s sentiment. Aggregation could be done on some arbitrary field such as hashtag or location. That aggregated level of data could then be pushed to an HBase database as well as a real-time dashboard built using SignalR. A visual repesentation of the previously describe topology is seen below.

Getting Started

So you’ve made it through the overview and your itching to get started, but noticed that I never discussed what it takes to set-up a Storm cluster. Rather than an inadvertent oversight this was deliberate since Storm clusters can be easily provisioned as a service within Microsoft Azure. If you are familiar with provisioning an HDInsight cluster, the process is nearly identical. For those that are not familiar with Hadoop, you will find the Storm option under the Data Services and HDInsight within the add menu as seen below. Using the quick create, you can specify a cluster name, size, subscription, storage account and an admin password and be off an running.

Wrap-Up

This post was foundational as we introduced Apache Storm. We reviewed the basic architecture and discussed common terminology you need to know to get started. Over the next couple of posts we will look at the steps necessary to build and deploy an example topology using C#. Till next time!

Chris

The Set-Up

The output of the Mahout MapReduce job is a file of recommendations in a pivoted format. The format of the file is as follows which each user represented on a single line:

{USERID}    [{BEERID}:{ESTIMATEDRATING},{BEERID}:{ESTIMATEDRATING},{BEERID}:{ESTIMATEDRATING},....]

In order to make use of these recommendations we need to unpivot the data such that each user/beer recommendation is on a single line in comma-separated value (CSV) format. To do this we will use a couple of iterations of Hive queries to transform and filter the recommendations before loading.

Creating the Hive Job

The transformation of the Mahout generated recommendations into a CSV file occurs in four steps. I have documented the steps below but will leave the detail explanation of how Hive queries work for some other time.

Step 1 - Create a Hive table over-top of the Mahout output

DROP TABLE IF EXISTS RecommendationsRaw;
CREATE EXTERNAL TABLE RecommendationsRaw (
    USERID INT,
    RECOMMENDATIONS STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
LOCATION '${hiveconf:MahoutInput}';

Step 2 - Strip the square-brackets (‘[‘ and ‘]’) and load data into a clean staging table

DROP TABLE IF EXISTS RecommendationsStaged;
CREATE EXTERNAL TABLE RecommendationsStaged (
    USERID INT,
    RECOMMENDATIONS STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
LOCATION '${hiveconf:MahoutStaging}';

INSERT OVERWRITE TABLE RecommendationsStaged
SELECT
    USERID,
    REGEXP_REPLACE(
        REGEXP_REPLACE(RECOMMENDATIONS, 
                "\\[", ""), 
            "\\]", "") AS RECOMMENDATIONS
FROM RecommendationsRaw;

Step 3 – Create a new Hive table over leverage Hive’s support for complex types (arrays and stucts)

DROP TABLE IF EXISTS MahoutRecommendations;
CREATE EXTERNAL TABLE MahoutRecommendations (
    USERID INT,
    RECOMMENDATIONS ARRAY<STRUCT<BEERID:STRING,RATING:STRING>>
)
ROW FORMAT DELIMITED 
FIELDS TERMINATED BY '\t'
COLLECTION ITEMS TERMINATED BY ','
MAP KEYS TERMINATED BY ':'
LINES TERMINATED BY '\n'
LOCATION '${hiveconf:MahoutStaging}';

Step 4 – Unpivot the data and write to a CSV formatted file

DROP TABLE IF EXISTS Recommendations;
CREATE EXTERNAL TABLE Recommendations (
    USERID INT,
    BEERID INT,
    RATING STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
LOCATION '${hiveconf:RecommendationsOutput}';

INSERT OVERWRITE TABLE Recommendations
SELECT
   USERID,
   r.BEERID,
   r.RATING
FROM MahoutRecommendations
LATERAL VIEW explode(RECOMMENDATIONS) RECOMMENDATION_TABLE AS r;

Taking all four pieces of the script, we assemble them into a single file with an *.hql extension. The file should then be uploaded your blob storage account. The Azure Data Factory pipeline will reference this script based on the configuration described in my previous blog post.

Creating the SQL Server Table Type & Procedures

To efficiently load data back into our Azure SQL Database we use the definition of a custom table type and a stored procedure. Our recommendation data is then passed as a table variable to the stored procedure which is ultimately responsible for handling the heavy lifting and performing any last transformations required to load the data into the database table structures. The table-type and stored procedure DDL are listed below and will be reference later when we specify our Azure Data factory pipeline.

CREATE TYPE [dbo].[RecommendationType] AS TABLE(
    [UserID] [int] NULL,
    [BeerID] [int] NULL,
    [EstimatedScore] [decimal](18, 2) NULL
)

CREATE PROCEDURE [dbo].[IngressUserRecommendations]
    @Recommendations dbo.RecommendationType READONLY
AS
    BEGIN TRY
        BEGIN TRAN
            DELETE FROM Recommendation

            INSERT Recommendation (UserID, BeerID, EstimatedScore)
            SELECT
                UserID, BeerID, EstimatedScore
            FROM @Recommendations
        COMMIT TRAN
    END TRY
    BEGIN CATCH
        IF @@TRANCOUNT > 0
        BEGIN
            ROLLBACK TRAN
        END
    END CATCH
GO

Wiring in the Azure Data Factory Pipeline

Recall from my prior post that we defined a MahoutRecommendations dataset that acts as the input into this new pipeline. Before we create the pipeline however, we need to define two new datasets. The first dataset represents the output generated by the fourth step of the Hive job in out Azure Blob Storage account. For this dataset, we need to define a schema structure since it will be used to map columns and load the data back into our Azure SQL database. The JSON configuration is provided below.

{
    "name": "Recommendations",
    "properties":
    {
        "structure":  
        [ 
            { "name": "UserID", "type": "INT"},
            { "name": "BeerID", "type": "INT"},
            { "name": "EstimatedScore", "type": "STRING"}
        ],
        "location": 
        {
            "type": "AzureBlobLocation",
            "folderPath": "dfdemo/recommendations/{Year}_{Month}_{Day}/",
            "format":
            {
                "type": "TextFormat",
                "columnDelimiter": ","
            },
            "linkedServiceName": "StorageLinkedService",
            "partitionedBy":
            [
                { name: "Year", value: { type: "DateTime", date: "SliceStart", format: "yyyy" } },
                { name: "Month", value: { type: "DateTime", date: "SliceStart", format: "%M" } },
                { name: "Day", value: { type: "DateTime", date: "SliceStart", format: "%d" } }
            ]
        },
        "availability": 
        {
            "frequency": "Day",
            "interval": 1            
        }
    }
}

The second dataset acts as our connection back to the Azure SQL Database. Note in the JSON below, I went ahead and specified the Recommendations table in the dataset, but that will not be used by our pipeline.

{
    "name": "RecommendationsSQL",
    "properties":
    {
        "location":
        {
            "type": "AzureSqlTableLocation",
            "tableName": "Recommendations",
            "linkedServiceName": "AzureSqlLinkedService"
        },
        "availability": 
        {
            "frequency": "Day",
            "interval": 1            
        }
    }
}

Finally, we define the pipeline. This pipeline use the MahoutRecommendations dataset as a input and consists of two activities: a HDInsight Activity and a Copy Activity. The HDInsight Activity references the HDInsight Linked Service defined in the previous blog entry and uses the Hive transformation to execute the *.hql script we defined above. The extended properties node is used to pass the various Hive configuration properties required by the script. The Copy Activity handles the egress of data from the Hive transformation back to the Azure SQL database. In this configuration we reference our two new datasets and configure the source and sinks for the movement. In the sink definition we reference the new SQL Table Type and Stored Procedure created above. The entire JSON configuration winds up looking as follows:

{
    "name": "EgressPipeline",
    "properties":
    {
        "description" : "Unpivot the Mahout Recommendation output and load to Azure SQL Database",
        "activities":
        [
            {
                "name": "Unpivot",
                "description": "Unpivot Mahout Recommendation output",
                "type": "HDInsightActivity",
                "inputs": [ {"name": "MahoutRecommendations"}],
                "outputs": [ {"name": "Recommendations"} ],
                "linkedServiceName": "HDILinkedService",
                "transformation":
                {
                    "type": "Hive",
                    "extendedProperties":
                    {
                        "MahoutInput": "$$Text.Format(
                            'wasb://<CONTAINER>@<ACCOUNT>.blob.core.windows.net/mahout/recommendations/{0:yyyy_MM_dd}/', 
                                Time.AddHours(SliceStart, 0))",
                        "MahoutStaging": "$$Text.Format(
                            'wasb://<CONTAINER>@<ACCOUNT>.blob.core.windows.net/mahout/stage/{0:yyyy_MM_dd}/', 
                                Time.AddHours(SliceStart, 0))",
                        "RecommendationsOutput": "$$Text.Format(
                            'wasb://<CONTAINER>@<ACCOUNT>.blob.core.windows.net/recommendations/{0:yyyy_MM_dd}/', 
                                Time.AddHours(SliceStart, 0))"
                    },        
                    "scriptpath": "dfdemo\\scripts\\unpivotrecommendations.hql",                
                    "scriptLinkedService": "StorageLinkedService"
                },
                "policy":
                {
                    "concurrency": 1,
                    "executionPriorityOrder": "OldestFirst",
                    "retry": 1,
                    "timeout": "01:00:00"
                }
            },
            {
                "name": "EgressData",
                "description": "Egress recommendations to Azure SQL DB",        
                "type": "CopyActivity",
                "inputs": [ {"name": "Recommendations"} ],
                "outputs": [ {"name": "RecommendationsSQL"} ],    
                "transformation":
                {
                    "source":
                    {                               
                        "type": "BlobSource"
                    },
                    "sink":
                    {
                        "type": "SqlSink",
                        "SqlWriterTableType": "RecommendationType",
                        "SqlWriterStoredProcedureName": "IngressUserRecommendations"
                    }            
                },
                "Policy":
                {
                    "concurrency": 1,
                    "executionPriorityOrder": "OldestFirst",
                    "style": "StartOfInterval",
                    "retry": 0,
                    "timeout": "01:00:00"
                }
            }
        ]
      }
}

As in the previous post, to update the data factory with all the new pieces we use PowerShell.

Wrap-Up

Using the Azure Portal, you can browse the completed Data Factory using the diagram view (see below) to see the final result. When the pipeline runs it will create a HDInsight Cluster on-demand (if one does not exist from a prior step), execute the Hive script transformation and then load the data to the Azure SQL Database.

This example used multiple pieces of the new Azure Data Factory service but has only scratched the surface of its overall capabilities. Keep in mind that this service is still in Preview and will evolve. This evolution may change some of the current capabilities discussed and will certainly introduce new capabilities. I hope these posts sparks your interest in this new technology and moves you to give it a shot in your own use case. As always feel free to let me know if you have any questions or suggestions. Till next time!

Chris

Getting Ready to Ride

Mahoutis a scalable machine learning library that runs as MapReduce job within an HDInsight cluster. While a discussion of Mahout particulars is outside the scope of this blog, in order to leverage this library we simply need to execute a MapReduce job with the correct configuration. Recalling from the prior post, we defined linked services which allowed for access to Azure SQL and Blob data source, then datasets which act as both inputs and outputs for pipelines which are responsible for the heavy lifting. Before worrying about the running any Mahout job we first need to define a linked service for a HDInsight cluster Two options exists for creating a HDInsight cluster. You can either bring your own HDInsight cluster or have one created on-demand by Azure Data Factory. For this demo, we will use the on-demand variety. To create an on-demand cluster, we first must create and configure a LinkedService which access to the Azure Blob Storage account that will serve as a container for HDInsight as seen in the JSON below.

{
    "name": "HDIStorageLinkedService",
    "properties":
    {
        "type": "AzureStorageLinkedService",
        "connectionString": "DefaultEndpointsProtocol=https;AccountName=<ACCOUNT />;AccountKey=<ACCOUNT_KEY />"
    }
}

Next, we use JSON to configure the actual on-demand HDInsight service.

{
    "name": "HDILinkedService",
    "properties": 
    {
    "type": "HDInsightOnDemandLinkedService",
    "clusterSize": "4",
    "jobsContainer": "DataFactoryJobs",
    "timeToLive": "00:05:00",
    "linkedServiceName": "HDIStorageLinkedService"
    }
}

The final step is to create both linked services using the following PowerShell command.

New-AzureDataFactoryLinkedService -ResourceGroupName $ns -DataFactoryName $df -File .\HDIStorageLinkedService.json
New-AzureDataFactoryLinkedService -ResourceGroupName $ns -DataFactoryName $df -File .\HDILinkedService.json

With the required Linked Services in place, we can turn our attention to inputs and outputs. The input we will use for our new Pipeline is the output from the Ingress step (the UserRatingsBlob dataset) we defined in the prior post. That means that it is only necessary to define one new dataset. In this step our output dataset is only used as scaffolding to wire our pipelines together. This makes it unnecessary to define a structure or schema. To define the new Mahout output data set we can use the following JSON:

{
    "name": "MahoutRecommendations",
    "properties":
    {
        "location": 
        {
            "type": "AzureBlobLocation",
            "folderPath": "<CONTAINER />/mahout/recommendations/{Year}_{Month}_{Day}/",
            "linkedServiceName": "StorageLinkedService",
            "partitionedBy":
            [
                { name: "Year", value: { type: "DateTime", date: "SliceStart", format: "yyyy" } },
                { name: "Month", value: { type: "DateTime", date: "SliceStart", format: "%M" } },
                { name: "Day", value: { type: "DateTime", date: "SliceStart", format: "%d" } }
            ]
        },
        "availability": 
        {
            "frequency": "day",
            "interval": 1
        }
    }
}

After the JSON file has been created on your local file system, the PowerShell command below will create the new dataset.

New-AzureDataFactoryTable -ResourceGroupName $ns -DataFactoryName $df –File .\MahoutRecommendations.json

After all the required Linked Services and datasets are created, we can turn our attention to the Pipeline that will ultimately generate the recommendations.

Ride the Elephant

The Azure Data Factory contains a number of built in transformation types that allow you to manipulate data using Hive, Pig, MapReduce or even a bit of custom C# code. Since Mahout runs as a MapReduce job, we will leverage the MapReduce type using the JSON below.

{  
       "name": "MahoutPipeline",       
       "properties":{  
          "description":"Pipeline uses collaborative filtering to generate recommendations",
          "activities":[  
             {  
                "name":"MahoutActivity",
                "description":"Map Reduce job to generate Mahout recommendations",
                "type":"HDInsightActivity",
                "inputs":[  
                   {  
                      "Name":"UserRatingsBlob"
                   }
                ],
                "outputs":[  
                   {  
                      "Name":"MahoutRecommendations"
                   }
                ],
                "linkedServiceName":"HDILinkedService",
                "transformation":{  
                   "type":"MapReduce",
                   "className":"org.apache.mahout.cf.taste.hadoop.item.RecommenderJob",
                   "jarFilePath":"dfdemo/mahout/jars/mahout-examples-0.9.0.2.1.9.0-2196-job.jar",
                   "jarLinkedService":"StorageLinkedService",
                   "arguments":[  
                      "-s",
                      "SIMILARITY_PEARSON_CORRELATION",
                      "--input",
                      "wasb://@.blob.core.windows.net/reviews/",
                      "--output",
                      "$$Text.Format('wasb://@.blob.core.windows.net/mahout/recommendations/{0:yyyy_MM_dd}/', Time.AddHours(SliceStart, 0))",    
                      "--maxSimilaritiesPerItem",
                      "10",
                      "--tempDir",
                      "$$Text.Format('wasb://@.blob.core.windows.net/mahout/temp/{0:yyyy_MM_dd}/', Time.AddHours(SliceStart, 0))"                        
                   ]
                },
                "policy":{  
                   "concurrency":1,
                   "executionPriorityOrder":"OldestFirst",
                   "retry":1,
                   "timeout":"01:00:00"
                }
             }
          ]
       }
    }

A couple of things to point out in the configuration. Beyond the the input, outputs and linked services I want to call your attention to the MapReduce specific items. Note that the Mahout jar is stored in a blob storage container (dfdemo/mahout/jars) and a linked service is provided to allow for access to the library. Next, note that all the command-line or runtime arguments required by the Mahout RecommenderJob class are define in the arguments configuration section. These include the similarity calculation method, the maximum number of recommendations to generate for each user, the input, output and temp directories that will be used when the job runs. Finally, Mahout presents an interesting challenge since it neither cleans-up after itself nor allows for overwriting of files. This means that both the Temp and less-so the Output directory pose a potential issue in a system where Mahout will be run daily or on whatever cyclical schedule you define. To avoid errors, it is necessary to use the slice to make these folders unique on each run.

New-AzureDataFactoryPipeline -ResourceGroupName $ns -DataFactoryName $df –File .\PipeLines\MahoutPipeline.json
Set-AzureDataFactoryPipelineActivePeriod -ResourceGroupName $ns -DataFactoryName $df –Name MahoutPipeline -StartDateTime 2015-01-24Z -EndDateTime 2015-01-24Z -Force

Lastly, after the JSON config file is in place, create the pipeline and set the activity period to create the slices and run the pipeline as seen in the preceding code snippet.

Wrap-Up

Using the Azure Portal UI, verify the Data Factory is in order and that the diagram for the workflow looks similar to the screenshot below.

When the job runs, the Data Factory will spin up a HDInsight cluster on-demand using the configuration you specified. This step will take longer than anything else in the workflow. You can monitor the workflow by observing the event log or by verifying that that the slice is running as expected by drilling into the MahoutRecommendations dataset.

After the job completes, you can browse the recommendation results from blob storage.

In the next post, we will wrap-up this demo as we use the Hive transformation to un-pivot the Mahout recommendation results and then the CopyActivity to load or egress the data back to our Azure SQL Database.

Till next time!

Chris

The Set-Up

Before diving in, lets set the stage with a little background. The base of this demo is a website in which users can browse and rate beers and subsequently receive intelligent recommendations. Beers are rated on a scale from 1 to 5. Users and their ratings data is stored in a Azure SQL database called Beer. The review data is run through Mahout using collaborative filtering.

Before you get started you want to make sure you have a couple of things. You will need an Azure account, access to the new Azure Portal and Azure PowerShellinstalled and configured on your machine. The example I will present uses an Azure SQL database, Azure Blob storage and of course Azure Data Factory. For brevity, I will not discuss setting up Azure resources outside of the Data Factory.

Step 1: Creating the Data Factory

The Data Factory is the container for all subsequent activities we will discuss. There are currently two ways to create the Data Factory: the Azure Portal UI and PowerShell. Using the Azure Portal to create the Data Factory is a trivial task. Click on the ‘+ New’ button and select Azure Data Factory. Within the dialog you will need to specify a globally unique name, a resource group, the subscription (if you have multiple) and the geographical region.

You can likewise perform the same operation using Azure PowerShell. Before you begin executing commands, you’ve switched to AzureResourceManager mode and also that you are authenticated and your subscription is selected.

Switch-AzureMode AzureResourceManager

Add-AzureAccount 

Select-AzureSubscription -SubscriptionName "Visual Studio Ultimate with MSDN"

To create the Data Factory through PowerShell the following command is used. Please note that I am using PowerShell variables to simplify my script and will not repeat the declarations subsequently.

$df = "Brewhaus"
$ns = "BluewaterDataFactory"
$loc = "West US"

#Create the Data Factory
New-AzureDataFactory -ResourceGroupName $ns -Name $df -Location $loc -Force -ErrorAction Stop

After PowerShell has reported that the Data Factory was created successful, you can browse the Data Factory dashboard in the Azure Portal. Tip: To simplify working with this dashboard is to pin it to your Startboard to avoid having to constantly browse for it.

Step 2: Defining Linked Services

With the Data Factory in place, the next step is to define the Linked Services required for our workflow. Linked Services can be define as one of five types: AzureStorageLinkedService, AzureSqlLinkedService, OnPremiseSqlLinkedService, HDInsightOnDemandLinkedService and HDInsightBYOCLinkedService. For our purposes we are only concerns with connecting two an Azure SQL database and Azure Blob Storage. This requires two Linked Services one for each. While Linked Services can be defined through the portal, I will define these using a JSON definition and PowerShell. To define the Linked Service for an Azure SQL database, we construct the required JSON containing a connection string to the appropriate database as seen below.

{
    "name": "AzureSqlLinkedService",
    "properties":
    {
        "type": "AzureSqlLinkedService",
        "connectionString": " Server=tcp:<SERVER>.database.windows.net,1433; Database=Beer; User ID=<USER_ID>; Password=<PASSWORD>; Trusted_Connection=False; Encrypt=True; Connection Timeout=30"
    }
}

For a storage account Linked Service, the JSON contains the blob storage account name and key.

{
    "name": "StorageLinkedService",
    "properties":
    {
        "type": "AzureStorageLinkedService",
        "connectionString": " DefaultEndpointsProtocol=https; AccountName=<ACCOUNT_NAME>; AccountKey=<ACCOUNT_KEY>"
    }
}

To create the Linked Services the following PowerShell commands are used taking care to use the correct path to the JSON configuration files on your machine.

#Create Linked Services
New-AzureDataFactoryLinkedService -ResourceGroupName $ns -DataFactoryName $df -File .\AzureSqlLinkedService.json
New-AzureDataFactoryLinkedService -ResourceGroupName $ns -DataFactoryName $df -File .\StorageLinkedService.json

These Linked Services function as connection managers for your Data Factory. In the subsequent steps, we will define datasets and pipelines which reference these connections for the purpose of either accessing or processing data.

Step 3: Defining Datasets

The datasets or more accurately tables that we define provide the input and output schemas needed for this workflow. These schemas are where our we define the column (name and data type) structure, the location as well as the availability of the data. Like Linked Services, we must define two Tables one for each using JSON. Please note that as of the writing of the post creating datasets through the portal UI is not supported. Starting with source data located in the Azure SQL database, it is necessary to pull only three columns of data (UserID, BeerID and OverallRating) for this exercise. Since the data is coming from an Azure SQL database it is not necessary to specify a structure as it will be inferred from the database schema. This table points to our Azure SQL Linked Service and references the BeerReview table which contains the needed data. This demo will use a daily pull and does not have any external dependencies as is reflected in the availability element below.

{
    "name": "UserRatings",
    "properties":
    {
        "location":
        {
            "type": "AzureSqlTableLocation",
            "tableName": "BeerReview",
            "linkedServiceName": "AzureSqlLinkedService"
        },
        "availability": 
        {
            "frequency": "Day",
            "interval": 1,
            "waitonexternal": {}           
        }
    }
}

The blob data is an exact mirror of our source data. The big difference in each definition is how the location is specified. With blob storage we are not only writing data files to the configured account but we are partitioning the data into monthly folders. This is also where we could define any formatting options such as column and row delimiters or even the serializer used (current Text or Avro). For our demo, we will accept the defaults.

{
    "name": "UserRatingsBlob",
    "properties":
    {
        "location": 
        {
            "type": "AzureBlobLocation",
            "folderPath": "dfdemo/userratings/year={Year}/month={Month}/",
            "fileName": "ratings{Year}{Month}{Day}",
            "partitionedBy":
            [
                { name: "Year", value: { type: "DateTime", date: "SliceStart", format: "yyyy" } },
                { name: "Month", value: { type: "DateTime", date: "SliceStart", format: "%M" } },
                { name: "Day", value: { type: "DateTime", date: "SliceStart", format: "%d" } }
            ],
            "format":
            {
                "type": "TextFormat",
                "columnDelimiter": ","
            },
            "linkedServiceName": "StorageLinkedService"
        },
        "availability": 
        {
            "frequency": "Day",
            "interval": 1            
        }
    }
}

With the JSON in hand we can create both datasets using the following PowerShell snippet.

#Create Tables
New-AzureDataFactoryTable -ResourceGroupName $ns -DataFactoryName $df –File .\Tables\UserRatings.json
New-AzureDataFactoryTable -ResourceGroupName $ns -DataFactoryName $df –File .\Tables\UserRatingsBlob.json

Step 4: Creating a Pipeline

The final piece in the little puzzle we have going is the pipeline. The data pipeline is where we tie together our input and output. Activities such as transforming or enriching the data are define in the pipeline as we will explore in the next post. For this post, we will define only a single activity, the built in CopyActivity. Like data sets previously there is currently no way to define pipeline in the Portal UI, so JSON and PowerShell will be our go-to solution. In the JSON, we reference the two datasets and define the transformation. The transformation uses a SQL Data Reader as the source and filters based on the slice date. Since both input and output schemas are identical it is not necessary to perform and mapping between the source and destination.

{
    "name": "CopyRatingsPipeline",
    "properties":
    {
        "description" : "This pipelines moves User Ratings from an Azure SQL Database to Azure Blob Storage",
        "activities":
        [
            {
                "name": "CopyActivity",
                "description": "Data Mover", 
                "type": "CopyActivity",
                "inputs":  [ { "name": "UserRatings"  } ],
                "outputs":  [ { "name": "UserRatingsBlob" } ],
                "transformation":
                {
                    "source":
                    {
                        "type": "SqlSource",
                        "SqlReaderQuery": "$$Text.Format(' SELECT UserID, BeerID, Overall FROM dbo.BeerReview WHERE [ReviewDate] = \\'{0:MM/dd/yyyy}\\'', Time.AddHours(SliceStart, 0))"
                    },
                    "sink":
                    {
                        "type": "BlobSink"
                    }
                },
                "policy":
                {
                    "Concurrency": 1,
                    "ExecutionPriorityOrder": "OldestFirst",
                    "Retry": 1,
                    "Timeout": "01:00:00"
                }
            }
        ]
    }
}

With the JSON definition we can use PowerShell to create the pipeline as seen below.

#Create Pipelines
New-AzureDataFactoryPipeline -ResourceGroupName $ns -DataFactoryName $df –File .\PipeLines\CopyRatingsPipeline.json

After the PowerShell command completes successfully, browse can browse your Data Factory in the Azure portal. You can see the Linked Services, Datasets and Pipeline that were all created in the preceding steps. From the summary or main dashboard, click on the ‘Diagram’ button to browse a visual representation of the orchestration or workflow.

Step 5: Setting the Pipeline Activity Period

With all the pieces in place, it is only necessary to define a schedule for the pipeline. The PowerShell command takes the usual parameters as well as a start and optional end time. Set-AzureDataFactoryPipelineActivePeriod -ResourceGroupName $ns -DataFactoryName $df –Name CopyRatingsPipeline

-StartDateTime 2015-01-24Z -EndDateTime 2015-02-02Z -Force

After the activity period has been defined we can browse the slices and their processing status within the output data set.

Wrap-Up

In this posts, we create a new Data Factory and then defined Linked Services which act as connection managers within the Data Flow. Next, we defined data sets that become the input and outputs for our pipeline. The pipeline is responsible for moving the data needed for recommendation generation from an Azure SQL database to the configured blob storage account. In the next post, we will expand the demo by creating an On-Demand cluster running the Mahout MapReduce job to generate beer recommendations. Till Next Time!

Chris

The boundaries between on-premise and cloud-born data continue to blur with more and more organization moving to hybrid data landscapes. The blurring of these lines introduces a number of complexities from data locality, to scale and even the diversity of the data which we must consume and processed. To meet these challenges head-on, Microsoft introduced a new managed cloud service known as Azure Data Factory (ADF).

Four Pillars of Azure Data Factory

So what is Azure Data Factory? That question is best answered by describing the four (current) pillars of ADF.

1. Moving on-premise data to the cloud (and cloud data to on-premise)

   One of the primary use-cases of Azure Data Factory is as a pipeline to move or copy data across the on-premise/cloud boundary. Whether ingress or egress from the cloud, ADF is built to facilitate not only the orchestration of data movement but the transformations necessary to move your data to and from a variety of sources and destinations. The currently supported sources and destinations are: SQL Server (On-Premise), Azure Blobs, Azure Tables and Azure SQL databases. Please also note also that ADF can be used to move data from one cloud source to another.

2. Processing data at scale (HDInsight)

   While moving data from point A to point B is useful in a number of situations, often times it is necessary to transform and enrich your data along the way. Handling this processing can be non-trivial and when scale is an issue. The Azure Data Factory can handle processing data of any size by leveraging HDInsight. Using either a cluster of your own or one that Azure Data Factory creates on demand, you can process, transform and enrich your data using C#, Pig, Hive and even MapReduce. 

3. Integrate and enrich disparate data

   The diversity of data and how you bring it all together presents interesting and unique challenges. How do you process that JSON data from your social media platform to enrich the more traditional and relational data from your on-premise CRM system? What about all that XML data provided by your partner or that old-school order management system? With Azure Data Factory, you can build both complex (and trivial)workflows, orchestrating the outputs from one pipeline into the inputs of another. For example, use one pipeline to shred that JSON data with a Pig Script, while another loads your transactional CRM data from an on-premise SQL Server into Azure blob storage. A third pipeline could then take both outputs, integrating them together using Hive or a Custom Activity written in C#.

4. Orchestrate & Monitor data processing workflows

   If you done any amount of ETL work, you know first hand that its one thing to build a workflow and yet another to fully operationalize it. Too often in the ETL work things  like logging and monitoring are after thoughts. Not with Azure Data Factory. Whether you use the Azure Dashboard Portal or PowerShell, you not only have robust access to the details of every process or workflow in your Data Factory but out of the box alerting in the event that something does go awry.

Wrap-Up

So now that you have an idea of what the Azure Data Factory is, I suspect that some of you like me are itching to start using it. Over the next couple of posts, we will dig into the Data Factory, explore specific use cases and walk-through functional examples. If you would like to get a jump start, I’d invite you to check out the documentation and examples over at: https://azure.microsoft.com/en-us/documentation/services/data-factory/.

Lastly, keep in mind that Azure Data Factory is currently in preview. If you haven’t previously worked with any of the Azure Services in Preview that basically means that it can be will be subject to changes with little to no notice as it evolves towards a more formal GA release.

Till next time!

Chris