After using Azure queue and table store separately with Python I implemented a best practice to handle large messages using table store and the queue together. The flowchart on this page shows the logic commonly used when implementing a queue listener with a polling mechanism. This flowchart was implemented below.

A single queue message can be up to 64KB. However, what if you have larger messages or files that you'd like to manage via a queue? You could put the large message in table store or file in blob store. Next, add an entry to the queue that can be used to uniquely identify that particular table store row or blob.

Note: Python 2.7.9 was used for the code below. All code below is unsupported and is for illustrative purposes only. Also, see the unhandled exceptions section below.

Publisher

Below I implemented a publisher that adds an artificially "long" message to table store (when you see this message below just play along and pretend it's long…). Next, the unique partition key and row key for that message is added to a queue message.

Subscriber

I implemented a subscriber to process the queue and table store entry added by the publisher above. The subscriber checks the queue for a message. If a message exists it is retrieved then the unique partition and row key is parsed. Next, it uses that partition and row key to retrieve the message from table store. Lastly, it deletes the message from the queue and the table store entry.

Also, rather than check the queue repeatedly (which can be costly), I used a back-off delay algorithm. After checking the queue if nothing exists it backs off at an ever increasing duration until a maximum duration is reached. For example, the code below sleeps for 2, 4, 8, 16, 32 and 64 seconds. Between each sleep duration it checks the queue. If there is something in the queue it is retrieved per above then starts sleeping again at 2 seconds.

The publisher and subscriber below can be run in separate command windows so you can see them exchanging messages.

Also, if the queue and table store become empty just run the publisher again in a separate command prompts. You'll see the subscriber pick up the message and back-off accordingly.

Publisher.py

import string

import unicodedata

import base64

from azure.storage import TableService, Entity, QueueService

from operator import itemgetter

# This generates random words so the messages in table store look kinda interesting.

def generate_random_word(wordLen):

    word = ''

    for i in range(wordLen):

        word += random.choice(string.ascii_letters + string.digits)

    return word

def main():

     # Add your account name and key below

     Acct_name = 'add here'

     Acct_key = 'add here'

     # Name of the table store table and queue.

     Task_table = 'tasktable'

     Partition_key = 'Tasks'

     Task_queue = 'taskqueue'

     # Number of messages to add to table store and thus the queue.

     Max_messages = 3

     # Create a table if it doesn't exist. 

     table_service = TableService(account_name=Acct_name, account_key=Acct_key)

     table_service.create_table(Task_table)

     # Create a queue if it doesn't exist. 

     queue_service = QueueService(account_name=Acct_name, account_key=Acct_key)

     queue_service.create_queue(Task_queue)

     # Add a few entries to table store.

     for rowCnt in range(1, Max_messages):

          print '----------'

          print 'Publisher'

          print ' '

          # Generate a random word between x and y chars long to put into the description.

          wordLen = random.randint(10,25)

          randWord = 'Pretend this is a really long description... '

          randWord += generate_random_word(wordLen)

           print 'Random description is:', randWord

          # Add data to the table below. Priority field in the message is just extra data and is not used for anything.

          task = {'PartitionKey': Partition_key, 'RowKey': str(rowCnt), 'description' : randWord, 'priority' : 200}

              table_service.insert_entity(Task_table, task)

          # For the data we just added to table store, add the unique partition and row key to the queue.

          # The partition key and row key in the queue message will look like PartitionKey-RowKey

          PR_Key = Partition_key

          PR_Key += '-'

          PR_Key += str(rowCnt)

          print 'Partition and row key is ', PR_Key

          queue_service.put_message(Task_queue, base64.b64encode(PR_Key))

          # Show the length of the queue.

          queue_metadata = queue_service.get_queue_metadata(Task_queue)

          print 'Messages in the queue:', queue_metadata['x-ms-approximate-messages-count']

          print '----------'

main()

-----

After the publisher runs, you'll see something like the following in table storage and the queue. I used Azure Storage Explorer to view this data.

----- 

Subscriber.py

import base64

import time

from azure.storage import TableService, Entity, QueueService

# Add your account name and key here

Acct_name = 'add here'

Acct_key = 'add here'

# Name your table store table and queue

Task_table = 'tasktable'

Task_queue = 'taskqueue'

# Params used to check the queue and slowly back-off if it is empty. Start with a minimum/fastest rate and back-off to the maximum/slowest rate to check the queue.

Min_delay_sec = 2

Max_delay_sec = 64

# Create a table if it doesn't already exist.

table_service = TableService(account_name=Acct_name, account_key=Acct_key)

table_service.create_table(Task_table)

# Create a queue if it doesn't already exist.

queue_service = QueueService(account_name=Acct_name, account_key=Acct_key)

queue_service.create_queue(Task_queue)

# Start an infinite loop to read from the queue. Crtl+C to kill it.

while 1 == 1:

    # A 2 step process to dequeue a message; get the message then further below it is deleted.

    messages = queue_service.get_messages(Task_queue, numofmessages=1, visibilitytimeout=1)

    if messages.queue_messages:

         Current_delay_sec = Min_delay_sec

         print '----------'

        print 'Subscriber'

        print ' '

        for message in messages:

            PR_Key = base64.b64decode(message.message_text)

            print 'Message text has partition and row key:', PR_Key

            # Parse out partition key and row key from PR_Key. The partition key and row key will look like: PartitionKey-RowKey

            PR_Key = PR_Key.split('-', 2)

            print "Partition key:", PR_Key[0]

            print "Row key:", PR_Key[1]

            # Get the table store message using the unique partition key and a row key

            task = table_service.get_entity(Task_table, PR_Key[0], str(PR_Key[1]))  

            print 'Message description:', task.description

            print 'Message priority:', task.priority

            print('----------') 

            # I foolishly assume everything was retrieved just fine. Delete the queue entry and delete the table store entry using the partition and row key.
            queue_service.delete_message(Task_queue, message.message_id, message.pop_receipt)
            table_service.delete_entity(Task_table, PR_Key[0], str(PR_Key[1]))

    # However, if the queue was empty then back-off per below.   

    else:

        if Current_delay_sec < Max_delay_sec:

            Current_delay_sec = Current_delay_sec * 2

        print 'The queue is empty. Will check the queue again in seconds:', Current_delay_sec

        time.sleep(Current_delay_sec)

----------

When the subscriber runs it looks like the following. Note the queue check re-try is backing off up to 64 seconds.

Unhandled exceptions?! Say it isn't so!

Yes, it's true! I have unhandled exceptions in the code above. For example if you run the publisher twice in a row without running the subscriber to delete entries from table store the publisher will fail since it tried to add yet another entry to table store with the same row key. You can dream-up ways to work around this fault.

Below is some more info in case you'd like to learn more:

How to install the Python Azure SDK

Python and Table storage

Python and Queue storage

1   [234:5.0...

3   ...272:4.649266]

hadoop fs -put u.data u.data

hadoop jar C:\hdp\mahout-0.9.0.2.1.3.0-1981\core\target\mahout-core-0.9.0.2.1.3.0-1981-job.jar org.apache.mahout.cf.taste.hadoop.item.RecommenderJob -s SIMILARITY_LOGLIKELIHOOD --input u.data --output udata_output

hadoop fs -getmerge udata_output udata_output.txt

PS C:\WINDOWS\system32> get-azurestoragecontainer <your container name here>

Blob End Point: https:// <storage account name >.blob.core.windows.net/

Name                 PublicAccess      LastModified                                           

----                    ------------         ------------     

container1           Off                   1/11/2015 12:10:23 AM +00:00                           

1. My test file was in this location “c:\temp\1.txt”.
2. From the PowerShell prompt I ran the following to upload the file to blob store:

Set-AzureStorageBlobContent -Blob “TestTempFile” -Container container1 -File “c:\temp\1.txt” -Force

3. You can use Azure Storage Explorer to see the blob named "TestTempFile" or run the following from PowerShell prompt to get that file and place it in a different location on your local machine - I used c:\hdp. Also, the -Blob param is case sensitive (must be "TestTempFile"):

Get-AzureStorageBlobContent -Blob TestTempFile -Container container1 -Destination c:\hdp -Verbose

Get-AzureStorageBlob -Container container1 -Blob TestTempFile -Verbose

Container Uri: https://< storage account name>.blob.core.windows.net/container1

Name                   BlobType   Length             ContentType           LastModified       SnapshotTime             

----                      --------      ------                -----------              ----------            -----------             

TestTempFile      BlockBlob     65                   application/octet-stream  1/11/2015 11:08:35 PM +...

Error: New-AzureHDInsightCluster : Unable to complete the cluster create operation. Operation failed with code '400'. Cluster left behind state: 'Error'. Message: 'ClusterUserNameInvalid'.

Remove-AzureHDInsightCluster -name <add bad cluster name here>

• You can sign into your cluster using remote desktop
• open the Hadoop command prompt
• run the following:

C:\apps>hadoop fs -ls /example/jars

My results are shown below:

Found 1 items

-rw-r--r--   1 hdpinternaluser supergroup     270298 2015-01-18 05:56  /example/jars/hadoop-mapreduce-examples.jar

$clusterName = "<your HDInsight cluster name here>"

# Define the MapReduce job

# Run the job and show the standard error

"Get-AzureStorageBlobContent : Access to the path 'C:\WINDOWS\System32\example\data\WordCountOutput' is denied.

1. Get-AzureStorageBlobContent -Blob example/data/WordCountOutput/part-r-00000 -Container container1 -Destination c:\hdp -Force -Verbose

2. cat C:\hdp\example\data\WordCountOutput\part-r-00000 | findstr "there"

$storageAccountName = "<your storage account name > "

$containerName = "<your container name>"

## I added the 2 variables below. Add your root destination subdir below.

$Rootdestination = "<your root subdir here>"

$FileIsHere = "$Rootdestination\example\data\WordCountOutput\part-r-00000"

 # Create the storage account context object

$storageAccountKey = Get-AzureStorageKey $storageAccountName | %{ $_.Primary }

$storageContext = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKey 

# Download the output to local computer.

## I added the -Destination switch below to specify where I want the file. I also like -Verbose too :-)

Get-AzureStorageBlobContent -Container $ContainerName -Blob example/data/WordCountOutput/part-r-00000 -Context $storageContext -Destination $Rootdestination -Force -Verbose

## Display the output. I included the full path where the file is located.

cat $FileIsHere | findstr "there"

• can handle massive amounts of structured or messy, unstructured data that relational DB's would struggle with. Massive amounts of data includes billions of rows and more than a million columns.
• functions on-top of Hadoop Distributed File System (HDFS)
• allows data formats to vary from row to row
• supports very fast updates and inserts to smaller data sets without "locking" tables like a relational DB would and provides nearly instant access to that inserted/updated data.
• can do much more than I've summarized here

• key/value
• structured documents (e.g.: HTML, JSON, XML)
• graph (think of nodes with relationships and directions between those nodes)  
• columnar (data is stored in columns rather than rows).

cd %HBASE_HOME%\bin

hbase shell

hbase(main):001:0>

 create '<table name>', '<column family 1>', '<column family 2>', ...

create 'employee', 'name', 'address'

 describe '<table name>'

describe 'employee'

 List

put 'employee', 'id1', 'name:first', 'Bob'

put 'employee', 'id1', 'name:last', 'Jones'

put 'employee', 'id1', 'address:street', '123 Main Street'

put 'employee', 'id1', 'address:city', 'Seattle'

put 'employee', 'id1', 'address:state', 'WA'

put 'employee', 'id1', 'address:country', 'USA'

put 'employee', 'id1', 'address:zip', '98102'

put 'employee', 'id2', 'name:first', 'Jane'

put 'employee', 'id2', 'name:last', 'Smith'

put 'employee', 'id2', 'address:street', '456 4th Ave'

put 'employee', 'id2', 'address:city', 'Renton'

put 'employee', 'id2', 'address:state', 'WA'

put 'employee', 'id2', 'address:country', 'USA'

put 'employee', 'id2', 'address:zip', '98104'

scan '<table name>'

scan 'employee'

get '<table name>' ,'<row#>'

get 'employee', 'id1'

count '<table name>'

count 'employee'

scan 'employee', {FILTER => "ValueFilter(=, 'binary:Bob')"}

scan 'employee', {FILTER => "ValueFilter(=, 'binary:WA')"}

delete 'employee', 'id1', 'name:last'

delete 'employee', 'id1', 'name:first'

scan 'employee'

ROW                COLUMN+CELL

 id1                  column=address:city, timestamp=1420868942823, value=Seattle

 id1                  column=address:country, timestamp=1420868942901, value=USA

 id1                  column=address:state, timestamp=1420868942870, value=WA

 id1                  column=address:street, timestamp=1420868942776, value=123 Main Street

 id1                  column=address:zip, timestamp=1420868942948, value=98102

  id2                  column=address:city, timestamp=1420868943104, value=Renton

  id2                  column=address:country, timestamp=1420868943167, value=USA

  id2                  column=address:state, timestamp=1420868943136, value=WA

  id2                  column=address:street, timestamp=1420868943058, value=456 4th Ave

  id2                  column=address:zip, timestamp=1420868944739, value=98104

  id2                  column=name:first, timestamp=1420868942995, value=Jane

  id2                  column=name:last, timestamp=1420868943026, value=Smith

get 'employee', 'id1'

deleteall '<table name>', '<row>'   (There are some optional params that I won't mention here.)

deleteall 'employee', 'id1'

Truncate '<table name>'

truncate 'employee'

disable '<table name>'

drop '<table name>'

disable 'employee'

drop 'employee'

• After signing into Microsoft Azure choose HDInsight -> Create an HDInsight Cluster -> Hadoop. 
• Give your cluster a name for *.azurehdinsight.net. 
• A single data node cluster is fine for testing purposes. 
• In that process you'll be asked to associate your Hadoop cluster with a storage account. This is necessary so that your data can live even after your HDInsight cluster has been deleted. 
• After you choose "Create HDInsight Cluster" it will take a few minutes to create your cluster. 

• Getting started with Microsoft Azure HDinsight.
• HDInsight background info and creating your first Hadoop cluster with HDInsight in minutes.
• Create an Azure storage account.

• Once you gain access to your cluster, on the desktop open the Hadoop command prompt and enter cd %hive_home%\bin which will take you to \apps\dist\hive-<version>\bin.
• Run "hive" at the command prompt. 
• You should see the hive command prompt as "hive>"

Yet another step in my journey learning about big data. Below are some things I've learned along the way.

The image here provides a high-level overview of Hadoop. (I'm still trying to find a more detailed architecture diagram that I can freely re-use).

After setting up an HDInsight cluster on Azure I dove in and wrote some Hive commands. However, before going too far there are a few key pieces to understand namely Hive, map-reduce, Hadoop Distributed File System and HCatalog.  

In summary: 

Hive functions differently than a relational DB management system. Hive is an abstraction layer on top of map-reduce and actually turns your HQL queries into map-reduce jobs. "So... why not just write map-reduce jobs?" you might ask. Well you could but you'd need to write the actual code in Java, C# or another language rather than just writing SQL-like statements. I'm sure there are times where an HQL query would suffice but other times where an actual map-reduce job would need to be written. 

Also, there are many details behind Hive and HQL that I won't go into here. The Hortonworks University has a really good introductory tutorial on Hive. 

Since HDFS is a file system you will find commands that you would expect like file copy, create a subdirectory, delete files, etc. The HDFS commands I used most frequently from the Hadoop command line are:

• hadoop dfs -mkdir HDFS_destination_folder --> Creates an HDFS subdirectory "HDFS_destination_folder"

• hadoop dfs -put c:\local_subdir\filename.txt HDFS_destination_folder --> Copies filename.txt to the HDFS_destination_folder

• hadoop dfs -ls HDFS_destination_folder --> Shows all of the files in the HDFS_destination_folder 

• hadoop dfs -get /HDFS_destination_folder/file.txt c:\local_subdir --> Gets file.txt from HDFS and places it in your local c:\local_subdir

• A full list of commands can be found by entering "hadoop dfs -help". In adddition, this seems to be a good summary of commands.

• Map-reduce utilizes HDFS. For example, when an HQL query is submitted by a user the master node uses a "map" process to assign parts of that query to data nodes to execute. The data nodes execute their assigned portion and return their individual results back to the master node. The master node then aggregates or "reduces" the individual results into a cohesive result that is returned to the user who executed the HQL query. 

• HCatalog provides an abstraction over the files in HDFS. Hive, map-reduce and other parts of Hadoop use HCatalog to simplify reading and writing to HDFS. I found this to be a good tutorial of HCatalog.

I'm skipping over many other parts of Hadoop mainly because I haven't used them yet. Realistically, you may not need to use everything in the Hadoop toolset. Some of the other parts include Pig, Sqoop, etc. This a good description of each of the Hadoop tools.

In the next post I'll import a file into a hive table and create some queries. 

I'm currently focused on learning about big data. Over a series of posts I'll show the path I took from being new to big data to eventually doing something worthwhile with Big Data, specifically using HDInsight which is the Microsoft Azure Hadoop distribution. If big data is new to you then I hope you can learn a few things below.  

If you like the overviews above you'll need a place to try the tutorials. I'm aware of 3 options.