I will leave this blog and its existing messages in place for archival purposes, but please update your bookmarks and RSS feeds if you want to keep tracking my adventures :-)

Our friends at MS Open Tech have recently published a Vagrant Azure Provider that allows Vagrant to control and provision machines in Windows Azure. I would like to show you how to use this provider to start a VM and then automatically provision it with a development environment suitable for PHP/LAMP and Symfony2 development.

First, you should download and install Vagrant for Windows. Then, installing the Azure plugin is as simple as:

vagrant plugin install vagrant-azure 

Then we are going to base our configuration on the "dummy Azure box" provided by MS Open Tech:

vagrant box add azuredummybox https://github.com/msopentech/vagrant-azure/raw/master/dummy.box 

Next, create an empty directory as your project root, and create a Vagrantfile in it. Here is a Vagrantfile for Azure, let's look at it in detail.

Vagrant.configure('2') do |config|
    config.vm.box = 'trusty64'

    config.vm.provider :azure do |azure, override|
        override.vm.box = 'azuredummybox'
        override.ssh.username = 'azureuser' # assigned below
        override.ssh.password = 'Pass123!' # assigned below

        azure.mgmt_certificate = 'C:\Dev\management_cert.pfx'
        azure.mgmt_endpoint = 'https://management.core.windows.net'
        azure.subscription_id = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
        azure.storage_acct_name = 'mystorage' # optional. A new one will be generated if not provided.

        # Ubuntu Precise 12.04 LTS
        #azure.vm_image = 'b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_4-LTS-amd64-server-20140428-en-us-30GB'
        # Ubuntu Trusty 14.04 LTS
        azure.vm_image = 'b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140416.1-en-us-30GB'

        azure.vm_user = 'azureuser' # defaults to 'vagrant' if not provided
        azure.vm_password = 'Pass123!' # min 8 characters. should contain a lower case letter, an uppercase letter, a number and a special character

        azure.vm_name = 'phpdevbox' # max 15 characters. contains letters, number and hyphens. can start with letters and can end with letters and numbers
        azure.cloud_service_name = 'phpdevbox' # same as vm_name. leave blank to auto-generate
        #azure.deployment_name = '' # defaults to cloud_service_name
        azure.vm_location = 'West Europe' # e.g., West US
        azure.ssh_private_key_file = 'C:\Dev\id_rsa'
        azure.ssh_certificate_file = 'C:\Dev\myCert.pem'

        # Provide the following values if creating a *Nix VM
        azure.ssh_port = '22'

        # Open HTTP port
        azure.tcp_endpoints = '80,8000'
    end

    config.vm.provision :shell, :path => "bootstrap.sh"
end

Here are the parameters you need to change:

• override.ssh.username should be the same as azure.vm_user: the login that Vagrant will use to connect via ssh.
• override.ssh.password and azure.vm_password: the password.
• azure.mgmt_certificate: you need to point Vagrant to your Azure management certificate. There are a couple ways to obtain it, the simplest is to extract it from your Publish Settings file.
• azure.subscription_id: this is your Subscription ID, you will find it in your Publish Settings file as well.
• azure.vm_image: the name of the image you want to use; I included two examples for Ubuntu Precise Pangolin and Trusty Tahr, but you should double-check for the latest versions, for example using azure vm image list from the CLI tools.
• azure.vm_name: the host name for your VM.
• azure.cloud_service_name: this will be the public name for your VM; preferably the same as the host name.
• azure.vm_location: the name of the data center you want to use; azure vm location list to retrieve a list of locations.
• azure.ssh_private_key_file and azure.ssh_certificate_file are used to configure your VM's ssh access. We have a tutorial explaining how to generate Windows Azure compatible keys for Linux.
• azure.tcp_endpoints: let's open both the standard HTTP port (80) and port 8000 in order to run the PHP built-in Web server from userspace.

So far, we have configured everything we need to create and provision the Virtual Machine, now we need to configure the development environment for LAMP, i.e. install PHP, MySQL, etc. An easy way to do it is to instruct Vagrant to run a shell script to bootstrap the VM, this is what the following line does:

config.vm.provision :shell, :path => "bootstrap.sh"

This will automatically run the bootstrap.sh script in the VM. Let's have a look at the script.

#!/bin/bash

# Add Node repo
add-apt-repository ppa:chris-lea/node.js

apt-get update

# LAMP etc.

# MySQL password
debconf-set-selections <<< 'mysql-server mysql-server/root_password password Pass123!'
debconf-set-selections <<< 'mysql-server mysql-server/root_password_again password Pass123!'

# LAMP
apt-get -q -y install lamp-server^

# Misc PHP stuff
apt-get -q -y install php-apc php5-xdebug php5-intl

# Git
apt-get -q -y install git

# Node
apt-get install -q -y python-software-properties python g++ make nodejs

# Azure CLI
npm install azure-cli -g

# PHP configuration

for f in /etc/php5/cli/php.ini /etc/php5/apache2/php.ini
do
  sed -i "s/;date.timezone =/date.timezone = Europe\/Paris/" $f
  echo "xdebug.max_nesting_level=250" >> $f
done

This script basically installs everything you need to run a LAMP application, plus the Azure Command-Line Interface and configures some Symfony2-specific parameters. In more detail:

• Add Chris Lea's repository to facilitate installing a recent version of Node.
• Update the repository database.
• Pre-configure a root password for MySQL.
• Install the magic LAMP server bundle.
• Install Git.
• Install Node.JS and a few required packages.
• Configures a couple of recommended parameters for Symfony2 in the php.ini.

Now all you need to do to start your development environment is to run the magic Vagrant commands to start and provision the VM:

vagrant up --provider=azure vagrant provision 

The machine is now ready to use! You can log in using the following command:

vagrant ssh 

Now that your development machine environment is entirely automated, you can check it into your source code repository and keep track of any changes you need to make to it. Other developers can use this configuration in order to exactly reproduce the same development environment, automatically.

Let's test our new environment, for example we can check out a sample Symfony2 project:

git clone https://github.com/tomconte/symfony-azure.git 

Run Composer:

cd symfony-azure curl -sS https://getcomposer.org/installer | php php composer.phar install 

Symfony will prompt you for your configuration information; you can accept all the defaults, but don't forget to give the right root password for MySQL! Now you should be able to check everything is OK and run the demo app:

php app/check.php php app/console server:run 0.0.0.0:8000 

I hope this article gave you a good idea of how to use Vagrant together with Azure to easily spin up automated and standardized development environments.

Update: this article describes how to install MariaDB manually on Azure, using the Classic stack. If you are looking for an out-of-the-box solution, you will now find MariaDB Enterprise Cluster in the Azure Marketplace : a production-grade configuration that you can deploy in one click. Check it out!

Install MariaDB Galera Cluster and create a VM Image

First, we need to install the MariaDB/Galera bits on a virtual machine that we will use as an image to create all the instances in our cluster. Let's create an Ubuntu 12.04 VM:

azure vm create mariadb b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_3-LTS-amd64-server-20140130-en-us-30GB --location "West Europe" --vm-size small --ssh 22 --ssh-cert myCert.pem azureuser 

Then we will install the MariaDB Galera Cluster by following the installation instructions:

sudo apt-get install python-software-properties sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xcbcb082a1bb943db sudo add-apt-repository 'deb http://mirrors.linsrv.net/mariadb/repo/5.5/ubuntu precise main' sudo apt-get update sudo apt-get install mariadb-server 

We will now deprovision, shut down and capture the VM image.

In the VM, run:

sudo waagent -deprovision 

Then log off and from your admin workstation:

azure vm shutdown mariadb azure vm capture mariadb mariadb-galera-image 

Create the Virtual Machines

Now that we have a base image, let's start the virtual machines that will form the cluster. Looking at the Galera Cluster deployment variants, I am aiming for variant 3a, i.e. a three-node cluster with a distributed load-balancer, in this case using the JDBC driver built-in load balancing capabilities. However I would also like to be able to test variant 3, using the Windows Azure load-balancer. This means I need to have the VMs in the same Cloud Service (to be able to load-balance the traffic) and in a Virtual Network (so I can address each node with a stable IP).

I am going to use three nodes, as recommended in the Getting Started document to avoid "split-brain" situations. However, note that Galera also provides an arbitrator that can be used as a lightweight member instead of a full node.

Let's create an Affinity Group:

azure account affinity-group create --location "West Europe" galerawest 

And a VNet:

azure network vnet create --address-space 10.0.0.0 --cidr 8 --subnet-name mariadb --subnet-start-ip 10.0.0.0 --subnet-cidr 24 --affinity-group galerawest galeravnet 

And now will create three VMs for the cluster. We need to put them in the right VNet/subnet, and we use the --connect option to put them in the same Cloud Service.

azure vm create -v --vm-name mariadb1 --virtual-network-name galeravnet --subnet-names mariadb --affinity-group galerawest \ --vm-size small --ssh 22 --ssh-cert ./certs/myCert.pem --no-ssh-password mariadbcluster mariadb-galera-image tom azure vm create -v --vm-name mariadb2 --virtual-network-name galeravnet --subnet-names mariadb --affinity-group galerawest \ --vm-size small --ssh 23 --ssh-cert ./certs/myCert.pem --no-ssh-password --connect mariadbcluster mariadb-galera-image tom azure vm create -v --vm-name mariadb3 --virtual-network-name galeravnet --subnet-names mariadb --affinity-group galerawest \ --vm-size small --ssh 24 --ssh-cert ./certs/myCert.pem --no-ssh-password --connect mariadbcluster mariadb-galera-image tom 

We now have our three nodes. For the sake of simplicity, we won't add a DNS server to the setup and just use the IP addresses of our machines. In our case, these will be 10.0.0.4, 10.0.0.5 and 10.0.0.6.

Configuring the cluster

We are now going to configure our cluster. I used Codership's Galera configuration page as reference, as well as a couple how-to articles:

• http://planet.mysql.com/entry/?id=616914
• https://www.digitalocean.com/community/articles/how-to-configure-a-galera-cluster-with-mariadb-on-ubuntu-12-04-servers
• http://www.severalnines.com/clustercontrol-mysql-galera-tutorial

First, stop the mysqld that is probably running on all the nodes after the initial installation:

sudo service mysql stop 

We are going to create a new configuration file for the cluster parameters, i.e.:

vi /etc/mysql/conf.d/cluster.cnf 

And here is the configuration I used:

[mysqld] query_cache_size=0 binlog_format=ROW default-storage-engine=innodb innodb_autoinc_lock_mode=2 query_cache_type=0 bind-address=0.0.0.0 # Galera Provider Configuration wsrep_provider=/usr/lib/galera/libgalera_smm.so #wsrep_provider_options="gcache.size=32G" # Galera Cluster Configuration wsrep_cluster_name="test_cluster" wsrep_cluster_address="gcomm://10.0.0.4,10.0.0.5,10.0.0.6" # Galera Synchronization Congifuration wsrep_sst_method=rsync #wsrep_sst_auth=user:pass # Galera Node Configuration wsrep_node_address="10.0.0.4" wsrep_node_name="mariadb1" 

As you can see, I gave my cluster a wsrep_cluster_name and listed the nodes' IP addresses in wsrep_cluster_address. You have copy this configuration file on all the nodes, and change the wsrep_node_name and wsrep_node_address parameters on each.

You should also copy Debian "maintenance" configuration from the first node to all the others, so that the credentials for the special debian-sys-maint user are the same on all nodes. Just copy the contents of /etc/mysql/debian.cnf to the other nodes. If you want to quickly edit a configuration file on a remote node as root, you can use:

ssh -t 10.0.0.6 sudo vi /etc/mysql/debian.cnf 

Adding disks

I am solely focusing on testing clustering here, but at this point you should really attach a bunch of disks to each node, create a RAID array and change the datadir in MySQL's my.cnf in order to store the data properly.

Create Storage Accounts, one per VM:

for i in {1..3}; do azure storage account create --affinity-group galerawest galerastor$i; done 

Let's create 4 x 256GB disks per VM, each group of 4 going to a separate Storage Account:

for j in {1..3}; do for i in {1..4}; do azure vm disk attach-new mariadb$j 256 http://galerastor$j.blob.core.windows.net/disks/mariadb-$j-disk-$i.vhd; done; done 

Once you have created and attached the disks, you should see four new devices appear; /dev/sdc to /dev/sdf:

root@mariadb1:~# ls -l /dev/sd* brw-rw---- 1 root disk 8, 0 Mar 18 15:14 /dev/sda brw-rw---- 1 root disk 8, 1 Mar 18 15:14 /dev/sda1 brw-rw---- 1 root disk 8, 16 Mar 18 15:14 /dev/sdb brw-rw---- 1 root disk 8, 17 Mar 18 15:14 /dev/sdb1 brw-rw---- 1 root disk 8, 32 Mar 18 15:14 /dev/sdc brw-rw---- 1 root disk 8, 48 Mar 18 15:14 /dev/sdd brw-rw---- 1 root disk 8, 64 Mar 18 15:14 /dev/sde brw-rw---- 1 root disk 8, 80 Mar 18 15:14 /dev/sdf 

We are now going to create a RAID striped array using mdadm.

On each machine, install mdadm:

apt-get install mdadm 

On Ubuntu, APT insists on installing Postfix as a dependency; just select "No configuration" in the Postfix configuration screen that pops up (you can always run dpkg-reconfigure postfix later).

Now let's create the stripez:

mdadm --create --verbose /dev/md0 --level=stripe --raid-devices=4 /dev/sdc /dev/sdd /dev/sde /dev/sdf 

Save the configuration:

mdadm --detail --scan >> /etc/mdadm/mdadm.conf 

Create the filesystem, the mount point, and mount the disk:

mkfs.ext4 /dev/md0 mkdir /data mount -t ext4 /dev/md0 /data 

If you already have some data, you will need to move it to the new location. I will go for the easy option of moving the whole datadir and pointing to the new location with a symlink. You should not have to reconfigure AppArmor as instructed in the link above.

service mysql stop mv /var/lib/mysql /data ln -s /data/mysql /var/lib/mysql service mysql start 

You can do this operation on each node, one by one.

Starting the cluster

Now let's start the cluster on the first node, using --wsrep-new-cluster to create the cluster.

sudo service mysql start --wsrep-new-cluster 

And on the other nodes:

sudo service mysql start 

You should now be able to see the three nodes in the cluster:

mysql -u root -p -e 'SELECT VARIABLE_VALUE as "cluster size" FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME="wsrep_cluster_size"' +--------------+ | cluster size | +--------------+ | 3 | +--------------+ 

Testing the cluster

Let's create a test database and table:

mysql -u root -p -e 'CREATE DATABASE playground;' mysql -u root -p -e 'CREATE TABLE playground.jdbctests ( id INT NOT NULL AUTO_INCREMENT, t VARCHAR(64), h VARCHAR(64), d DATE, PRIMARY KEY(id) );' 

Now you should be able to connect to any node, insert data in the table, select from another node, etc., and make sure the replication works. If you need to allow access to the database:

grant all on playground.* to 'root'@'%' identified by '<password>'; 

Let's use a little test Java program too see how load-balancing works. You can download the MySQL JDBC driver, and read up on the multi-host connections capabilities.

I used some sample source code from the following blog post:

• http://openlife.cc/blogs/2012/june/howto-use-mysql-jdbc-loadbalancer-galera-multi-master-clusters

I changed the output messages somewhat and added logging of the node where the connection was established, so we can check our connections are load-balanced.

I also modified the JDBC URL:

• listed the nodes we want to load-balance
• added loadBalanceBlacklistTimeout (see How to use JDBC (Connector/J) with MySQL Cluster)
• added connectTimeout in case a node is down

Here is the code:

import java.sql.*;

public class MysqlTest
{
    public static void main (String[] args)
    {
        Connection conn = null;

        while(true)
        {
            // Connect to MySQL
            try
            {
                String userName = "root";
                String password = "xxx";
                String hosts = "10.0.0.4,10.0.0.5,10.0.0.6";
                String url = "jdbc:mysql:loadbalance://" + hosts + "/playground?loadBalanceBlacklistTimeout=30000&connectTimeout=1000";
                Class.forName ("com.mysql.jdbc.Driver").newInstance ();
                conn = DriverManager.getConnection (url, userName, password);
                System.out.println ("Connected to " + conn.getMetaData().getURL());
            }
            catch (Exception e)
            {
                System.err.println ("Cannot connect to database server: " + e);
            }

            try
            {
                Statement s = conn.createStatement ();
                int count;
                count = s.executeUpdate("INSERT INTO jdbctests (t, h, d) VALUES ('Inserted from Java code.', @@hostname, now());");
                s.close ();
                //System.out.println (count + " rows were inserted");
            }
            catch (SQLException e)
            {
                System.err.println ("Error when executing SQL statement: " + e);
            }

            // Close connection
            if (conn != null)
            {
                try
                {
                    conn.close ();
                    //System.out.println ("Database connection terminated");
                }
                catch (Exception e)
                {
                    System.err.println( "Error when closing database connection: " + e );
                }
            }
            try { Thread.sleep(1000); } catch( Exception e ) { }
        }
    }
}

You can compile and run the program like this:

javac MysqlTest.java java -cp ./mysql-connector-java-5.1.29/mysql-connector-java-5.1.29-bin.jar:. MysqlTest 

You should see connections going out to all the nodes in the cluster at random.

You can test the behavior in case of node failure by shutting down one of the nodes (e.g. from the admin portal). You should not see any errors, just one of the nodes disappearing from the log.

Failover

To use failover instead of load-balancing, you can change the JDBC URL to:

jdbc:mysql://" + hosts + "/playground?loadBalanceBlacklistTimeout=30000&connectTimeout=1000 

i.e. remove the "loadbalance" option. However this will use standard failover where the secondary host is considered a read-only slave.

To configure multi-master failover:

String hosts = "address=(type=master)(host=10.0.0.4)(protocol=tcp)(port=3306),address=(type=master)(host=10.0.0.5)(protocol=tcp)(port=3306),address=(type=master)(host=10.0.0.6)(protocol=tcp)(port=3306)"; String url = "jdbc:mysql://" + hosts + "/playground?autoReconnect=true&failOverReadOnly=false&connectTimeout=1000"; 

This extended syntax allows you to specify that the failover hosts are also masters.

Using the Windows Azure load balancer

As discussed above, you can also take advantage of the integrated Windows Azure load balancer (WALB) to implement the connection to the cluster.

Let's create a load-balancer set:

azure vm endpoint create --lb-set-name mysql mariadb1 3306 azure vm endpoint create --lb-set-name mysql mariadb2 3306 azure vm endpoint create --lb-set-name mysql mariadb3 3306 

Now you can use your Cloud Service address, mariadbcluster.cloudapp.net in my case, to connect to your cluster. The connections will automatically be load-balanced on all nodes using a round-robin algorithm.

Running some benchmarks

Here are some notes on how to benchmark your MariaDB cluster using sysbench.

Install sysbench:

sudo apt-get install sysbench sudo apt-get install mysql-client-core-5.5 

Create a test database using the mysql client:

create database sysbench; grant all on sysbench.* to 'root'@'%' identified by 'xxx!'; 

I will run the benchmark client using the load-balanced endpoint address that I created above.

Prepare the benchmark, e.g.:

sysbench --test=oltp --oltp-table-size=1000000 --mysql-host=mariadbcluster.cloudapp.net --mysql-db=sysbench \ --mysql-user=root --mysql-password='xxx' prepare 

Run the benchmark, e.g. 1 minute, 4 threads, simple test:

sysbench --test=oltp --oltp-table-size=1000000 --mysql-host=mariadbcluster.cloudapp.net --mysql-db=sysbench \ --mysql-user=root --mysql-password='xxx' --max-time=60 --oltp-read-only=on --num-threads=4 --max-requests=0 run 

5 minutes, 16 threads, complex test:

sysbench --test=oltp --oltp-table-size=1000000 --mysql-host=mariadbcluster.cloudapp.net --mysql-db=sysbench \ --mysql-user=root --mysql-password='xxx' --max-time=300 --oltp-read-only=on --num-threads=16 \ --max-requests=0 --oltp-test-mode=complex run 

Attaching disks

The first thing to do is to attach some new disks to the virtual machine. I will not detail the process here as it is well documented elsewhere. Here is one way to do it:

Get-AzureVM -ServiceName $sn -Name $n | Add-AzureDataDisk -CreateNew -DiskSizeInGB 128 -DiskLabel "DataDisk0" -LUN 1 | Add-AzureDataDisk -CreateNew -DiskSizeInGB 128 -DiskLabel "DataDisk1" -LUN 2 | Update-AzureVM

This will create two new raw devices in your virtual machine.

Remote PowerShell

We are going to use PowerShell remoting to run scripts on a Virtual Machine without having to connect using Remote Desktop. You can read a complete overview of Remote PowerShell on Scripting Guy's blog for example, but here is what we need to use in a nutshell:

First, we need to retrieve the WinRM URI that will be used to connect to the remote VM:

$uri = Get-AzureWinRMUri -ServiceName $sn -Name $n

Then we need to construction a PowerShell session option object to work around those pesky security restrictions... Of course I wouldn't advise doing this in production, and read up on how to install the correct certificates.

$option = New-PSSessionOption -SkipCACheck -SkipCNCheck

And finally we need to collect some credentials, this command will open a UI prompt for you to type in a username and password.

$cred = Get-Credential

Using these variables, we should be able to start an interactive PowerShell session on the remote host:

Enter-PSSession -ConnectionUri $uri -SessionOption $option -Credential $cred

Later, we will use Invoke-Command to run a script remotely.

Using DISKPART

You can feed DISKPART a text file containing the commands to run. Here is what you typically need to do to create a striped volume using the two disks we just attached. The disk and volume indices are hard-coded, so make sure to change them if your disk configuration differs from the Windows Azure standard! The indices below are based on the standard VM configuration of one persistent disk on C: and one temporary, local disk on D:.

select disk 2
clean
convert dynamic
select disk 3
clean
convert dynamic
create volume stripe disk=2,3
select volume 2
assign letter=E
format fs=ntfs quick

Putting it all together

Now we can assemble our two scripts. The first one creates the disks, then calls the second one on the remote instance via Invoke-Command.

remote_createstripe.ps1:

$sn = 'arriis'
$n = 'iis1'

$uri = Get-AzureWinRMUri -ServiceName $sn -Name $n
$option = New-PSSessionOption -SkipCACheck -SkipCNCheck
$cred = Get-Credential

Get-AzureVM -ServiceName $sn -Name $n | Add-AzureDataDisk -CreateNew -DiskSizeInGB 128 -DiskLabel "DataDisk0" -LUN 1 | Add-AzureDataDisk -CreateNew -DiskSizeInGB 128 -DiskLabel "DataDisk1" -LUN 2 | Update-AzureVM

$session = New-PSSession -ConnectionUri $uri -SessionOption $option -Credential $cred

Invoke-Command -Session $session -FilePath C:\dev\ps\diskstripe\createstripe.ps1

The second script just creates a text file containing the DISKPART commands and runs it.

createstripe.ps1:

New-Item -Name createstripe.txt -ItemType file -Force

Add-Content -Path createstripe.txt "select disk 2"
Add-Content -Path createstripe.txt "clean"
Add-Content -Path createstripe.txt "convert dynamic"
Add-Content -Path createstripe.txt "select disk 3"
Add-Content -Path createstripe.txt "clean"
Add-Content -Path createstripe.txt "convert dynamic"
Add-Content -Path createstripe.txt "create volume stripe disk=2,3"
Add-Content -Path createstripe.txt "select volume 2"
Add-Content -Path createstripe.txt "assign letter=E"
Add-Content -Path createstripe.txt "format fs=ntfs quick"

diskpart /s createstripe.txt

Hope this helps!

One well-known characteristic of the Windows Azure load balancer is that it only provides a round-robin algorithm, and does not support so-called "sticky" sessions. This is not a problem for most Web sites, however it can prove challenging when migrating more complex projects to the Cloud, where the architecture may rely on sticky sessions provided by an on-premises load balancing solution.

We typically recommend using our IIS Application Request Routing (ARR) module to implement sticky session if they are required for the application. There are several resources describing this solution, here are a few interesting links:

• Using the Application Request Routing Module : this is the standard installation documentation from the IIS team, that you could for example follow to deploy ARR on Windows Virtual Machines.
• The excellent Richard Astbury from our partner Two10Degrees has an AzureARR project hosted on Github that packages ARR in a Web Role for use in Cloud Services.

However, I recently came across a partner who required more control over the sticky-session algorithm that what ARR provides. In order to better understand the requirements, let's first look at how ARR implements session stickyness.

The technique ARR uses is usually known as a "Cookie Insertion" method. The idea is that ARR, standing in between a client browser and the server farm, will intercept incoming requests, choose a server to "stick" the session to, and will insert in the server response a custom HTTP Cookie that contains the name of the server. To be precise, ARR does not insert the server name in clear text, but rather a hash of the name, but the effect is the same. In subsequent requests, the browser will "honor" the Cookie and send it back to ARR. ARR will then keep on sending the requests in that "session" to the server whose name is in the Cookie.

To illustrate, let's look at some HTTP requests using curl:

$ curl -i http://arriis.cloudapp.net/
HTTP/1.1 200 OK
Content-Type: text/html
Last-Modified: Mon, 29 Jul 2013 10:35:50 GMT
Accept-Ranges: bytes
ETag: "7c57e964478cce1:0"
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
X-ServerName: iis1
Date: Mon, 16 Sep 2013 13:06:44 GMT
Content-Length: 10

hello iis1

This first request is made directly to the load-balanced endpoint of a Cloud Service consisting of two IIS Virtual Machines. As you can see, the request was served by a VM named "iis1". If I run this same command again, the next request will be served by "iis2", the second server in my Cloud Service: (response headers edited for brevity)

$ curl -i http://arriis.cloudapp.net/
HTTP/1.1 200 OK
X-ServerName: iis2

hello iis2

The "X-ServerName" header is not standard in IIS, I added it manually in order to facilitate troubleshooting the load-balancing setup. You would typically remove it in a production environment! Adding the header can be done using the custom HTTP Response Headers section in IIS Manager.

Now that we have seen the standard round-robin algorithm offered by Windows Azure, let's look at the HTTP responses when ARR is added as a front-end to the IIS servers:

$ curl -i http://arrarr.cloudapp.net/
HTTP/1.1 200 OK
Content-Length: 10
Content-Type: text/html
Last-Modified: Mon, 29 Jul 2013 10:34:59 GMT
Accept-Ranges: bytes
ETag: "d092b446478cce1:0"
Server: Microsoft-IIS/7.5
Set-Cookie: ARRAffinity=6e04a77b7c6917600d9ff7ee980a4d4a8a0dfb135247289a9c58cf88587bec23;Path=/;Domain=arrarr.cloudapp.net
X-Powered-By: ASP.NET
X-ServerName: iis2
X-Powered-By: ARR/2.5
X-Powered-By: ASP.NET
Date: Mon, 16 Sep 2013 13:12:43 GMT

hello iis2

As you can see, we now have a couple additional HTTP headers in the response: the "X-Powered-By: ARR/2.5" confirms that we are going through ARR, and the Set-Cookie header asks the client to add a Cookie named "ARRAffinity" to all subsequent requests.

To simulate a well-behaved browser that honors Cookie headers, we can manually add the Cookie in curl:

$ curl -i --cookie ARRAffinity=6e04a77b7c6917600d9ff7ee980a4d4a8a0dfb135247289a9c58cf88587bec23 http://arrarr.cloudapp.net/
HTTP/1.1 200 OK
X-ServerName: iis2
X-Powered-By: ARR/2.5

hello iis2

As long as we keep on sending the Cookie header, all our requests will now go to the same server in our farm. Fantastic!

For most use cases, Cookie Insertion Just Works and is very easy to setup using ARR. However, here is one very specific case where it falls short!

Where Cookie Insertion falls short

My partner has the following situation: he has a number of customers who use a library, or SDK, to connect to his Web services. Session affinity is required for his services to function, because a good amount of customer data is cached in memory in order to accelerate his processing algorithms. Because the client is a code library that does not maintain client-side state, it will not honor Cookies that are sent by the application or ARR: it only sends the header values (and Cookies) as hard-coded in the library.

They currently implement session affinity using a hardware load-balancing device that provides "rule-based load balancing". This load-balancing technique does not require inserting an additional Cookie: it uses the value of an existing application Cookie to calculate a hash that points to a given server. The load-balancing device is configured to inspect the value of a specific application Cookie which is sent by the library, let's call it "CUserID".

How can we reproduce this behavior using ARR?

Since we are looking for a hash-based load balancing algorithm, let's look at what ARR offers, extracted from the TechNet documentation for ARR:

• Server variable hash – Distributes traffic based on a hashed value of a server variable.
• Query string hash – Distributes traffic based on the hashed value of the query string value. When more than one query string name is specified, the concatenated string of the corresponding query string value is used for the hash.
• Request hash – Distributes traffic based on the hashed value of the configured server variable or URL. For example, if the server variable is QUERY_STRING, the hashed value is based on the names in the request query string.

Unfortunately, none of these options can be directly used to hash on a Cookie value!

But here is what we can do by adding a simple Rewrite Rule: we can capture the value of the Cookie using a Condition, and then create a custom Server Variable that contains the value of the Cookie. We can then use the Server Variable Hash algorithm on this custom variable to bind the session based on the Cookie value!

Let's look at the ARR configuration step by step.

First, you should disable Client Affinity if it was enabled.

Then, using IIS Manager, in the URL Rewrite module at the server level, add a new rule before the ARR rule (in this example, "myfarm_with_cookie"):

Configure the rule to match regular expressions (not wildcards - this is important for the next steps). Use .* as the pattern to match.

In the Conditions panel, add a new condition, where you will match a pattern on the HTTP_COOKIE server variable. For the pattern, use something like this:

CookieName=([0-9.A-Za-z]+)

This will match the Cookie value and store it in {C:1}.

Now in the Server Variables panel, create a new variable, e.g. LB_COOKIE in my example, and assign to it the value {C:1} to retrieve the value of the Cookie matched by the condition.

Finally, in the Action panel, just route the request to the server farm like the default rule does.

Now you have a brand new server variable you can use to base your load-balancing on!

Just go in the ARR module configuration and select the "Server variable hash" algorithm, and type the name of your variable in the text box, e.g. LB_COOKIE in my case.

You should now have a brand new Cookie-based load balancing setup, with no Cookie insertion.

Updated 3/20/2014 to use the latest version of the command-line tool.

Introduction

In this tutorial, I would like to show you how to install CDH, the Cloudera Hadoop distribution on Windows Azure Virtual Machines, using their automated installer and management toold called Cloudera Manager.

For this tutorial, since we will be deploying the platform on Linux VMs, I am going to assume that we are going to install everything from a Linux command line exclusively, using our cross-platform CLI tools and the Management Portal Web interface when required. In other words: you don't need a Windows machine to follow this tutorial ;-)

In summary, here are the steps that we are going to follow:

• Preparing our SSH keys for authentication on the cluster
• Creating the Virtual Network that defines which internal IP addresses the VMs will use
• Creating our VMs
• Configuring the DNS so that all machines can resolve their internal names and IP addresses
• Run the Cloudera Manager installer to create the Hadoop cluster

I hope you enjoy this document, and please send feedback!

Preparing our SSH keys

First, let's create an SSH key and certificate to use for authentication. This will be useful in order to log on to all our VMs without being prompted for passwords. You can find a more detailed SSH procedure in the Windows Azure documentation for Linux Virtual Machines. This is slightly different from what you would do usually to generate new SSH keys (i.e., use ssh-keygen).

$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout myPrivateKey.key -out myCert.pem
Generating a 2048 bit RSA private key
....+++
.......................................................+++
writing new private key to 'myPrivateKey.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:FR
State or Province Name (full name) [Some-State]:92
Locality Name (eg, city) []:Issy-les-Moulineaux
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Microsoft
Organizational Unit Name (eg, section) []:DPE
Common Name (e.g. server FQDN or YOUR name) []:Thomas Conte
Email Address []:tconte@microsoft.com

Windows Azure needs the SSH public key to be embedded in a PEM formatted certificate, but you could enter any information you want, it doesn't matter. It's only the generated public key we are interested in!

Let's secure the key and move it to our .ssh directory so that it used by default.

$ chmod 600 myPrivateKey.key
$ mv myPrivateKey.key ~/.ssh/id_rsa

I have found that Cloudera Manager does not like the private key format very much though. Let's convert it right now and keep the generated file somewhere, we will need it later:

$ openssl rsa -in ~/.ssh/id_rsa -out PrivateKeyCDH

Creating the Virtual Network

We will use the cross-platform command-line tools to create our VMs. If you haven't got them installed, first go through the steps to install the Windows Azure Command-Line Tools for Mac and Linux. I won't go into the details here, but once installed you should be able to type azure from the command line and get a friendly response back, plus you must import your publish settings so that the CLI is connected to your Windows Azure subscription.

Update: you can now create a Virtual Network using the command-line tool.

Create the Affinity Group:

azure account affinity-group create --location "West Europe" ezwest

Register the DNS server:

azure network dnsserver register -i mycdh 10.0.0.4

Create the Virtual Network:

azure network vnet create --address-space 10.0.0.0 --cidr 8 --subnet-name cdh --subnet-start-ip 10.0.0.0 --subnet-cidr 24 --affinity-group ezwest --dns-server-id mycdh cdhvnet

Read on if you want to understand more about the Virtual Network configuration, otherwise you can just skip to the next step!

The first thing we need to create is an Affinity Group, a kind of alias that will represent the datacenter where all our infrastructure will be created. In the Portal, on the left-hand side, at the bottom of the menu, click on Settings, then click on the Affinity Groups tab at the top of the main area. Click on Add an Affinity group, give it a name (I will use "cdh-AG") and select the datacenter (I will use North Europe).

The simplest way to create a complete Virtual Network configuration is to import a ready-made configuration file. Here is the one I used, you can save it into a file and edit it as needed. It is also possible to create a Virtual Network "manually" from the Portal and then export this configuration to a file to inspect or modify it (see Annex 1).

If you look at the XML configuration file below, you will find the following information:

• The Virtual Network name, cdh, and the Affinity Group where it will be created, cdh-AG
• The address space that will be used to assign IP addresses to the Virtual Machines (10.0.0.0/20)
• Within this address space, one or several subnets the VMs will be deployed in, 10.0.0.0/23, each being given a name that you can use when configuring the Virtual Machines (Subnet-1)
• The IP address of a DNS server (more about this below), which of course resides within the address space of the Virtual Network, i.e. 10.0.0.4 in this case
• A reference from the Virtual Network to the DNS server: all VMs will get assigned this DNS when they request their network configuration from Windows Azure

The trick here is: how did I choose 10.0.0.4 as the address of the DNS server? I know by experience that Windows Azure will allocate IP addresses sequentially within the Subnet, and that in this case, the first one it will allocate is 10.0.0.4. In short, this means that the first VM I start will become the DNS for the rest of the machines in the Virtual Network.

This is really a shortcut, and the longer way is to create the Virtual Network in the Portal, start the first VM, make a note of its IP address, and then register it as a DNS using New, Networks, Virtual Network, Register DNS Server. Importing the configuration file is just a faster way to get everything configured!

And now the configuration file:

<NetworkConfiguration xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.microsoft.com/ServiceHosting/2011/07/NetworkConfiguration">
  <VirtualNetworkConfiguration>
    <Dns>
      <DnsServers>
        <DnsServer name="mycdh" IPAddress="10.0.0.4" />
      </DnsServers>
    </Dns>
    <VirtualNetworkSites>
      <VirtualNetworkSite name="cdh" AffinityGroup="cdh-AG">
        <AddressSpace>
          <AddressPrefix>10.0.0.0/20</AddressPrefix>
        </AddressSpace>
        <DnsServersRef>
          <DnsServerRef name="mycdh" />
        </DnsServersRef>
        <Subnets>
          <Subnet name="Subnet-1">
            <AddressPrefix>10.0.0.0/23</AddressPrefix>
          </Subnet>
        </Subnets>
      </VirtualNetworkSite>
    </VirtualNetworkSites>
  </VirtualNetworkConfiguration>
</NetworkConfiguration>

Make sure you use your Affinity Group name in the AffinityGroup attribute of the VirtualNetworkSite element!

To import this configuration, in the Portal, go to New, Networks, Virtual Network, and select Import Configuration. The Virtual Network will be created and will then appear in the Networks module in the Portal.

Creating our Admin/DNS VM

We will prepare a Storage Account in our new Affinity Group, where the Virtual Machine disks will be stored. We can do this from the command line:

$ azure account storage create --affinity-group cdh-AG cdhstorage

Now let's find a good Linux image to use. You can use the following command to list all current images.

$ azure vm image list

In my case I am going to choose Ubuntu 12.04:

b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_2-LTS-amd64-server-20130415-en-us-30GB

Now let's create our first VM. This VM will become our DNS server and will also be used to run the Cloudera Manager. For this first step we ask for a "small" size since we don't need to run much stuff. We specify that we want to open the SSH port for remote shell, and add our SSH certificate so that the VM is properly setup for authentication. We also specify our Virtual Network and Subnet so that the VM gets assigned an IP address from the prefix we selected. Finally, we need to specify the Affinity Group, obviously the same we used when creating the Virtual Network.

$ azure vm create --virtual-network-name cdh --subnet-names Subnet-1 --affinity-group cdh-AG --vm-size small \
--ssh 22 --ssh-cert ./myCert.pem mycdh \
b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_2-LTS-amd64-server-20130415-en-us-30GB tom 'xxx'
info:    Executing command vm create
+ Looking up image
+ Looking up cloud service
+ Creating cloud service
+ Retrieving storage accounts
+ Configuring certificate
+ Creating VM
info:    vm create command OK

You can check the configuration details of your new VM using the following command:

$ azure --json vm list

This will output a JSON document containing the details for all your VMs, and the one you just created should look like this:

{
    "DNSName": "mycdh.cloudapp.net",
    "VMName": "mycdh",
    "IPAddress": "10.0.0.4",
    "InstanceStatus": "ReadyRole",
    "InstanceSize": "Small",
    "InstanceStateDetails": "",
    "OSVersion": "",
    "Image": "b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_2-LTS-amd64-server-20130415-en-us-30GB",
    "DataDisks": "",
    "Network": {
        "Endpoints": [
            {
                "LocalPort": "22",
                "Name": "ssh",
                "Port": "22",
                "Protocol": "tcp",
                "Vip": "137.117.198.34"
            }
        ]
    }
}

As you can see, the internal IP address assigned to your VM is indeed 10.0.0.4, and thus will be used as DNS server by all the VMs we will create next. Now let's turn this VM into an actual DNS!

You can now ssh into your new VM, if your SSH configuration is correct you should be able to log in without a password prompt. Otherwise just make sure you correctly copied your private key file into your .ssh directory.

Configuring the DNS using BIND

Now if you look at your DNS configuration, you should see something like this:

# cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.0.0.4
search 52601348a18744ec9cbf2e9c992072da.mycdh.3744114705.europewest.internal.cloudapp.net

This is fine, however in order to install our DNS we need to be able to resolve names until we have installed BIND on our VM! For now, temporarily comment out 10.0.0.4 and use a public DNS:

#nameserver 10.0.0.4
nameserver 8.8.8.8

Now you should have DNS resolution in order to run apt-get and install BIND:

# apt-get install bind9
# /usr/sbin/named

Now that BIND is running, you should be able to switch your DNS back to 10.0.0.4 in /etc/resolv.conf.

We are going to configure BIND in order to resolve our host names. We don't need a real domain, as this name resolution will only take place within the bounds of our deployment; I am going to use a fake TLD, "internal", and setup the forward and reverse zone files.

You can edit named.conf.local and add the following lines:

zone "internal" {
        type master;
        file "/etc/bind/db.internal";
};

zone "10.in-addr.arpa" {
        type master;
        file "/etc/bind/db.10";
};

You then basically copy db.empty as a starting point and add your configuration. This should look like this:

In db.internal:

$TTL    86400
@       IN      SOA     localhost. root.localhost. (
                              1         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                          86400 )       ; Negative Cache TTL
;
@       IN      NS      ns.internal.
ns      IN      A       10.0.0.4
mycdh   IN      A       10.0.0.4

In db.10:

$TTL    86400
@       IN      SOA     localhost. root.localhost. (
                              1         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                          86400 )       ; Negative Cache TTL
;
@       IN      NS      ns.internal.
4.0.0   IN      PTR     mycdh.internal.

Now you should run a few tests using nslookup to make sure everything is in order, e.g. the forward and reverse resolutions work.

To make BIND re-read its configuration file, find its PID using ps aux or in /var/run/named/named.pid, and send it a SIGHUP signal, i.e. kill -HUP (pid).

Initializing the cluster

We are now ready to create more virtual machines! These will be the actual nodes in our Hadoop cluster, and we will name them with numeric suffixes so that we can identify them easily. We will use a similar command line as above, except we are going to request larger intance sizes.

You can quickly create four instances from the Bash prompt using a mini-script like this:

$ for i in 1 2 3 4; do
> azure vm create --virtual-network-name cdh --subnet-names Subnet-1 --affinity-group cdh-AG --vm-size large --ssh 22 --ssh-cert ./myCert.pem mycdh-$i b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-12_04_2-LTS-amd64-server-20130415-en-us-30GB tom 'Pass123!'
> done

This should give you four more machines in a few minutes. You can quickly check the DNS names and IP addresses allocated like this for example:

$ azure --json vm list | egrep '(DNSName|IPAddress)'
    "DNSName": "mycdh-2.cloudapp.net",
    "IPAddress": "10.0.0.6",
    "DNSName": "mycdh-3.cloudapp.net",
    "IPAddress": "10.0.0.7",
    "DNSName": "mycdh-4.cloudapp.net",
    "IPAddress": "10.0.0.8",
    "DNSName": "mycdh-1.cloudapp.net",
    "IPAddress": "10.0.0.5",
    "DNSName": "mycdh.cloudapp.net",
    "IPAddress": "10.0.0.4",

These are the name and addresses that we will add to our BIND configuration in order to resolve all the names and addresses.

We are then going to add some additional storage, for my test I will use 150GB drives, however you could use as much as 1TB per disk, and even add several of these disks to each machine if you need even more storage.

$ for i in 1 2 3 4; do
> azure vm disk attach-new mycdh-$i 150
> done

Now we are going to switch to the first machine we created, our "admin" machine, to further configure our new VMs. Let's copy our SSH key over so that we can connect to all other VMs from our first one:

$ scp ~/.ssh/id_rsa mycdh.cloudapp.net:.ssh/

Once logged on our admin machine, let's add the IP addresses of our nodes to BIND's configuration (use kill -HUP again on the named process to force BIND to re-read its configuration files).

In db.internal:

mycdh-1 IN      A       10.0.0.5
mycdh-2 IN      A       10.0.0.6
mycdh-3 IN      A       10.0.0.7
mycdh-4 IN      A       10.0.0.8

In db.10:

5.0.0   IN      PTR     mycdh-1.internal.
6.0.0   IN      PTR     mycdh-2.internal.
7.0.0   IN      PTR     mycdh-3.internal.
8.0.0   IN      PTR     mycdh-4.internal.

To make things easier, you can also add the following line to /etc/resolv.conf in order to make "internal" the default domain searched, so that you don't have to type the complete name each time:

search internal

You should now be able to log on to all three other VMs using their FQDN, this will initialize the SSH trust and make sure you can connect without being prompted for a password.

$ ssh mycdh-1
$ ssh mycdh-2
$ ssh mycdh-3
$ ssh mycdh-4

Now we have to make it so we can run commands on all nodes as root via sudo. We are going to overwrite the configuration created by the Windows Azure Agent at first boot (/usr/sbin/waagent) to make sure we can sudo without a password. Of course you should replace the account there by the account you created when you ran the azure vm create commands. First on our admin machine:

# echo "tom ALL = (ALL) NOPASSWD: ALL" > /etc/sudoers.d/waagent

Let's do this on all our nodes:

$ for i in mycdh-1 mycdh-2 mycdh-3 mycdh-4
> do
> ssh -t $i "sudo bash -c 'echo \"tom ALL = (ALL) NOPASSWD: ALL\" > /etc/sudoers.d/waagent'"
> done

Now you can run commands as root without ever being prompted for a password. What really happens is that you ssh implicitly using your current user name ("tom" in my case), since it exists on all machines. You can then execute "sudo" via ssh, and pass it a command to execute as root in turn, e.g.:

$ ssh tomcdh-2 sudo id

If you get warnings from sudo about not being able to resolve host names, you need to add the "search" option to resolv.conf on the other machines as well:

$ for i in mycdh-1 mycdh-2 mycdh-3 mycdh-4
> do
> ssh -t $i "sudo bash -c 'echo \"search internal\" >> /etc/resolv.conf'"
> done

You will then need to partition, format and mount the new disk on every machine. You can easily log in as root using:

$ ssh -t mycdh-1 sudo -i

And then use the following shell snippet. It will partition with sfdisk the Windows Azure volume that you attached previously using azure vm disk attach-new, format it, add it to fstab, and finally mount it on /data.

echo "," | sfdisk /dev/sdc
mkfs.ext3 /dev/sdc1
echo "/dev/sdc1 /data ext3 defaults 0 0" >> /etc/fstab
mkdir /data
mount /data

Do this for all four VMs. Now you should have a formatted disk mounted on /data on every of your future nodes.

Preparing for installation

Let's apt-get update on our admin VM and all four nodes:

$ sudo apt-get update
$ for i in mycdh-1 mycdh-2 mycdh-3 mycdh-4; do ssh -t $i sudo apt-get update; done

I am now looking at the "Cloudera Manager Free Edition Installation Guide", and more precisely the section "Automated Installation of Cloudera Manager and CDH".

Let's download and run the installer:

$ wget http://archive.cloudera.com/cm4/installer/latest/cloudera-manager-installer.bin
$ chmod +x ./cloudera-manager-installer.bin
$ sudo ./cloudera-manager-installer.bin

You should now see the beautiful retro-vintage text-based Cloudera Manager installation screen! Accept (and read) all the licenses, then the installation will proceed and, if everything goes well, you will be presented with the following success screen.

As instructed, we need to connect to the Web application on port 7180. Since we are running on a server VM, we don't have a browser we can launch to open localhost, so let's instead open the port to the outside world:

$ azure vm endpoint create mycdh 7180

Note :this currently fails in the CLI, see Issue #543. For now we have to use the Portal to create endpoints on VMs that are part of a Virtual Network.

In the Management Portal, go to the Virtual Machines module, and select your admin VM in the list ("mycdh" in my case), then go to the Endpoints tab. You should see a rule fort TCP port 22 (SSH). On the dialog that appears, select Add Endpoint, click Next, then give a name to your new endpoint (e.g. "ClouderaManager"), and enter 7180 for the Public and Private port fields. The new rule should now appear in the list of Endpoints.

Now you can connect to the Cloudera Manager interface using the external URL for your admin machine (e.g. in my case, http://mycdh.cloudapp.net:7180/) and start the Web-based installation wizard.

Using the Cloudera Manager to deploy the Hadoop cluster

On the "Thank you" screen, click Continue.

On the "Specify hosts", use the following pattern to describe your hosts:

mycdh-[1-4].internal

You need to specify the FQDN, otherwise you will run into problems later when the JobTracker looks up the IP address for validation: the reverse lookup must exactly match the hostname used.

On the first "Cluster Installation" screen, I removed Impala but otherwise I left all the other options by default.

On the "SSH credentials" screen, you will need to check "Another user" and enter the username used to log on your VMs. You will also need to check "All hosts accept same private key" and provide the key we generated above. I run Cygwin on my Windows machine and I used this command to copy the key from my Linux workstation in Windows Azure to C:\TEMP:

$ scp tom@tombuntu.cloudapp.net:~/cloudera_beta/PrivateKeyCDH /cygdrive/c/TEMP

Click Continue and confirm that you are not using an SSH passphrase.

On the "Installation in progress" screen, everything should install successfully. If you encounter SSH credentials issue, this screen will help you troubleshoot the problem.

The next screen will install all the selected parcels to your machines.

On the "Host inspection" screen, everything should be green! Pay special attention to host name resolution warnings at this stage, they may cause problems later if they are not resolved now.

On the "CDH4 services" screen, choose the services you want to install. I will choose "Core with Real-Time Delivery" for a base Hadoop stack plus HBase. If you want to fine-tune which services will run on which hosts, you can click on "Inspect Role Assignments" to check what will run where. You will see that the first machine gets a larger share of services, and in production you might want to make it a larger instance size.

On the "Database setup" screen I kept the Embedded Database, then clicked on Test Connection, then Continue.

When you reach the "Review configuration changes" page, remove the /mnt/resource disk from all nodes running the HDFS service (this is a local, non-persistent disk and we don't want to store data on it). You will also need to lower the tolerance accordingly (i.e. set it to 0 in this case). Likewise for the NameNode and MapReduce services, remove all occurences of /mnt/resource.

That's it! All the services should start successfully. Congratulations!

Accessing the Hue Web UI

Now you may want to access the nice Web UI that Cloudera integrates in their distribution. On the main Services page, click on "hue1" and you will see a status page where you will find a "Hue Web UI" button. Press it, and it should fail! The management portal redirects you to the internal address of the Hue node, e.g. http://mycdh-1.internal:8888/, and this address is not known from the outside world. You should use the corresponding public address, i.e. mycdh-1.cloudapp.net in my case, and open the port (8888) by creating an endpoint in the Windows Azure Management Portal. You can proceed in the same manner whenever you need to access a service directly on one of the nodes.

Annex 1: Manually creating a Virtual Network

Connect to your Windows Azure management portal, and select New, Networks, Virtual Network, Quick Create, and just select a name and a location (data center) for your Virtual Network; all our VMs will later be created in the data center you select. You can also select the size and the address prefix, I will just go for the defaults.

Once the Virtual Network is created, click on its name and then on Dashboard to see how it was configured. You will see that an Affinity Group was automatically created, for example if you named your Virtual Network "cdh" if will be associated with a "cdh-AG" Affinity Group. Now if you click on Configure, you will find another important piece of information: a Subnet was automatically created for you, typically named "Subnet-1" if you went for the Quick Create defaults. We will need these two parameters later to create our Virtual Machines.

Annex 2: Installation the Windows Azure command-line tools for Mac/Linux

First install Node.JS, don't forget to check on the Node.JS downloads page what is the latest version!

$ sudo apt-get update
$ sudo apt-get install build-essential
$ wget http://nodejs.org/dist/v0.10.5/node-v0.10.5.tar.gz
$ tar xf node-v0.10.5.tar.gz
$ cd node-v0.10.5/
$ ./configure
$ sudo make install

Then install the Windows Azure CLI using npm:

$ sudo npm install azure-cli -g

Then initialize the Windows Azure CLI:

$ azure account import Azure-credentials.publishsettings

There is a limitation with this simple method: you cannot upload file larger than 64 MB without using a block-based upload approach, that will let you upload files up to 200 GB in size. We have some sample Python code in the Windows Azure Python Developer Center that shows how to upload large files using blocks.

If you just want to interact with Windows Azure Storage from the command line on a vanilla Linux installation though, chances are that Python is the scripting environment that is pre-installed, not requiring you to compile anything before you can start working.

Here is a quick cheat sheet on how to use the Python SDK from the command line:

Install the Windows Azure SDK for Python

Just download the latest tarball from Git, extract the contents, and run the install script as root to install the SDK:

curl -L -O https://github.com/WindowsAzure/azure-sdk-for-python/archive/master.tar.gz
tar xzf master.tar.gz
cd azure-sdk-for-python-master/src
sudo python setup.py install

Now in order to make it simpler to use the Blob Service using the Python module, you can set your Storage credentials as environment variables. You can find these values in the Windows Azure Management Portal, just select your Storage Account and click on the "Manage Access Keys" button to open the pop-up dialog that shows both the account name and the secret keys.

export AZURE_STORAGE_ACCOUNT=tcontepub
export AZURE_STORAGE_ACCESS_KEY='secret key'

Now it's super simple to access Blob Storage directly from the Python interpreter; just run python, import the module and create the BlobService object:

from azure.storage import BlobService
blob_service = BlobService()

Now here are a few code snippets you can use...

List containers in the account

for i in blob_service.list_containers():
   print i.name

Create a container

blob_service.create_container('foo')

Upload a file into a container

The first parameter is the container name, then the name of the Blob that will be created, then the data to be sent (typically the contents of a file you read from the local filesystem).

blob_service.put_blob('foo', 'README.txt', file('/usr/share/doc/grep/README').read(), 'BlockBlob')

Hope this helps. You can find more information and examples on the Windows Azure SDK for Python Github page.

The Dataset

First, let's discuss the dataset: I think Pig is particularly well suited to analysis of log files, where the structure is not always very formal and where Pig's flexibility really makes our life easier. I chose to use a publicly available dataset, Wikipedia's "Page view statistics for Wikimedia projects", which contains hourly dumps of page view counters for all of Wikipedia's projects. The page describes the dataset and how it is built, but here is a little extract of one of the log files, pagecounts-20130401-000000.gz:

fr Paris 100 15325959
fr Paris_en_chansons 2 92560
fr Paris_quadrifolia 1 16023
fr Paris_qui_dort 1 9933
fr Paris_sous_les_bombes 1 15323
fr Paris_va_bien 1 11360
fr Parisianisme 2 21184
fr Parisii 1 15944

As you can see, the logs are in a simple ASCII format. This snippet means that on April 1st 2013, during the first hour in the day (you get that bit from the file name), there were in the "fr" Wikipedia project, 100 views for the page for "Paris", 2 views for "Paris en chansons", etc. The last number is the total bytes sent.

Let's say we want to do some analysis of the page views on Wikipedia on April 1st and maybe see what good pranks were going on around the world! In order to have a full view of the traffic for that whole day we need to analyze all the logs for that day, which represents 24 gzipped files, containing 160 millions lines for a total of 7+ GB uncompressed data.

This is not a huge dataset, but it is significant enough to allow us to utilize the HDInsight cluster. And once we have a working analysis for this dataset, we can reuse the exact same method to run on a bigger sample, for example a full month, and scale our HDInsight cluster accordingly in order to get our results faster!

So, let's review all the steps needed to run our analysis.

Create a storage account

HDInsight gives you the option to use Windows Azure Storage to store your data. This is very useful in the case of ephemeral Hadoop clusters, because it will leave the data intact even when you delete the cluster; you can then choose whether you want to keep the data around to run more analysis jobs later, or if you would rather delete the data in order to save on storage costs.

To create a storage account for your HDInsight cluster, log into the Windows Azure Management Console, choose New, Storage, Quick Create, and just choose a unique DNS prefix for your account. Please note that if you are using the Public Preview of HDInsight, you will need to create your account in the "East US" region.

Create the HDinsight cluster

Now that you have a storage account for your data, you will need to create the actual HDInsight compute cluster. This is not a terribly difficult task: just go to New, Compute, HDInsight cluster, Quick Create, and just fill in the options: cluster DNS prefix, number of nodes (4 in our example), and a very secret administration password.

Once you click the OK icon, the cluster will go through several stages before it's fully operational, and this could take a few minutes; here is the final status you should see in the console:

Connect to the head node

Now that your cluster is up and running, there are a few things you can do directly from the Web management console. However, the best way to play around with Hadoop is probably to drop to the command line, and you can do that by connecting to the Head node via RDP: just click on the Connect button at the bottom of the HDInsight page on the Windows Azure Management Portal. Use the admin credentials you used when creating the cluster.

Once you are logged on the Head node, you will find a few shortcuts placed on the Desktop:

• Hadoop Command Line
• Hadoop MapReduce Status
• Hadop Name Node Status

The first you can try is double-click on the "Hadoop MapReduce Status" to open the status Web page. CLicking on the JobTracker link will take you to the page you can use to track the progress of running jobs.

To enter the Hadoop command line, double click on the "Hadoop Command Line" shortcut. A friendly help message will appear and leave you with a blank command line. Here you can start playing with HDInsight and get your bearings.

Now that we are reading to do some work, let's see how we can get our data into the system.

How to download the data efficiently!

There are many ways to import data into the HDInsight cluster in order to work with it. In the default configuration, HDInsight is configured to use Windows Azure Storage, which means that we could use any storage explorer tool (like Cloud Storage Studio or CloudXplorer) to upload data in the storage account we created above.

However, because I am located in Europe and both my source data and the HDInsight cluster are located in the United States, I am going to try and optimize the download times by retrieving the data directly from the Head node (which should be reasonably fast since I am at least downloading from the same continent!) and then copy the data from the Head node to Windows Azure Storage, which will be very fast because they are located in the same data center.

Before we start downloading, just take a quick look at the disk configuration for the Head node: as you can see, you will find most of the available node storage in the C: drive, which has around 2 TB of free disk space. This is probably the right place to use as temporary storage for our data files!

Run Notepad on the server and just copy/past the following PowerShell script, then save it e.g. in C:\TEMP\GetFiles.ps1 (create that directory first if it doesn't exist).

$wc = New-Object System.Net.WebClient
foreach ($i in 0..23) {
    $n = "{0:D2}" -f $i
    echo $n
    $url = "http://dumps.wikimedia.org/other/pagecounts-raw/2013/2013-04/pagecounts-20130401-" + $n + "0000.gz"
    $wc.DownloadFile($url, [System.IO.Path]::Combine($pwd.Path, $url.SubString($url.LastIndexOf("/")+1)))
}

This script will proceed to download in the current directory all 24 files from the Wikipedia dumps Web site, for the 1st of April; of course you can easily change the script to download another day if you prefer. I have found that sometimes there are missing hourly files, in which case the download will fail with a 404 error; you can ignore these, it just means that we will be missing some information for a given day.

To use the script, run PowserShell using the shortcut in the default Windows Server task bar. Then create a subdirectory in C:\TEMP (e.g. "wp0401") and run the script from there:

PS C:\TEMP> mkdir wp0401
PS C:\TEMP> cd .\wp0401
PS C:\TEMP\wp0401> ..\GetFiles.ps1

Using Grunt

Now that we have downloaded our source data, let's use Grunt, the interactive shell interface to Pig, to copy our source data into the Hadoop cluster so we can run jobs on it. From the Hadoop command line, change to the Pig directory and run Pig:

c:\apps\dist\hadoop-1.1.0-SNAPSHOT>cd ..\pig-0.9.3-SNAPSHOT
c:\apps\dist\pig-0.9.3-SNAPSHOT>.\bin\pig.cmd
2013-04-11 14:38:49,966 [main] INFO  org.apache.pig.Main - Logging error messages to: C:\apps\dist\hadoop-1.1.0-SNAPSHOT\logs\pig_1365691129964.log
2013-04-11 14:38:50,260 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: asv://myhdi@tomhdi.blob.core.windows.net
2013-04-11 14:38:51,068 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: jobtrackerhost:9010

The informational message confirms that Pig is now connected to our Windows Azure Storage account; you can interact with it using the usual Hadoop commands as available from Grunt, e.g. ls, cd, mkdir, etc. For example, in order to copy all our data from C:\TEMP\wp0401 where we downloaded it, you can run something like:

grunt> copyFromLocal  C:\TEMP\wp0401 .

This will copy the whole directory (it will take a little while), and you can check everything is there like this:

grunt> ls
asv://myhdi@tomhdi.blob.core.windows.net/user/admin/.Trash      <dir>
asv://myhdi@tomhdi.blob.core.windows.net/user/admin/wp0401      <dir>
grunt> ls wp0401
asv://myhdi@tomhdi.blob.core.windows.net/user/admin/wp0401/pagecounts-20130401-000000.gz<r 1>   89406292
asv://myhdi@tomhdi.blob.core.windows.net/user/admin/wp0401/pagecounts-20130401-010000.gz<r 1>   81022448
asv://myhdi@tomhdi.blob.core.windows.net/user/admin/wp0401/pagecounts-20130401-020000.gz<r 1>   83019186

The "ASV" prefix means that the files are stored in Windows Azure Storage, which means that it will stay there even after you shut down the HDInsight cluster.

The Pig script

And now let's look at the Pig script we are going to run!

Actually, Grunt lets you run Pig jobs interactively, which means that you can copy/paste each line in the command window and thus build the script gradually. Pig will not really execute each line when you enter it, but it will build its processing pipeline, and it will only start the process once it reaches the final "store" statement.

Typically, if you want to run a quick test to have a look at the results, and validate or tweak your processing, you are not going to run the script on the complete dataset but on a sample, for example in our case we could use just one file instead of all 24 files for one day.

records = load '/user/admin/wp0401/pagecounts-20130401-*.gz' using PigStorage(' ') as (project:chararray, page:chararray, requests:int, size:int);
filtered_records = filter records by project == 'en';
records_by_page = group filtered_records by page;
sum_pages = foreach records_by_page generate group, SUM(filtered_records.requests);
result = order sum_pages by $1 desc;
result_limit = limit result 100;
store result_limit into 'wp_april_1st';

This script will produce the result we are looking for, i.e. the most popular English language pages on Wikipedia on April 1st. Let's read it line by line!

records = load '/user/admin/wp0401/pagecounts-20130401-*.gz' using PigStorage(' ') as (project:chararray, page:chararray, requests:int, size:int);

This first statement is our LOAD command, where we instruct Pig while files to read, and how to parse them. In this case, the fields are separated by space characters, and we just have three fields: the "Wikipedia project", which also gives us the language, the page name, the number of requests within the hour, and the total bytes sent for that page. As you can see, we can give names to these fields, and assign them a data type like "chararray" (string) or int.

filtered_records = filter records by project == 'en';

The FILTER statement is used, you guessed it, to filter our records. Here, we only keep records from the main Wikipedia project in English.

records_by_page = group filtered_records by page;

Now, because we want to aggregate all the requests per page, we GROUP the records using the "page" field.

sum_pages = foreach records_by_page generate group, SUM(filtered_records.requests);

The FOREACH directive is used to create a projection of our data; here we are going to generate a new record set containing the group name (remember from the previous step, this is the "page" field) and the SUM of the values of the "requests" field. To understand where the "filtered_records.requests" expression comes from, you could use the handy DESCRIBE command to visualize the structure of the records generated by the GROUP command:

grunt> describe records_by_page;
records_by_page: {group: chararray,filtered_records: {(project: chararray,page: chararray,requests: int,size: int)}}

This is how you can see that you need to sum the "requests" field from the "filtered_records" tuple in order to find the total number of requests for a given page.

result = order sum_pages by $1 desc;
result_limit = limit result 100;

These two lines are fairly self-explanatory: ORDER will sort our records, and LIMIT will only keep the first 100, because we certainly do not want to deal with the incredibly long tail of Wikipedia pages that are only accessed once per day!

store result_limit into 'wp_april_1st';

And our final statement will tell Pig to store the results in the "wpapril1st" file. This is the statement that will actually tell Pig to work its magic and start running the MapReduce jobs. As you can guess, depending on the amount of data that you LOADed in the first statement, the computation can take quite a while.

That's all, folks!

The output in Grunt is fairly verbose, and while the jobs are running you can check their progress using the Hadoop Status Page link on the Desktop (Grunt also gives you some URLs pointing directly to the jobs status pages). Pig will generate and chain several MapReduce jobs as required.

At the end, Pig will give you some interesting stats about the run, including the total time taken to process the data, the amount of data read, the MapReduce jobs generated, etc. For example, in my test, it took about 6 minutes to process 160 million lines of logs:

• Started At: 2013-04-16 07:26:39
• Finished At: 2013-04-16 07:33:58
• Input(s): Successfully read 163288474 records (8954 bytes) from: "/user/admin/wp0401/pagecounts-20130401-*.gz"
• Output(s): Successfully stored 100 records in: "asv://myhdi@tomhdi.blob.core.windows.net/user/admin/wpapril1st"

And of course, the result! The easiest way to read the results in our case (because there are only 100 records) is to just cat the file from Grunt:

cat wp_april_1st

But you could also use copyToLocal to copy the file to your local disk and read it using Notepad. Without further ado, the results:

Main_Page                       8695731
Special:Search                  2735038
Special:Random                  1844895
April_Fools%27_Day              502216
Diaper                          405342
Dumpster                        379215
Durian                          352748
Airport_terminal                345508
Waffle                          295425
The_Walking_Dead_(TV_series)    265686
undefined                       259862
Game_of_Thrones                 235499
Ground_Zero_(1987_film)         232729

No surprise, the main page comes first, followed by two "special" pages. "April Fool's Day" is the most viewed page after that. I think the next three results, "Diaper", "Dumpster" and "Durian", plus "Waffle" after that may have something to do with Google's Google Nose joke :-)

Hope you enjoyed this little tutorial!

There is already a blog post out there showing you how to access the Management API using cURL (http://blog.toddysm.com/2012/01/accessing-windows-azure-rest-apis-with-curl.html) however it doesn't cover the creation of VMs, so here are a few more samples.

First, the blob post above shows you how to create a new certificate and upload it to the Windows Azure management portal. But there is a simpler way, using a "Publish Settings" profile, that you use for example with the cross-platform Command Line Tools.

Here is the article that will explain you how to download this publishing profile: https://www.windowsazure.com/en-us/manage/linux/common-tasks/manage-certificates/

As explained in the article, you can extract from the profile the Base64-encoded certificate necessary to authenticate your calls to the Management API.

Let's say you saved the Base64 version in a file called cert.base64; here's how I decoded it:

python -c 'print open("cert.base64", "rb").read().decode("base64")' > cert.pfx

Then I checked it using OpenSSL: (password is empty)

openssl pkcs12 -info -in cert.pfx

And finally you can convert it to PEM format: (again, password is empty)

openssl pkcs12 -in cert.pfx -out cert.pem -nodes

Now you can used this certificate with the -E option in cURL! You should also extract from the Publish Settings file your Subscription ID, that you will use to construct the URLs below.

Here are some examples... Again, the goal is to create a Linux virtual machine; in order to do that, we must first create a Hosted Service, and then "deploy" a Linux VM inside this Hosted Service. Here are the steps using cURL. All the URLs begin with your subscription ID, that you can find in the publishing profile as well.

Create a Hosted Service:

curl -v -d @CreateHostedService.xml \
-H 'Content-Type: application/xml' \
-H 'x-ms-version: 2010-10-28' \
-E cert.pem: \
https://management.core.windows.net/(subscription-id)/services/hostedservices

The CreateHostedService.xml file contains the parameters for the Hosted Service:

<?xml version="1.0" encoding="utf-8"?>
<CreateHostedService xmlns="http://schemas.microsoft.com/windowsazure">
  <ServiceName>testfromapi</ServiceName>
  <Label>VGVzdCBmcm9tIEFQSQ==</Label>
  <Description></Description>
  <Location>West Europe</Location>
</CreateHostedService>

The most important part in this file are the Location element, in which you indicate the data center you want to create this Hosted Service in (I chose West Europe), and the ServiceName element, that will be used later to deploy a VM.

List the Hosted Service:

curl -H 'Content-Type: application/xml' \
-H 'x-ms-version: 2010-10-28' \
-E cert.pem: \
https://management.core.windows.net/(subscription-id)/services/hostedservices/testfromapi

This should return your newly created Hosted Service.

List OS images:

curl -H 'Content-Type: application/xml' \
-H 'x-ms-version: 2012-03-01' \
-E cert.pem: \
https://management.core.windows.net/(subscription-id)/services/images

This will return a list of all available OS images. Let's use Ubuntu: CANONICAL__Canonical-Ubuntu-12-04-amd64-server-20120528.1.3-en-us-30GB.vhd

Now the last step, create the VM:

curl -v -d @TestLinuxVM.xml \
-H 'Content-Type: application/xml' \
-H 'x-ms-version: 2012-03-01' \
-E cert.pem: \
https://management.core.windows.net/(subscription-id)/services/hostedservices/testfromapi/deployments

Please note the URL references the name of the Hosted Service we created earlier ("testfromapi" in my example).

Now let's look at the XML file:

<Deployment xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.microsoft.com/windowsazure">
  <Name>testlinux</Name>
  <DeploymentSlot>Production</DeploymentSlot>
  <Label>testlinux</Label>
  <RoleList>
    <Role i:type="PersistentVMRole">
      <RoleName>testlinux</RoleName>
      <OsVersion i:nil="true" />
      <RoleType>PersistentVMRole</RoleType>
      <ConfigurationSets>
        <ConfigurationSet i:type="LinuxProvisioningConfigurationSet">
          <ConfigurationSetType>LinuxProvisioningConfiguration</ConfigurationSetType>
          <HostName>testlinux</HostName>
          <UserName>azure</UserName>
          <UserPassword>P@ssw0rd!</UserPassword>
          <DisableSshPasswordAuthentication>false</DisableSshPasswordAuthentication>
        </ConfigurationSet>
        <ConfigurationSet i:type="NetworkConfigurationSet">
          <ConfigurationSetType>NetworkConfiguration</ConfigurationSetType>
          <InputEndpoints>
            <InputEndpoint>
              <LocalPort>22</LocalPort>
              <Name>SSH</Name>
              <Protocol>tcp</Protocol>
            </InputEndpoint>
          </InputEndpoints>
        </ConfigurationSet>
      </ConfigurationSets>
      <DataVirtualHardDisks />
      <Label>dGVzdGF0b3Rv</Label>
      <OSVirtualHardDisk>
        <MediaLink>https://portalvhdsfxcplfk8zny2r.blob.core.windows.net/vhds/testfromapi-testlinux-2012-11-29-593.vhd</MediaLink>
        <SourceImageName>CANONICAL__Canonical-Ubuntu-12.04-amd64-server-20120924-en-us-30GB.vhd</SourceImageName>
      </OSVirtualHardDisk>
      <RoleSize>Small</RoleSize>
    </Role>
  </RoleList>
</Deployment>

This one is more complicated! There are good articles on Michael Washam's blog if you want to dig into the details, but here are the important parts:

• In the LinuxProvisioningConfigurationSet, enter the login and password for the user that will be created on the machine.
• The InputEndpoint definition opens port 22 (SSH).
• In the OSVirtualHardDisk section, you must indicate the URL of the blob that will contain the new OS Disk for the machine; you can choose any name for the VHD, as long as it is unique! Also make sure the URL points to a Storage Account located in the same data center as the Hosted Service you created above, otherwise the call will fail (you cannot boot a machine in Dublin with an OS Disk in Singapore :)

This should give you a running Linux VM.

You can retrieve the details of the VM once it's created:

curl -H 'Content-Type: application/xml' \
-H 'x-ms-version: 2012-08-01' \
-E cert.pem: \
https://management.core.windows.net/(subscription-id)/services/hostedservices/testfromapi/deployments/testlinux

You will get an XML response including, among other details, the status of the instance (Running, Stopped, etc.) and, most importantly, the public port that was assigned for SSH when the VM was created (look for the PublicPort element). This should let you connect to your new VM via SSH.

Enjoy!