Ich bin umgezogen (blogmäßig)! Unter http://oliviaklose.com schreibe ich weiterhin über Azure, Big Data, Machine Learning und ähnliches. Wir sehen uns im neuen Blog :)

Diese Machine Learning Serie ist wie folgt aufgebaut (und wird natürlich stets erweitert):

1 Was ist das (nicht) und wann?
2 Supervised versus Unsupervised Learning
3 Machine Learning Prozess
4 Azure Machine Learning: Was und Warum?
5 Experiment in Azure ML: Binäre Klassifizierung – Flugverspätungen

Wie man sehen kann, wird diese Serie sich mit Microsoft Azure Machine Learning beschäftigt – ein Dienst von Microsoft Azure, mit dem Entwickler Predictive Analytics-Modelle anhand von Machine Learning-Algorithmen bauen und als Web Service veröffentlichen können.

Wo findet man Machine Learning?

Machine Learning findet man überall. Meine Lieblingsbeispiele von Machine Learning findet man auch gleich im Alltag: die Kinect, Skype Translator und Netflix.

Die Kinect ist ein gutes Beispiel, wo Machine Learning und Computer Vision sich überlappen. Das Ziel von Machine Learning bei der Kinect ist es, alle Körperteile in allen möglichen Positionen, Körpergrößen und –formen in Super-Echtzeit zu tracken/folgen:

Um die Sensorik innerhalb der Kinect kümmere ich mich jetzt nicht, sondern eher, was danach kommt: die erfassten Bilder zu verstehen. Angefangen mit einem Tiefenbild wird anhand von Random Forests jedes einzelne Pixel des Tiefenbilds (ohne Hintergrund) in eines der Körperteil-Kategorien klassifiziert. Daraufhin werden die Körperbereiche in “body joints” (quasi Gelenkpositionen) mit Clustering “zusammengefasst”, um diese dann miteinander zu einem Skelett zu verbinden. Natürlich ist das hier sehr oberflächlich behandelt; das Paper Real-Time Human Pose Recognition in Parts from a Single Depth Image (2011) von MS Research  gibt noch  viel tiefgründigere Informationen. Was nennenswert ist, ist dass für das Trainieren der Machine Learning Modelle hinter der Kinect (u.a. Random Forests und Clustering) 100.000 verschiedene Köperhaltungen vor der Kinect “durchgespielt” wurden.

Was für großartige Sachen das Microsoft Research (Cambridge), in den Bereichen Computer Vision und Machine Learning machen und wo man ihre Forschungen im alltäglichen Leben wiederfindet, könnt ihr in diesem Blog nachlesen.

Skype Translator ist ein weiteres Beispiel, wo einige Bereiche von Machine Learning (also Spracherkennung, automatische Übersetzung und weitere Machine Learning Methoden) unseren Alltag beeinflussen. Was Skype Translator macht, ist zunächst Deine Stimme oder eher Deine Worte erkennen (Deep Learning oder Deep Neural Networks), die erkannten Worte “bereinigen” und dann in eine andere Sprache übersetzen. Beim Übersetzen sind wiederum Machine Learning Modelle im Einsatz. Natürlich kann das Modell nicht alles schon perfekt aussagen; deswegen wir das System immer weiter verbessert, indem wir Feedback geben. In diesem Video kann man Skype Translator schön im Einsatz sehen – vor allem kurz nach der gewonnen Fußball-WM 2014 :)

Und natürlich kann man im Wohnzimmer Netflix fast gar nicht mehr wegdenken. Hinter den persönlichen Empfehlungen versteckt ebenfalls sich Machine Learning. Diese Empfehlungen (Collaborative Filtering) basieren auf vielen Faktoren: was Du bisher gesehen hast, wie oft Du einen Film geschaut hast, wie schnell Du eine Serie verschlungen hast, ob und wann Du in einem Film oder einer Serie pausiert hast, etc. Langfristig soll der Empfehlungsdienst die Übersichtsseite über alle angebotenen Filme und Serien ersetzen, vor allem weil die bisherigen Empfehlungen von Netflix sowieso schon als Navigation für 75% der Abonnenten dient. Sehr interessant ist, wie Netflix Big Data für die Erstellung von neuen eigenen Serien (Netflix Original Series) nutzt. So wurde 100 Millionen USD investiert, um auf eine übliche Pilotfolge zu verzichten und gleich ein 2-Staffel-Remake aus der gleichnamigen BBC-Miniserie House of Cards zu produzieren. Dieses durchdachte Risiko basiert auf Daten, die besagen, dass Fans von der Original-BBC-Serie ebenfalls Filme mit Kevin Spacey oder produziert von David Fincher verschlingen. Hat sich jedenfalls bei dem Zuwachs von 17 Millionen Abonnenten gelohnt – und für den Zuschauer selbst alle Male.

Darüber hinaus ist Predictive Maintenance ein guter Anwendungsfall von Machine Learning in der Industrie: sei es in der Produktion, darüber benachrichtig zu werden, dass Gerät/Sensor A zu einer Wahrscheinlichkeit von 90% in den nächsten zwei Wochen ausfallen wird, oder in der Kundenansprache, wenn man als Kunde einen Anruf erhält, weil aufgrund des Fahrverhaltens die Bremsscheiben in der nächsten Woche auszutauschen.

Was ist Machine Learning?

Also akzeptiert, Machine Learning kann man in allen möglichen Szenarien gut einsetzen. Was ist denn jetzt genau Machine Learning? Für Machine Learning gibt es zwei gängige Definitionen:

A computer program is said to learn from experience E with respect to
some class of tasks T and performance measure P,
if its performance at tasks in T, as measured by P
improves with experience E.

Tom M. Mitchell, Machine Learning, 1997

In anderen Worten, je mehr Erfahrungen E wir machen, desto besser können wir die Aufgabe T lösen (gemessen an Leistung P).

Also ganz nach dem Motto “Übung macht den Meister”. Zum Beispiel, mein Schachspiel verbessert sich, je mehr Schach ich gegen verschiedene Spieler spiele:

• T = Schach spielen,
• P = Anzahl an Schachspielen, die ich gewonnen habe,
• E = Anzahl an Schachspielen, die ich gegen andere gespielt habe.

Ein anderes Beispiel ist die Platzierung von Werbungen auf Internetseiten. Mein Ziel ist, so viele Nutzer zum Anklicken der einen Werbung zu bringen. Je mehr Leute auf die Internetseite kommen, können wir mehr über diese in Erfahrung bringen, ob sie überhaupt auf die Werbung klicken oder nicht. Dies können wir dann wiederum in die Optimierung der Werbung (in Farbe, Größe, etc.) einfließen lassen, um die Werbung und dann auch die Klickrate wieder zu verbessern.

Eine weitere Definition von Machine Learning ist wie folgt:

Machine learning is programming computers to optimize a
performance criterion using example data or past experience .

Introduction to Machine  Learning, Second Edition, MIT Press

In dieser Definition ist speziell von Beispieldaten und historischen Daten die Rede. Was oft unterschätzt wird, ist, dass für das Erstellen von Machine Learning Modellen bereits vorhandene Daten nötig sind, und zwar viele von denen. Basiert auf den Modellen kann man dann bei künftigen Events “die Zukunft vorhersagen”.

Zusammengefasst kann man Machine Learning (ML) so darstellen:

Bei ML reden wir von System, die

• sich automatisch an neue Trends (durch eingeflossenes Feedback) und ohne manueller Eingabe anpassen,
• neue Daten auf Basis von historischen Daten vorhersagen bzw. Wahrscheinlichkeiten von Ausgängen berechnen,
• Aktionen optimieren (Deskriptive Statistik),
• Informationen aus vorhandenen (Beispiel-)Daten extrahieren, oder
• aus einem großen Datentopf die relevanten Daten und Spalten/Attribute herausfinden und dementsprechend die Daten zusammenfassen.

Wir werden auf einige Punkte detaillierter in 3 Supervised versus Unsupervised Learning eingehen.

Was ist Machine Learning nicht?

Ebenso wichtig ist schon mal im Voraus die richtige Erwartungshaltung gegenüber ML zu haben, und zu verstehen, was ML nicht ist:

Machine Learning funktioniert nur, wenn man eine Datenmodellierung vornimmt und die einzelnen Spalten/Attribute/Features genauer untersucht (Feature Engineering), da nicht jedes Feature relevant ist. Die Erwartung, dass mit Machine Learning automatisch (ohne Eigenarbeit) einem gesagt wird, welche Daten denn relevant sind oder was ich am Montag machen soll, ist falsch – also kein “Garbage-In-Knowledge-Out”. Den Datenmüll muss man schon vor dem ML-Modellbau säubern.

ML ist auch nicht dafür gedacht, Regeln auswendig zu lernen. Wenn bereits viele Regeln vorhanden sind, die alles abdecken, ist ML einfach ein Overkill. ML ist nur dann in einem Bereich gut geeignet, wenn man nicht alles mit Regeln abdecken kann, z.B. Handschrift- oder Spracherkennung.

Wann soll ich Machine Learning einsetzen?

Joseph Sirosh, CVP für Machine Learning bei Microsoft, hat auf einer Konferenz vier Regeln aufgelistet, wann ML eine gute Option ist:

In Deutsch:

• Komplex, z.B. Natural Language Processing, Handschrifterkennung, Computer Vision
• Zum Skalieren, z.B. Spam, Betrugsaufdeckung (Fraud Detection), Gesundheitswesen (CTs,…)
• Personalisiert, z.B. Amazon, Netflix
• Autonomes Lernen, z.B. Robotik

Pi mal Daumen: Wenn Du viele Daten bereits vorhanden hast (Millionen von Datenpunkten), eignet sich ML ganz gut. Wenn jedoch schon viele Regeln vorhanden sind oder man wenige Daten hat, sollte man die Finger von ML zunächst noch lassen.

Here, we were walking through the architectural considerations and decisions made in building an HDInsight solution. Short reminder: HDInsight is a Hadoop implementation as a platform as a service (PaaS) on Microsoft Azure. The HDInsight solution was to drive visitor experience and provide a personalised view using recommendations.

The session is structured along the typical Data Warehouse workflow:

As we go through every step we highlight the agony of choice between various technologies (both open source and Microsoft Azure services) especially in the big data space:

Mapping the technology options to each step within the data warehouse workflow:

And the final implementation workflow:

You can find the presentation also here on slideshare.

Contents

1 Introduction
2 Step-by-Step: Build the Infrastructure
3 Install a Hadoop Distribution

Step-by-Step: Install a Hadoop Distribution

1. Install Ambari Server
2. Install Hadoop

Now that we have set up the infrastructure for a Hadoop cluster in Azure, it is time to get our hands dirty with installing the actual Hadoop distribution.

1. Install Ambari Server

We start off with installing an Ambari Server that allows for a “graphical” way of installing and deploying Hadoop.

a. Set Up Bits
b. Set Up Ambari
c. Start Ambari

a. Set Up Bits

Log onto your master node (in this case oldkHDPm) as root. This node will serve as the main Installation host. Download the Ambari repository. Since we use CentOS 6 as our platform, access the repository as follows:

wget http://public-repo-1.hortonworks.com/ambari/centos6/1.x/updates/1.5.1/ambari.repo

Following, copy the files to your repos.d:

cp ambari.repo /etc/yum.repos.d

More information can be found here in the Hortonworks documentation.

You can confirm that the repository is configured, by running yum repolist. You then obtain a list of repo id’s and repo names as marked in blue below. The command may vary depending on the platform (see here for more information).

Now, we can install the Ambari bits by running yum install ambari-server.

b. Set Up Ambari

Now that the Ambari server is installed, let us set it up. Run ambari-server setup.

Here, we do not customise the user account for the ambari-server daemon since we have already changed the root password. Likewise, we accept the default settings. More Information can be found here in the Hortonworks documentation.

c. Start Ambari

The Ambari server is set up and installed – ready to be started:

ambari-server start

To have a look at the Ambari server processes, type in:

ps –ef | grep ambari

In case, more than one process is running ambari, kill the other process as follows:

2 Install Hadoop

Now we are ready to install the Hadoop distribution, i.e. HDP 2.1, using Ambari. Likewise, we will go along the Hortonworks documentation (here).

Log into your DNS server and open an internet browser pointint to

http://{ambari-server}:8080

In this case: http://oldkHDPm.oldkHDP.oliviak.com:8080

Name your cluster (see Hortonworks documentation), e.g. oldkHDPcluster:

Select your desired stack (see Hortonworks docs). We choose the latest for the time being, i.e. HDP 2.1:

The next window specifies the install options. Before we go into it, we take a little de-tour, i.e. how to copy the SSH private key onto the DNS server.

Detour: How to Copy the SSH Private Key to the Local Machine

For that purpose, we install WinSCP that enables the secure file Transfer between a local and a remote computer. Once installed, log in using the credentials to the master node (i.e. oldkHDP.cloudapp.net, port 22):

Use the WinSCP client to download the private SSH key (i.e. id_rsa) of the master node into the DNS server:

Once downloaded into the “local” machine, i.e. our DNS server, we can browse for it in the “Install Options” window:

Additionally, type in all the target hosts of your Hadoop cluster. In this case, it includes the master node and the three worker nodes:

oldkHDPm.oldkHDP.oliviak.com
oldkHDPw[1-3].oldkHDP.oliviak.com

When registering and confirming, you will be prompted with another window containing the host name pattern expressions:

Success – the hosts are confirmed. Have a look at the Hortonworks documentation for more information.

You may or may not receive some warnings as shown in the yellowish area:

It turns out that the ntpd services are not running but are required to be. You could run the HostCleanup Python script on each host…

…or manually get the ntpd services to run, by running

chkconfig ntpd on

on each host, i.e. the master and all three worker nodes:

To check the status of the ntpd services, run

service ntpd status

Back in the browser on the DNS server, rerun the checks:

Next, you can choose the services you wish to install on your Hadoop Cluster (see HDP documentation).

Next, select the hosts on which certain master components should run (see HDP doc). In this case, I choose to assign the master components of the Hive Server and the Oozie Server to the master node.

With the Ambari wizard, slave components (i.e. DataNodes, NodeManagers and RegionServers) can be appropriately assigned to certain hosts in the next window (see HDP doc).

Now you can manage the configuration settings for the Hadoop components along the tabs:

For instance, under HDFS we change the directories from

to the following:

For the remaining tabs marked with warnings, credentials are required, such as Nagios,

…Hive…

…and Oozie:

More Information on customising the services related to your Hadoop cluster can be found here.

Finally, before deploying the Hadoop cluster you obtain the usual summary of configuration settings:

It contains the following information:

Services

And away you deploy:

Finally, you obtain a summary of your Hadoop installation efforts:

Done!

You have the Ambari GUI nicely displayed in front of you:

While hovering over the tiles, you obtain more information, such as on the network usage:

or the cluster load:

Contents

1 Introduction
2 Step-by-Step: Build the Infrastructure
3 Step-by-Step: Install a Hadoop Distribution

Step-by-Step: Build the Infrastructure

In this article, we set up the infrastructure and configure the virtual machines to enable the installation of a Hadoop cluster via an Ambari Server.

We heavily base our step-by-step guide on Benjamin’s great article How to install Hadoop on Windows Azure Linux virtual machines and Hortonworks’ documentation Hortonworks Data Platform – Automated Install with Ambari.

The infrastructure is set up as follows:

What you will learn in this post (amongst others):

• Create a DNS Server in Azure IaaS
• Linux on Azure:
  • Create a Linux VM
  • Log on to a Linux VM
  • Capture a Linux VM
  • Authenticate using SSH Keys
  • Set up passwordless SSH
  • Attach Disks
• Create several virtual machines using PowerShell

The article is structured as follows:

1. Create Virtual Network
2. Create Storage Account
3. Create DNS Server
4. Create Linux Image
5. Create Master and Worker Nodes
6. Configure DNS Server
7. Configure all nodes (root password, network, iptables, passwordless SSH, format empty disk)

1. Create Virtual Network

First, we will create a virtual Network in which the nodes for the Hadoop cluster will be created. In the Azure management portal, you simply click on New and custom create a virtual Network:

Choose a name for your virtual network of your liking. The datacenter location best suited for your Hadoop cluster is one nearby yours. Here you find an overview of all available Azure regions.

Later on, we will create a DNS server for distributing domain names for the nodes as well as acting as a desktop environment:

Let’s add some subnets: the first one for the DNS server,

In summary:

In the Azure documentation you can find more information on configuring a cloud-only virtual network in the management portal.

2. Create Storage Account

Each virtual machine requires a storage account. Either you can use an automatically generated storage account when creating a VM, but this usually ends up being some ugly looking name that noone can remember. Hence, take the proactive route and create a storage account beforehand:

We choose the locally redundant replication option to minimise costs.

In short:

Note that the location chosen here must be the same as the one from the virtual network created beforehand.

Some more Information on how to create a storage account can be found here in the Azure documentation.

3. Create DNS Server

In this paragraph, we will create the DNS Server for our Hadoop cluster. For simplicity, we use a VM based on Windows Server, but a Linux-based VM can obviously be also used for the purpose of a DNS Server.

We first will create an ordinary Windows Server VM, and then will assign the DNS role to it in the second step.

1. Create Windows Server VM
2. Add DNS Role
3. Create New DNS Zone

3.1 Create Windows Server VM

Let’s get started with creating our first VM in the virtual network. The short summary is as follows:

In longer version: click on New in the bottom left corner of the Azure management portal:

You choose your Image, i.e. Windows Server 2012 R2 Datacenter:

Now you specify the virtual machine’s name, e.g. oldkHDPdns, and its credentials:

We create a new cloud service (named <cloud-service> i.e. oldkHDP), and use the previously created virtual network to specify the first subnet. Likewise, we use the previously created storage account:

Once created, you obtain a nice overview of all important info on our DNS VM, such as the associated disk, its DNS name, internal IP address and much more:

3.2 Add DNS Role

So far we have created a general VM. To make it a DNS server, we will have to remotely connect to the virtual machine and add the DNS role. With Windows Server 2012 R2, this is easily done from the Server Manager:

This will take you through a nice and easy wizard to add the necessary roles and features for it to be a DNS server:

Voila, the VM can now be called a DNS Server!

3.3 Create New DNS Zone

Let’s now define zones in the DNS Server via the DNS Manager. You can reach it as usual from the ubiquitous Server Manager:

We add a new zone for the virtual network oldkHDP, in typical Windows style – a wizard! In short, you configure the following:

As a result, you will find the newly created forward lookup zone with two files contained:

Same will be done for creating the corresponding reverse lookup zone to translate IP addresses into DNS names:

In short:

You may want to turn off enhanced security for the Internet Explorer to simplify things later on:

4. Create Linux Image

1. Create Linux VM
2. Disable SELinux
3. Edit hosts file
4. Disable PackageKit
5. Capture Image

Here we want to create a custom Linux image that we will then use for creating the master node and the three worker nodes comprising our Hadoop cluster. Why the hassle? Well, alternatively we could use a normal Linux image but then will have to repeat certain configurations for each node. Now instead we are making the configurations once and create all nodes based on that.

4.1 Create Linux VM

In short:

More elaborate, let’s create a new virtual machine from the Azure management portal and choose the OpenLogic Image. Alternatively, one can also use an Image based on Ubuntu or Suse.

You can provide any name of your liking. Note that this VM will later be used as a custom image for the nodes of the Hadoop cluster, so a name with image or template might be suitable. For authentication we will provide a Password.

We use the cloud service we have created with the DNS server beforehand. Similarly, select the storage account created in 2. Create Storage Account.

And now we have created a Linux VM. More Information on how to create a virtual machine running Linux on Azure can be found here.

To remotely connect to our Linux VM, I recommend using PuTTY – a free SSH and telnet client for Windows. Type in the SSH Details marked above into the host name to connect with the Linux VM. To make life easier later, save the session as well.

And then log in with the credentials that you have provided when creating the virtual machine:

For more Information on logging on to a Linux VM, have a look at the Azure documentation.

4.2 Disable SELinux

The documentation provided by Hortonworks provides a very nice step-by-step guide. Disable SELinux to setup the Ambari server later on, see here in the Hortonworks documentation.

4.3. Edit Hosts file

The hosts file on each host within our to-be-created cluster needs to include the IP address and FQDN of each host, as written in the Hortonworks documentation. As opposed to doing it later on every single host, do these changes to the hosts file now in the Linux template that serves as the foundation for each node later on. In the Hortonworks docu

Our cluster will consist of 1 DNS server, 1 master node and 3 worder nodes. In the hosts file, you specify the IP address, the FQDN and the shortname, corresponding to the primary zone you have created in the DNS (2.3.2):

4.4. Disable PackageKit

Since we have CentOS installed, disable PackageKit as follows:

In the Hortonworks documentation, refer to here.

4.5. Capture Image

In Azure we can capture the state of this virtual machine into an image to deploy several virtual machine instances, i.e. master and worker nodes, later on. First, undo provisioning customisation by running

waagent –deprovision

and shut down the VM running

shutdown –h now

In the Azure management portal, click on Capture in the bottom.

More information on capturing a Linux virtual machine can be found here in the Azure documentation.

5. Create Master and Worker Nodes

1. Create Master Node and Attach Empty Disk
2. Create Worker Nodes

Now we can create the nodes for our Hadoop cluster using the template we have created in 2.4 Capture Linux Image.

5.1. Create Master Node and Attach Empty Disk

Start with creating a master node in the Azure management portal. In short:

As described beforehand when connecting to our Linux image, we can check the internal IP address and SSH details on the dashboard of the newly created master node in the Azure Management portal, and connect via PuTTY:

Once logged in, we can check the internal IP address one more time running sudo ifconfig.

Back in the Azure management portal, attach an empty disk to the master node of size 100 GB:

Shortly after, we see the empty disk listed on the dashboard of the master node:

5.2. Create Worker Nodes

The same applies to the three worker nodes, with the difference that they will be in Subnet-3. We can do this via the Azure management portal as done before in 5.1 Create master node, but since we would repeat the whole process three times, this literally calls out for PowerShell!

It essentially creates three VMs, each with following Information:

In the PowerShell script, an empty disk is attached to each VM, followed by the command updating the VM itself.

6. Configure DNS Server 

Now that the master node and the three worker nodes have been created, the DNS server needs to be configured. In other words, the nodes have to be inserted manually as hosts in the zone we have created in 3. 2 Add DNS Role. Add a new host for each node with the same information provided in 4.3 Edit Hosts File.

So adding the master node as a host looks as follows:

Eventually, you have inserted four additional hosts:

Following, edit the hosts file to be found in C:\Windows\System32\drivers\etc by inserting the new Information as follows:

Checking if the Connections have been established correctly, you can ping all nodes from the DNS Server…

…and from the master node:

7. Configure all nodes

1. Hadoop-specific and little configurations
   1. Change root password
   2. Edit network configuration file
   3. Configure iptables
   4. Repeat for all nodes
2. Set Up Password-less SSH
3. Format empty attached disks
   1. Master node
   2. Worker nodes

7.1 Hadoop-specific and little configurations

Here, we configure each node regarding aspects to either enable passwordless SSH or the installation of the Ambari Server later on.

7.1.1 Change root password

Why change the root password? Later on we want to install the Ambari server using the root account so that no password will be required. For that, change the root password in each host using the commend sudo passwd root. Start with the master node:

7.1.2 Edit network configuration file

As the Hortonworks documentation specifies, the desired network configuration for each host needs to be set, i.e. specifying the FQDN for each host.

7.1.3 Configure iptables

The setup of the Ambari Server requires certain ports to be open and available. Easiest option is to disable iptables temporarily:

See here in the Hortonworks documentation.

7.1.4 Repeat for all nodes

Now repeat these three steps (1. change root password, 2. edit the network configuration file, 3. configure iptables) for each node, i.e. the three worker nodes. Connect to each worker node given the SSH details on the Azure management portal.

Worker node 1:

Change root password for worker node 1:

Edit network configuration file in worker node 2:

Configure iptables in worker node 1:

Worker node 2:

Worker node 3:

7.2. Set Up Password-less SSH

Have a look at the Hortonworks documentation in here.

Start off with generating public and private SSH keys on the master node (i.e. Ambari Server host), by running

ssh-keygen –t rsa –P “”

Use default values.

Copy the SSH public key to the root account on all target hosts, i.e. worker nodes, by running

ssh-copy-id –i ~/.ssh/id_rsa.pub root@oldkHDPw1

Here, the root’s password is required, which is the reason for changing it beforehand. Check if you can connect to specified host by running ssh root@oldkHDPw1.

Once connected, set permissions on the .ssh directory and authorized_keys:

Repeat for all worker nodes and the master node itself:

Now, if you connect to the other nodes from the master node via SSH, you shall not be asked for a Password. Check it!

7.3. Format empty attached disks (for all nodes)

1. Master node
2. Worker nodes

Each node contained in our to-be-Hadoop-cluster as an empty data disk attached to it, but each is still offline. Hence, some initialisation is required, i.e. creating a new partition, a file system and a directory, for the data disk to be ready for use. Extensive information on how to attach a data disk to a Linux VM can be found here in the Azure documentation. We start with the master node:

7.3.1 Master node

List all partitions in the master node running fdisk –l. We obtain Information on the empty attached disk marked in blue:

Running the command grep SCSI /var/log/messages helps you find the identifier of the last data disk added (also marked in blue):

Now, we will create a new device on the empty attached disk, by first running fdisk /dev/sdc, and then typing in the following commands:

and accept default values:

Now, create the file system on the new partition, in this case an ext2/3 filesystem running mkfs.ext3 /dev/sdc1.

Following that, we need a directory to mount the new file system. Hence, run mkdir /mnt/datadrive to create the directory /mnt/datadrive. To mount the drive, run

mount /dev/sdc1 /mnt/datadrive

such that the data disk is ready to use as /mnt/datadrive.

Add the new drive to /etc/fstab:

/dev/sdc1 /mnt/datadrive ext3 defaults 1 2

7.3.2. Format empty attached disk: Worker Nodes

You repeat the same for all three worker nodes:

Now, the infrastructure is all set up and ready for the Hadoop installation!

Big thanks also go to Benjamin Guinebertière and Hans Laubisch for their support.

Contents

1 Introduction
2 Step-by-Step: Build the Infrastructure
3 Step-by-Step: Install a Hadoop Distribution

Introduction

While HDInsight is the Platform as a Service (PaaS) option for building and running a Hadoop cluster in Microsoft Azure, this article specifies its IaaS (Infrastructure as a Service) counterpart. With the IaaS option you have more flexibility in the choice of Hadoop distributions, Hadoop components and platform (e.g. Linux), amongst others.

This blog series elaborates the install of Hortonworks’ Hadoop distribution for Linux, HDP 2.1 for Linux. Alternatives for commercial Hadoop distributions on Linux include Cloudera (CDH) and MapR. Moreover, we will use CentOS as the Linux platform. In the end, we will have a four-node Hadoop cluster: one master node (also called NameNode) and three worker nodes (also called DataNode):

We heavily base our step-by-step guide on Benjamin’s great article How to install Hadoop on Windows Azure Linux virtual machines and Hortonworks’ documentation Hortonworks Data Platform – Automated Install with Ambari.

Before installing a Hadoop distribution though, the required environment needs to be prepared. Thus, the next article walks through the infrastructure setup for such a cluster on Microsoft Azure.

Contents

1 What is Mahout?
2 Step-by-Step: Mahout with HDInsight Interactive Style
3 Step-by-Step: Mahout with HDInsight PowerShell Style

Step-by-Step: Mahout with HDInsight PowerShell Style

In this episode of the series Mahout for Dummies, we deal with Mahout on HDInsight in a PowerShell manner. Ultimately, we go through the Random Forest Scenario detailed in previous post.

1. Upload Data
2. Create HDInsight Cluster
3. Mahout: general PowerShell command
4. Scenario: Random Forest
   1. Build forest
   2. Classify test data
5. Clean up
6. Scenario: Recommender Job
7. Wrapping up…

1. Upload Data

Here, we upload all data to the Azure blob storage necessary to build a random forest model from and then to test the model on. More specifically, training and test data will be uploaded. Note that information on the storage account (e.g. container name and storage context) must already be known.

Since Mahout is not installed on any HDInsight cluster by default (and hence not supported by Microsoft), the Mahout jar files also shall have to be uploaded to the blob storage.

2. Create HDInsight Cluster

We just create a simple HDInsight cluster, just like in the Azure PowerShell Series: Simple HDInsight. Alternatively, you could create one with additional functionality; see Azure PowerShell Series: Custom Create HDInsight.

In the Azure Explorer, you observe some libraries being uploaded, such as mapred, hive, etc.

Just like in the previous post Step-by-Step: Mahout with HDInsight Interactive Style, both the training and test data need to be located in the directory user/hdp/

3. Mahout: General PowerShell Command

The typical command for invoking Mahout from the Hadoop Command Line via RDP connection looks as follows:

Thus, it is an ordinary command running the program contained in specified JAR file. org.apache.mahout.classifier.df.tool.Describe is the class name being invoked, followed by mandatory and optional arguments. Translated into PowerShell:

In the case above, this translates into the following PowerShell command:

or a little more elaborate:

Note that the PowerShell commandlets have so far only defined the job but not triggered it yet. The Hadoop Job is started by the following command:

To automatically wait for the HDInsight job to process, you can insert the following

It gives an hour (i.e. 3600 seconds) for the HDInsight job to process . You can print out any output error as follows:

4. Scenario: Random Forest

In the previous section, we elaborated on how to construct a Mahout Job as a PowerShell command. Here, we go through an example using the Random Forest, just like in the previous post Step-by-Step: Mahout with HDInsight Interactive Style – Scenario Random Forest.

4.1. Build Forest

As a reminder, the command we used to build a forest in Interactive Style is the following:

Thus, the “translated” PowerShell command is

The output in PowerShell should look like this:

4.2. Classify Test Data

The “converted” PowerShell command of the classifying command proposed in Interactive Style is as follows:

Note that the Output shown above has the same format and very similar results to the previous post when done in interactive style.

5. Clean Up

Cleaning up involves the likes of removing and HDInsight cluster but also removing temporary directories. While the PowerShell command for deleting a single file is pretty straight forward, i.e.

deleting a folder structure comprises a loop in which every single file with specified file path prefix is removed.

6. Scenario: Recommender Job

As we saw in the first part of our Mahout for Dummies series, there are many algorithms included in the Mahout library other than the random forest.

In the blog by the Big Data Support team at Microsoft, there is a good post demonstrating the use of the RecommenderJob class on an HDInsight Cluster using PowerShell that you can read here. The source code of the RecommenderJob class can be looked up here on GitHub.

In this scenario, we are given two data files: one containing user ID’s and the other comprising the degrees of preference of users towards given items:

In ItemID.txt, the first column indicates the user ID, the second the item IDs and the final one denotes the degree of preference. Thus, ItemID.txt can be expressed in a more intuitive Format of a matrix, where the rows indicate the user and the columns denote the item IDs. The values inside the matrix themselves display the degree of preference, as given in the third column in ItemID.txt.

Here is the comprised PowerShell script for running RecommenderJob as in the Big Data Support blog.

The output files can be seen in the Azure Blob Storage Explorer as ususal:

The output file itself gives information on how likely which items could be of interest to which users, i.e. <user_id> [<item_id>: <degree-of-preference/interest>,…].

In such a way, we can insert the recommendations with their scores in the matrix from above:

7. Wrapping Up…

In this blog post, we went through two scenarios applying Mahout on HDInsight in PowerShell style: random forest and recommender. These scenarios are nicely wrapped around the usual suspects: uploading data, creating the HDInsight cluster and cleaning up afterwards.

Many thanks go to Alexei Khalyako and Bill Carroll for their support on Mahouting on HDInsight!

1 What is Mahout?
2 Step-by-Step: Mahout with HDInsight Interactive Style
3 Step-by-Step: Mahout with HDInsight PowerShell Style

What is Mahout?

Apache Mahout, 15.04.2014

Mahout is one of many Hadoop-related projects at Apache. Its mission is to build a scalable machine learning and data mining library. In other words, Mahout provides data science tools useful for detecting meaningful patterns in given data sets that are stored in HDFS (Hadoop Distributed File System). It is implemented on top of Hadoop and as of version 0.9 based on the infamous MapReduce paradigm.

Why the word Mahout? Traditionally, a mahout is an elephant rider and has its origins in the Hindi language. The mahout starts early on as a boy when being assigned an elephant.

Well, back to the machine learning library – that would contain numerous algorithms! Mahout is based on three “C-pillars” of machine learning implementations:

• Collaborative filtering (aka recommendation),
• Clustering, and
• Classification.

Collaborative Filtering (aka Recommendation)

Customers Who Bought This Item Also Bought, PaulsHealthBlog.com, 11.04.2014

You were looking at a product in Amazon, and there it is – a list of items recommended to you based on what other users also considered buying when looking at “your” product. Such recommender engines (also to be found in Netflix, Spotify, etc.) comprises all kinds of collaborative filtering algorithms. User behaviour is being mined to observe patterns and use as a recommendation for other users with similar likes and dislikes.

Clustering

Microsoft Clustering Algorithm, 11.04.2014

This family of machine learning entails the grouping of data units into natural clusters since they share similar characteristics. For instance, you tend to cluster customers into groups according to demographic information, say, without labelling these groups yet. Or we naturally group most food into sweet or salty things.

Classification

Given a dataset that we can learn from and build a data model, we then can classify new unknown data items. For instance, the eye colour is genetically influenced by more than one gene. By learning which genes would result in blue eyes, we can predict the eye colour of other people based on their genetic information.

What is the difference between clustering and classification? While in classification you are already given certain categories to classify your data, clustering involves naturally similar items. In other words, in the example of the blue eyes, we know from the beginning what we are looking for: blue eyes or no blue eyes, whereas labelling the groups shall still be established after clustering.

How does Mahout work?

Mahout provides the implementations of various ML algorithms – a list of them can be found on their site in the list of algorithms. Each one of them can be invoked via a command line. How it is done with HDInsight and PowerShell will be shown in the upcoming blog entries: Step-by-Step: Mahout with HDInsight Interactive Style and Step-by-Step: HDInsight with Mahout PowerShell Style.

Contents

1 What is Mahout?
2 Step-by-Step: Mahout with HDInsight Interactive Style
3 Step-by-Step: Mahout with HDInsight PowerShell Style

Step-by-Step: Mahout with HDInsight Interactive Style

But before heading right into Mahout, the HDInsight cluster shall be created. Please note that as of now, Mahout is NOT supported by Microsoft. Mahout is by default not installed on any HDInsight Cluster, but can be done in various ways, such as connecting to the head node via RDP or using PowerShell (see next article).

Update: Mahout is contained in the HDInsight Version 3.1 by default; more information can be found in the documentation What’s new in the Hadoop cluster Versions provided by HDInsight? Thus the second step Install Mahout on HDInsight can be skipped.

This article is a step-by-step guide on how to install and use Mahout on an HDInsight Cluster with a specific scenario involving Random Forests.

1. Create HDInsight Cluster
2. Install Mahout on HDInsight
3. Scenario: Random Forest
   1. Get data
   2. Generate descriptor file
   3. Build forest
   4. Classify test data
   5. What is Happening?
4. Wrapping up…

1. Create HDInsight Cluster

Prerequisite: You have already created a storage account.

Start off with going to the Microsoft Azure portal and click on new.

Pick a Name for your HDInsight cluster. Please note that we are using HDInsight Version 2.1. Set the datacenter region to the same region that your storage account is located in, in this case North Europe.

Obviously you configure credentials to your HDInsight Cluster:

As mentioned above, the prerequisite is that you have already created a storage account. To have a clean slate, I create a default container on which the HDInsight Cluster is based.

The process of creating an HDInsight Cluster includes a few Milestones such as configuring the provisioned virtual machines. The beauty of HDInsight is that you do not need to provision so-and-so-many virtual machines and install Hadoop on them – HDInsight provides this as a Service and does it automatically.

Once created, let us enable a remote desktop connection:

And let’s connect!

2. Install Mahout

Mahout is provided by HDInsight Version 3.1 by default. As you have connected remotely to it, open the file explorer and browse to C:\apps\dist. There you can see a list of Hadoop components supported by HDInsight 3.1:

Hence, you can ignore the rest of this paragraph and skip right to 3. Scenario: Random Forest.

In case you do deal with an earlier HDInsight Version (earlier than 3.1), then follow the steps described in this paragraph.

You can find the latest release version of Mahout on http://mahout.apache.org/ that you can download locally on your computer.

In the head node of your HDInsight Cluster (you have connected to it via RDPin the end of 1. Create HDInsight Cluster), open the File Explorer to create a new folder C:\ , such as C:\MyFiles.

Copy the Mahout Distribution zip file into C:\MyFiles in the head node. In this case, the latest release is version 0.9.

Extract the zip-file into C:\apps\dist.

Rename the extracted folder into mahout-x.x where x.x is the version number.

And that’s it – Mahout is installed on your HDInsight cluster!

3. Scenario: Random Forest

This step-by-step guide is based on the one documented in Mahout – Classifying with random forests only tailored to HDInsight 2.1.

1. Get Data

Before building a forest model, we need data to learn from as well as data to test our model from. The data sets used here can be downloaded from http://nsl.cs.unb.ca/NSL-KDD/. I am using KDDTrain+ARFF and KDDTest+.ARFF.

Check in the downloaded files that unnecessary lines are removed. More specifically, in KDDTrain+.arff remove the first 44 lines (i.e. all lines starting with @attribute). Otherwise, we will not be able to generate a descriptor file later on in 3.2.

Copy these two data files into C:\MyFiles in the head node (just like with the Mahout Distribution earlier).

Ok, we have our training and test data in our HDInsight cluster, but for Mahout do its magic on the precious data, the data needs to be in the HDFS (Hadoop Distributed File System). Yes, you are right – in HDInsight, we do not use HDFS; all data is stored in the Azure Blob Storage. Instead, the HDFS API is still used in HDInsight. So, all we need to do is copy local data into the Blob Storage.

There are many ways transferring local data into the blob storage. Here, we use the Hadoop shell commands. First we create a directory called testdata. Then we copy.

hdfs dfs -mkdir testdata
hdfs dfs -copyFromLocal C:/MyFiles/KDDTrain+.arff testdata/KDDTrain+.arff
hdfs dfs -copyFromLocal C:/MyFiles/KDDTest+.arff testdata/KDDTest+.arff

Many use the shell command

hadoop fs

but this command is deprecated.

Here’s a tip to avoid typing in the whole path: Copy path in the file explorer.

To see what is stored in all my storage accounts, I often use cerebrata’s Azure Explorer, but there are many other storage explorers recommended by us all listed here.

To double check, we see that the now copied data files are located in user/testdata/. Note that olivia is the user I configured to be the remote user, hence can remotely connect to the head node of my HDInsight cluster.

The way Mahout is compiled, the data needs to be in user/hdp/ though. In the Hadoop command line we’ll type in:

hdfs dfs -cp wasb://oliviakrf@oliviakstor.blob.core.windows.net/user/olivia/testdata/KDDTrain+.arff 
    wasb://oliviakrf@oliviakstor.blob.core.windows.net/user/hdp/testdata/KDDTrain+.arff
hdfs dfs -cp wasb://oliviakrf@oliviakstor.blob.core.windows.net/user/olivia/testdata/KDDTest+.arff 
    wasb://oliviakrf@oliviakstor.blob.core.windows.net/user/hdp/testdata/KDDTest+.arff

More generally, just replace the variables in <> with the names you have chosen accordingly (i.e. container, storage account and remote user):

hdfs dfs -cp wasb://<container>@<storage-account>.blob.core.windows.net/user/<remote-user>/testdata/KDDTrain+.arff 
    wasb://<container>@<storage-account>.blob.core.windows.net/user/hdp/testdata/KDDTrain+.arff
hdfs dfs -cp wasb://<container>@<storage-account>.blob.core.windows.net/user/<remote-user>/testdata/KDDTest+.arff 
    wasb://<container>@<storage-account>.blob.core.windows.net/user/hdp/testdata/KDDTest+.arff

Checking in Azure Explorer, the two data files can be found under user/hdp/testdata as desired.

2. Generate descriptor file

Before building a random forest model based on the training data in KDDTrain+.arff, a descriptor file is essential. Why? When building the model, all information in the training data needs to be labelled for the algorithm to know, which one is numerical, categorical or a label.

The command is as follows:

hadoop jar C:\apps\dist\mahout-0.9\mahout-core-0.9-job.jar 
org.apache.mahout.classifier.df.tools.Describe 
-p wasb:///user/hdp/testdata/KDDTrain+.arff 
-f testdata/KDDTrain+.info 
-d N 3 C 2 N C 4 N C 8 N 2 C 19 N L

Here, the main class of org.apache.mahout.classifier.df.tools.Describe is invoked; for more information on the source code, check out Mahout’s GitHub site on Describe.java. It takes three mandatory arguments, namely: p (short for path), f (short for file) and d (short for descriptor). Other optional arguments are h (help), r (regression) and Options. The p argument specifies the path where the data to be described is located, f defines the location for generated descriptor file and d provides Information on all attributes of given data, where N=numerical, C=categorical and L=label. More specifically, “N 3 C 2 N C 4 N C 8 N 2 C 19 N L” means that given data set starts off with a numerical attribute (N), followed by 3 categorical attributes (C), etc. and lastly with a Label (L).

Update: Note that if you use an HDInsight cluster based on version 3.1, change the path specifying the mahout jar file to
C:\apps\dist\mahout-0.9.0.2.1.5.0-2057\core\target\mahout-core-0.9.0.2.1.5.0-2057-job.jar

Since the descriptor file also needs to be in the directory user/hdp/, you either then copy the generated descriptor file into user/hdp/ or you might as well set parametre f to wasb:///user/hdp/testdata/KDDTrain+.info

hdsf dfs 
-cp wasb://<container>@<storageaccount>.blob.core.windows.net/user/<remoteuser>/testdata/KDDTrain+.info
wasb://<container>@<storageaccount>.blob.core.windows.net/user/hdp/testdata/KDDTrain+.info

Or generating the descriptor file in user/hdp/ straight away:

hadoop jar C:\apps\dist\mahout-0.9\mahout-core-0.9-job.jar 
org.apache.mahout.classifier.df.tools.Describe 
-p wasb:///user/hdp/testdata/KDDTrain+.arff 
-f wasb:///user/hdp/testdata/KDDTrain+.info 
-d N 3 C 2 N C 4 N C 8 N 2 C 19 N L

Update: If you use an HDInsight cluster of version 3.1, use the following path specifying the Mahout jar:

C:\apps\dist\mahout-0.9.0.2.1.5.0-2057\core\target\mahout-core-0.9.0.2.1.5.0-2057-job.jar

Checking in the Azure Explorer, we see KDDTrain+.info in the Directory user/hdp/testdata:

3. Build forest

Now we can finally build the random forest using the following command in the Hadoop command line:

hadoop jar C:\apps\dist\mahout-0.9\mahout-examples-0.9-job.jar 
org.apache.mahout.classifier.df.mapreduce.BuildForest 
-Dmapred.max.split.size=1874231 
-d wasb:///user/hdp/testdata/KDDTrain+.arff 
-ds wasb:///user/hdp/testdata/KDDTrain+.info 
-sl 5 -p -t 100 -o nsl-forest

In GitHub, you can look into the source code of used class (BuildForest) on GitHub. Thus, the mandatory arguments of the main class in BuildForest are d (data), ds (dataset), t (nbtrees), o (output). A more comprehensive list of arguments is the following:

In other words, a random forest model is computed on the basis of data provided in KDDTrain+.arff with additional description information in KDDTrain+.info with 100 trees saved in the directory nsl-forest/. The computation of a random forest uses the partial implementation (-p) and splits the dataset at each tree node by randomly selecting 5 attributes (-sl), whilst allowing a maximum of 1,874,231 data units per node (-Dmapred.max.split.size). Note that the maximum number of data units per node also indicates the partition size of each tree in the random forest, in this case 1/10 of the dataset; thus, 10 partitions are being used.

Update: As mentioned in 3.2 Generate Descriptor File, if you use an HDInsight cluster of version 3.1, use the following path for the mahout example jar:

C:\apps\dist\mahout-0.9.0.2.1.5.0-2057\examples\target\mahout-examples-0.9.0.2.1.5.0-2057-job.jar

The result in the Hadoop command line will look like this:

In the end, we can see how long it took to build the forest and also obtain further Information on the forest, such as the number of nodes or mean maximum depth of the forest.

To use the generated forest for classifying unknown test data, we copy it via hdfs commands into user/hdp/ as follows:

hdfs dfs 
-cp wasb://oliviakrf@oliviakstor.blob.core.windows.net/user/olivia/nsl-forest 
wasb://oliviakrf@oliviakstor.blob.core.windows.net/user/hdp/nsl-forest

or more generally speaking:

hdfs dfs 
-cp wasb://<container>@<storageaccount>.blob.core.windows.net/user/<remoteuser>/nsl-forest 
wasb://<container>@<storageaccount>.blob.core.windows.net/user/hdp/nsl-forest

In Azure Explorer you can see that the generated forest model (forest.seq) is stored in both user/olivia/ and user/hdp/.

4. Classify new data

We have generated a forest model in the step before in order to automatically classify new incoming data, i.e. KDDTest+.arff. The command we use here is

hadoop jar C:\apps\dist\mahout-0.9\mahout-examples-0.9-job.jar 
org.apache.mahout.classifier.df.mapreduce.TestForest 
-i wasb:///user/hdp/testdata/KDDTest+.arff 
-ds wasb:///user/hdp/testdata/KDDTrain+.info 
-m wasb:///user/hdp/nsl-forest 
-a -mr -o predictions

Update: If you use a cluster of HDInsight 3.1, use the following path name for the mahout examples jar file:

C:\apps\dist\mahout-0.9.0.2.1.5.0-2057\examples\target\mahout-examples-0.9.0.2.1.5.0-2057-job.jar

As usual more information can be found on Mahout’s GitHub site, more concretely in TestForest.java. What do the arguments mean? The mandatory arguments are input (-i) for the test data location, dataset (-ds) for the descriptor file location, model (-m) for the forest model location and output (-o) for the output location; optional boolean arguments are analyze (-a) for analysing the classification results, i.e. computing the confusion matrix, and mapreduce (-mr) to use Hadoop to distribute classification.

In this case, predictions are computed for the new test data located in wasb:///user/hdp/testdata/KDDTest+.arff with its associated descriptor file in wasb:///user/hdp/testdata/KDDTrain+.info using the previously built random forest in wasb:///user/hdp/nsl-forest; the output predictions are then stored in a text file in the directory predictions/. Additionally, a confusion matrix is computed (as you can see below in the Hadoop command line) and classification is being distributed using Hadoop.

The predictions are stored in user/olivia/predictions:

5. Woah, what’s happening?

Ok, so what just happened? Let’s first have a look at the summary in the Hadoop command line and another closer look at the output file containing the predictions.

The test data in KDDTest+.arff contained 22,544 instances that were classified in 1.4. Classify new data, and that you can also see in the first section Summary under Total Classified Instances. Thus you can see that 17,783 instances of them (i.e. 78%) were correctly classified, whereas 4,761 (21%) were incorrectly classified.

More details are provided in the confusion matrix, in nicer view:

In other words, 9,458 normal instances were correctly classified but 253 normal instances were incorrectly classified as anomaly, adding up to 9,711 actual normal instances. There are 17,783 correctly classified instances (= 9,458 + 8,325, i.e. normal-normal + anomaly-anomaly) compared to 4,761 (= 4,508 + 253) incorrectly classified instances.

The remaining statistics measures (Kappa and reliability) indicate a degree on overall consistency of measure, more specifically of agreement between raters.

And finally, what about the predictions that have been saved as an output of classifying the test data?

After some converting, you obtain a list of numbers of type double. What each double number indicates is the predicted category of each data instance, where 1.0 denotes the category anomaly.

4. Wrapping up…

We have created an HDInsight cluster such that the Mahout library (in this case version 0.9) could subsequently be installed. The Mahout library is very extensive and can be explored at its full glory in its GitHub site. Here, we went through a scenario using one of many Machine Learning religions, namely the Random Forest, based on the random forest tutorial on the Mahout site but tailored to HDInsight.

Update: There is an extensive guide on how to use Mahout on HDInsight to generate movie recommendations, found here on the Azure documentation – highly recommendable!

In the next Mahout article, we will explore the use of Mahout through the awesomeness of PowerShell.

In anderen Worten, zunächst wird ein SQL Server auf Azure mit einem automatisch generiertem Namen erstellt (ebenso auf PowerShell), und darauf basiert eine Azure SQL Datenbank. Um eine SQL Datenbank basiert auf dem Azure SQL Server per PowerShell zu erstellen, richten wir eine Firewall Regel ein, um alle Verbindungen (also alle IP Adressen) zum Server erlaubt werden:

Und hier nun zum PowerShell-Skript: