Les billets suivants de ce blog sont sur https://blog.3-4.fr.

Je tweete chaque nouveau billet.

Hi,

Next blog posts are at https://blog.3-4.fr.

I tweet for any new blog post.

:-)

Benjamin (@benjguin)

Comme on le voit dans ce diagramme, HDInsight est au cœur de la plateforme Big Data de Microsoft.

L’offre Azure Data Lake Analytics et l’offre HDInsight ont un positionnement similaire. Suivant que vous préférez utiliser un code portable tout en bénéficiant d’un service géré, d’une part ou que vous voulez être dans un monde plus centré sur Microsoft d’autre part, vous choisirez plutôt HDInsight ou plutôt Azure Data Lake Analytics.

Dans le cas d’HDInsight, vous pouvez exécuter du code Hive, Pig, Java, Python, Scala, … sur Hadoop, Storm ou Spark… Cela n’est pas très coûteux puisque vous pouvez créer les ressources de calcul nécessaires uniquement pendant le temps d’exécution des batches, et ce de façon automatisée. Si on prend par exemple 2 heures de batch pendant 30 j par mois, cela donne un coût de moins de 500 € pour un cluster avec 2 nœuds principaux et 10 nœuds de calcul :

NB : cela inclut du support qui permet de poser des questions sur votre code. En effet, supposons que vous ayez un problème avec une requête HIVE par exemple; si c’est sur HDInsight, vous pouvez demander au support Microsoft.

Cela dit, si vous devez développer sur cet environnement HDInsight, c’est moins optimal puisque l’environnement va tourner plus régulièrement (ex : 8 à 10 h / jour). De plus, on peut créer ou détruire un cluster HDInsight qui va retrouver ses données, mais il peut être plus confortable de travailler sur une machine virtuelle que l’on personnalise un peu avec ses outils, qu’on éteint ou allume en fonction des besoins. Il n’est pas nécessaire d’avoir un cluster complet en développement tant qu’on n’en n’est pas à la phase d’optimisation.

On peut aussi vouloir tester des fonctionnalités nouvelles d’Hadoop.

Pour tout cela, il y a le bac à sable (Sandbox) d’Hortonworks, disponible dans la marketplace d’Azure.

Ce modèle de machine virtuelle comprend des tutoriels, exemples, des outils pour découvrir Hadoop, Spark et aussi l’utiliser comme VM de développement.

Au niveau du prix, cela coûte le prix de la VM. Le modèle de licence est du Bring Your Own License. Voir https://azure.microsoft.com/en-us/marketplace/partners/hortonworks/hortonworks-sandbox/ pour plus d’informations.

Cette page contient un lien pour instancier une VM.

Il faut ensuite remplir un certain nombre de champs, comme pour la création d’une machine virtuelle standard. Dans les quelques copies d’écrans suivantes, on se place dans le cas de la création dans le mode « Resource Manager ».

Ce mode permet ensuite, lorsque la VM est créée, de configurer des règles de pare-feu depuis le portail (dans le mode « classic », il peut être nécessaire de passer par quelques lignes de commande).

Voici donc comment configurer les accès à votre VM :

Depuis le panneau (« blade ») de la VM dans le portail portal.azure.com, cliquez sur le groupe de ressources qui vous donnera accès au NSG (Network Security Group) du vNet. Là, vous avez accès aux différentes règles entrantes vers le réseau…

…

… auxquelles vous pouvez ajouter deux règles :

Une pour autoriser votre ou vos adresses IP à accéder à la VM et l’autre pour interdire toutes les autres adresses IP d’Internet. Remarquez le champ « Priority ».

Exemple :

et

Optionnellement, vous pouvez également donner un nom à l’adresse IP publique de votre VM, comme suit :

Par exemple, si vous avez déployé en North Europe et que vous avez donné comme nom hdp23sandbox, vous pouvez ensuite naviguer vers https://hdp23dansbox.northeurope.cloudapp.azure.com:8888 et voir une page d’accueil qui contient entre autres des tutoriels :

Il est bien sûr aussi possible de se connecter à votre VM en ssh.

Vous pouvez configurer l’accès aux comptes de stockage Azure en ajoutant des clefs valeurs dans le fichier /etc/hadoop/conf/hdfs-site.xml (sudo vi /etc/hadoop/conf/hdfs-site.xml)

- key: fs.azure.account.key.{storage account name}.blob.core.windows.net

- value: {{storage account name} account key (primary or secondary)}

Exemple:

Ce qui permet ensuite de travailler sur les fichiers du compte de stockage comme on travaillerait avec HDFS.

Pour rappel, la syntaxe est :

wasb://{container}@{storage account name}.blob.core.windows.net/[{dossier1/dossier2(…)}[/{file name}]

exemple :

qui correspond à

Vous pouvez instancier votre propre « sandbox » depuis cette page : https://azure.microsoft.com/en-us/marketplace/partners/hortonworks/hortonworks-sandbox/. Cliquez sur le bouton « Create Virtual Machine > »

@benjguin

Je publie ici les vidéos de secours que j’avais préparées. Ce sont des versions un peu plus longues que ce que j’ai pu présenter.

Vous trouverez la vidéo de la session complète sur le sujet à https://channel9.msdn.com/Blogs/Microsoft-Azure-France/Azure-Camp-9-DEC-2014-Lambda-Architecture

Mais revenons aux démos qui tournent autour de l’architecture suivante:

La première vidéo montre une vue d’ensemble de l’ingestion, du traitement en Storm et de la dataviz sans entrer dans le détail de la façon dont c’est fait.

Ensuite, on passe à Storm dans HDInsight:

puis à la DataViz en HTML5 et Excel avec d3js et SignalR:

Dans la démo Hortonworks, je montre comment on peut déployer la même topologie Storm sur un cluster Hadoop HDP fait de VMs linux installées initialement avec l’assistant de la markerplace Azure. On voit aussi comment on peut traiter les logs stockés dans le blob storage.

Ensuite, on montre qu’on peut aussi créer un cluster Cloudera, également via l’assistant de la marketplace Azure:

Arrivent ensuite les deux bases noSQL de la démo.

HBase:

et Datastax Enterprise / Cassandra en multi datacenter:

:-)

Benjamin (@benjguin)

Let’s test with a word cloud example in R. Mollie Taylor has posted one here (https://gist.github.com/mollietaylor/3671518) that we can reuse in Azure Machine Learning:

The details on how to create an Azure ML workspace, insert a dataset and an R script can be found here:

• https://azure.microsoft.com/en-us/documentation/articles/machine-learning-walkthrough-develop-predictive-solution/

for R, just use that module:

The input of the Web API is set to the input dataset of the R Script and the output is set to the R Device port. As a reminder, here is how the inputs and outputs are positioned in an R Script module:

the detail is available in the help documentation.

In our case the interesting ports to publish are the following:

and

After running the experiment, we can see the result in Azure ML Studio:

So, how could we retrieve the pictures from an API that is published that way:

Here is some sample script in Python that shows how to do it. The script is a modified version of the sample given in the API Help page for Batch Execution. The idea is to get the base64 encoded pictures from the output file and decode them out to local disk.

# -*- coding: utf-8 -*-

# How this works:
#
# 1. Assume the input is present in a local file
# 2. Upload the file to an Azure blob - you'd need an Azure storage account
# 3. Call BES to process the data in the blob. 
# 4. The results get written to another Azure blob.
# 5. Download the output blob to a local file
#
# Note: You may need to download/install the Azure SDK for Python.
# See: https://azure.microsoft.com/en-us/documentation/articles/python-how-to-install/

import urllib2
import json
import time
from azure.storage import *
import sys
import base64
import json

storage_account_name = 'a****obfuscated***4'
storage_account_key = '/aV****obfuscated***vXA76w=='
storage_container_name = 'benjguin'

input_file = ur"C:\be****obfuscated***os\WordCloud\conventions.csv"
output_file = ur'C:\be****obfuscated***os\WordCloud\myresults.csv'
input_blob_name = 'conventions.csv'
api_key = r'Cczx****obfuscated***WemQ=='
url = 'https://ussouthcentral.services.azureml.net/workspaces/a7c****obfuscated***756/services/d328e03****obfuscated***5c2/jobs'
uploadfile=True
executeBES=True

blob_service = BlobService(account_name=storage_account_name, account_key=storage_account_key)

if uploadfile:
    print("Uploading the input to blob storage...")
    data_to_upload = open(input_file, 'r').read()
    blob_service.put_blob(storage_container_name, input_blob_name, data_to_upload, x_ms_blob_type='BlockBlob')

input_blob_path = '/' + storage_container_name + '/' + input_blob_name
debug_blob = blob_service.get_blob(storage_container_name, input_blob_name)

if executeBES:
    print("Submitting the BES job...")
    connection_string = "DefaultEndpointsProtocol=https;AccountName=" + storage_account_name + ";AccountKey=" + storage_account_key
    payload =  {
                "Input": {
                    "ConnectionString": connection_string,
                    "RelativeLocation": input_blob_path
                    }
                }

    body = str.encode(json.dumps(payload))
    headers = { 'Content-Type':'application/json', 'Authorization':('Bearer ' + api_key)}
    req = urllib2.Request(url, body, headers) 
    response = urllib2.urlopen(req)
    result = response.read()
    job_id = result[1:-1] # remove the enclosing double-quotes

    url2 = url + '/' + job_id

    while True:
        time.sleep(1) # wait a second
        authHeader = { 'Authorization':('Bearer ' + api_key)}
        request = urllib2.Request(url2, headers=authHeader)
        response = urllib2.urlopen(request)
        result = json.loads(response.read())
        status = result['StatusCode']
        if (status == 0):
            print("Not started...")
        elif (status == 1):
            print("Running...")
        elif (status == 2):
            print("Failed...")
            break
        elif (status == 3):
            print("Cancelled...")
            break
        elif (status == 4):
            print("Finished!")
            result_blob_location = result['Result']
            sas_token = result_blob_location['SasBlobToken']
            base_url = result_blob_location['BaseLocation']
            relative_url = result_blob_location['RelativeLocation']
            url3 = base_url + relative_url + sas_token
            response = urllib2.urlopen(url3)
            with open(output_file, 'w') as f:
                f.write(response.read())
            break

outputdata=open(output_file)
outputtxt=outputdata.read()
outputdata.close()

s=outputtxt.index('\"{')
e=len(outputtxt)
o1=outputtxt[s+1:e-3]

jsonresult = json.loads(o1)
i=1
for gd in jsonresult['Graphics Device']:
    fname = output_file + "." + str(i) + ".png"
    print 'writing png #' + str(i) + ' to ' + fname
    f = open(fname, 'wb')
    f.write(base64.b64decode(gd))
    f.close()
    i += 1

print("Done!")

Here is a sample execution output:

Uploading the input to blob storage...
Submitting the BES job...
Running...
Running...
Running...
Running...
Running...
Running...
Running...
Finished!
writing png #1 to C:\be***obfuscated***os\WordCloud\myresults.csv.1.png
writing png #2 to C:\be***obfuscated***os\WordCloud\myresults.csv.2.png
Done!

The output sent back by Azure ML looks like this:

R Output JSON

"{"Standard Output":"RWorker pushed \"port1\" to R workspace.\r\nBeginning R Execute Script\n\n[1] 56000\r\nLoading objects:\r\n  port1\r\n[1] \"Loading variable port1...\"\r\npng \r\n  2 \r\nnull device \r\n          1 \r\n","Standard Error":"R reported no errors.","visualizationType":"rOutput","Graphics Device":["iVBORw0K***(...)***RvX/wFzB5s8eym6ZgAAAABJRU5ErkJggg==","iVBORw0KGgo***(...)***dVorBuiQAAAABJRU5ErkJggg=="]}"

You can see the pictures

well, Python does:

The resulting files are:

and

R has tons of great data visualisation. Have a look at those blogs for instance:

• https://www.r-statistics.com/tag/visualization/
• https://rdataviz.wordpress.com/

Benjamin (@benjguin)

R script is:

data.set <-data.frame(installed.packages());
# Select data.frame to be sent to the output Dataset port
maml.mapOutputPort("data.set");

you’ll find a little more than 400 packages.

Still you may need to use a package which is not known by Azure ML. Here is how to upload it to the environment.

NB: This post takes skmeans (k-means with a cosine distance) as an example, but this works for other packages as well.

Let’s suppose you have this code in R Studio locally.

NB: you can find information on how to setup your environment in this post. It’s in French, but bing translator is your friend.

library(skmeans)

set.seed(1234)
sample_data <- matrix(sample.int(1000, size = 20*500, replace = TRUE), nrow = 20, ncol = 500, 
                      dimnames=list(1:20, 1:500))

fit <- skmeans(sample_data,5)

result <- data.frame(list(rownames(sample_data), fit$cluster), row.names=NULL)
colnames(result) <- c("sample data row", "cluster")

print(result)

this will give this kind of result

If you try this in Azure ML, you’ll get the following result:

Here is how to have the script loading all the necessary packages in the Azure ML environment.

So let’s now see how you construct the skmeans_packages.zip and know which lines to write here:

On the local environment (in my case Windows), I remove the R packages that are installed in My Documents\R

then in R, I install the skmeans package:

install.packages("skmeans")

this gives the following result:

So I know I have to install the following packages in order:

• slam
• clue
• skmeans

Then I go to the temp folder:

I Zip the zips:

and rename this new zip file as skmeans_packages.zip

I then can upload it Azure ML:

NEW, DATASET, FROM LOCAL FILE

Then you’ll be able to find it as a saved dataset in your workspace:

After it has been connected to the third dot of the Execute R Script module instance, you’ll be able to find the content in src/ folder:

so, in order  to install skmeans and its two dependencies, then reference the skmeans library, you just have to enter the following lines:

install.packages("src/slam_0.1-32.zip", lib = ".", repos = NULL, verbose = TRUE)
install.packages("src/clue_0.3-48.zip", lib = ".", repos = NULL, verbose = TRUE)
install.packages("src/skmeans_0.2-6.zip", lib = ".", repos = NULL, verbose = TRUE)
library(skmeans, lib.loc=".", verbose=TRUE)

Azure ML has a pool of VM with docker-like containers (true Windows containers, named drawbridge) where the experiments run. So each time the script runs, it starts from a blank standard Azure ML environment. By bringing a zip, you add the files to that environment.

Hope this blog post will help you if you need R packages which are not in the 400+ preloaded ones in Azure Machine Learning!

Benjamin (@benjguin)

Here is a sample Azure Automation code that creates an HDInsight cluster, runs a job, and removes the cluster.

I used it for a demonstration at the French Hadoop User Group. You can find the video (in French) where I explain what is does here (go to 01:40:00) :

When you start it, you can specify the options you want to activate:

The code is the following:

workflow HDInsightJob
{
    param(
        [Parameter(Mandatory=$false)]
        [bool]$createCluster=$false,
        [Parameter(Mandatory=$false)]
        [bool]$runJob=$true,
        [Parameter(Mandatory=$false)]
        [bool]$removeCluster=$false
        )
    
    $subscription = 'Azdem169A44055X'
    $azureConnection=Get-AutomationConnection -Name $subscription
    $subscriptionId=$azureConnection.SubscriptionId
    $azureCertificate=Get-AutomationCertificate -Name 'azure-admin.3-4.fr'
    $clusterCredentials=Get-AutomationPSCredential -Name 'HDInsightClusterCredentials'
    
    InlineScript
    {
        echo "RunBook V140629c"
        echo $PSVersionTable
        import-module azure
        get-module azure
        
        $createCluster=$using:createCluster
        $runJob=$using:runJob
        $removeCluster=$using:removeCluster

        $subscription=$using:subscription  
        $subscriptionId=$using:subscriptionId  
        $azureCertificate=$using:azureCertificate
        $clusterCredentials=$using:clusterCredentials

        $defaultStorageAccount='monstockageazure'
        $clusterName = 'monclusterhadoop'
        $clusterContainerName = 'monclusterhadoop'
        $clusterVersion='3.0' #3.1, 3.0, 2.1, 1.6

        Set-AzureSubscription -SubscriptionName $subscription -SubscriptionId $subscriptionID -Certificate $azureCertificate -CurrentStorageAccount $defaultStorageAccount
        Select-AzureSubscription -Current $Subscription

        $storageAccount1 = (Get-AzureSubscription $Subscription).CurrentStorageAccountName
        $key1 = Get-AzureStorageKey -StorageAccountName $storageAccount1 | %{ $_.Primary }

        $storageAccount2Name = "wasbshared"
        $storageAccount2Key = (Get-AzureStorageKey -StorageAccountName $storageAccount2Name).Secondary

        if ($createCluster)
        {
            echo "will create cluster $clusterName"
            
            New-AzureHDInsightClusterConfig -ClusterSizeInNodes 3 |
                Set-AzureHDInsightDefaultStorage -StorageAccountName "${storageAccount1}.blob.core.windows.net" -StorageAccountKey $key1 -StorageContainerName $clusterContainerName |
                Add-AzureHDInsightStorage -StorageAccountName $storageAccount2Name -StorageAccountKey $storageAccount2Key |
                New-AzureHDInsightCluster -Name $clusterName -Version $clusterVersion -Location "North Europe" -Credential $clusterCredentials
            
            echo "cluster created"
        }

        $wasbdemo = "wasb://demo@monstockageazure.blob.core.windows.net"
        
        Use-AzureHDInsightCluster $clusterName

        if ($runJob)
        {
            echo "will start job"
            
            $pigJob = New-AzureHDInsightPigJobDefinition -File "$wasbdemo/scripts/iislogs-pig-jython/iislogs-client-ip-time.pig"
            $pigJob.Files.Add("$wasbdemo/scripts/iislogs-pig-jython/iislogs-client-ip-time-helper.py")
            $pigJob.Files.Add("$wasbdemo/scripts/iislogs-pig-jython/jython-2.5.3.jar")
            $startedPigJob = $pigJob | Start-AzureHDInsightJob -Credential $clusterCredentials -Cluster $clusterName
            
            echo "will wait for the end of the job"

            $startedPigJob | Wait-AzureHDInsightJob -Credential $clusterCredentials
            echo "--- log ---"
            Get-AzureHDInsightJobOutput -JobId $startedPigJob.JobId -TaskSummary -StandardError -StandardOutput -Cluster $clusterName
            echo "job done"            
        }
        
        
        if ($removeCluster)
        {
            echo "will remove cluster"
            Remove-AzureHDInsightCluster -Name $clusterName
            echo "removed cluster"
        }
        echo "done"
    }
}

Here is a sample log of that job:

RunBook V140629c
Name                           Value                                                                                    
----                           -----                                                                                    
PSRemotingProtocolVersion      2.2                                                                                      
BuildVersion                   6.3.9600.16406                                                                           
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0}                                                                     
PSVersion                      4.0                                                                                      
CLRVersion                     4.0.30319.19455                                                                          
WSManStackVersion              3.0                                                                                      
SerializationVersion           1.1.0.1                                                                                  
ModuleType Version    Name                                ExportedCommands                PSComputerName                
---------- -------    ----                                ----------------                --------------                
Binary     0.8.0      azure                               {Remove-AzureStorageAccount,... localhost                     
will create cluster monclusterhadoop
PSComputerName        : localhost
PSSourceJobInstanceId : b7a8dc2b-3e11-467f-8dad-2a1ac76da33f
ClusterSizeInNodes    : 3
ConnectionUrl         : https://monclusterhadoop.azurehdinsight.net
CreateDate            : 6/29/2014 7:57:56 AM
DefaultStorageAccount : monstockageazure.blob.core.windows.net
HttpUserName          : cornac
Location              : North Europe
Name                  : monclusterhadoop
State                 : Running
StorageAccounts       : {}
SubscriptionId        : 0fa85b4c-aa27-44ba-84e5-fa51aac32734
UserName              : cornac
Version               : 3.0.3.234.879810
VersionStatus         : Compatible
cluster created
Successfully connected to cluster monclusterhadoop
will start job
will wait for the end of the job
PSComputerName        : localhost
PSSourceJobInstanceId : b7a8dc2b-3e11-467f-8dad-2a1ac76da33f
Cluster               : monclusterhadoop
ExitCode              : 0
Name                  : 
PercentComplete       : 100% complete
Query                 : 
State                 : Completed
StatusDirectory       : 54d76c5f-a0a3-4df8-84d3-8b58d05f51fd
SubmissionTime        : 6/29/2014 8:12:17 AM
JobId                 : job_1404029421644_0001
--- log ---
2014-06-29 08:12:28,609 [main] INFO  org.apache.pig.Main - Apache Pig version 0.12.0.2.0.9.0-1686 (r: unknown) compiled Jun 10 2014, 17:27:26
2014-06-29 08:12:28,609 [main] INFO  org.apache.pig.Main - Logging error messages to: C:\apps\dist\hadoop-2.2.0.2.0.9.0-1686\logs\pig_1404029548609.log
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/C:/apps/dist/hadoop-2.2.0.2.0.9.0-1686/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/C:/apps/dist/pig-0.12.0.2.0.9.0-1686/pig-0.12.0.2.0.9.0-1686.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See https://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2014-06-29 08:12:29,609 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file D:\Users\hdp/.pigbootup not found
2014-06-29 08:12:29,844 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-06-29 08:12:29,844 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-06-29 08:12:29,844 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: wasb://monclusterhadoop@monstockageazure.blob.core.windows.net
2014-06-29 08:12:30,515 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-06-29 08:12:30,547 [main] INFO  org.apache.pig.scripting.jython.JythonScriptEngine - created tmp python.cachedir=D:\Users\hdp\AppData\Local\Temp\pig_jython_8612239229935485131
2014-06-29 08:12:32,781 [main] WARN  org.apache.pig.scripting.jython.JythonScriptEngine - pig.cmd.args.remainders is empty. This is not expected unless on testing.
2014-06-29 08:12:34,828 [main] INFO  org.apache.pig.scripting.jython.JythonScriptEngine - Register scripting UDF: helper.date_time_by_the_minute
2014-06-29 08:12:34,828 [main] INFO  org.apache.pig.scripting.jython.JythonScriptEngine - Register scripting UDF: helper.new_uuid
2014-06-29 08:12:35,594 [main] INFO  org.apache.pig.scripting.jython.JythonFunction - Schema 'hit_date_time:chararray' defined for func date_time_by_the_minute
2014-06-29 08:12:35,750 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY,ORDER_BY,FILTER
2014-06-29 08:12:35,812 [main] INFO  org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, DuplicateForEachColumnRewrite, GroupByConstParallelSetter, ImplicitSplitInserter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, NewPartitionFilterOptimizer, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]}
2014-06-29 08:12:35,844 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.textoutputformat.separator is deprecated. Instead, use mapreduce.output.textoutputformat.separator
2014-06-29 08:12:36,078 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-06-29 08:12:36,219 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
2014-06-29 08:12:36,297 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 3
2014-06-29 08:12:36,297 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 3
2014-06-29 08:12:36,594 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at headnodehost/100.85.198.84:9010
2014-06-29 08:12:36,734 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-06-29 08:12:36,750 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent
2014-06-29 08:12:36,750 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-06-29 08:12:36,750 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2014-06-29 08:12:36,750 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Reduce phase detected, estimating # of required reducers.
2014-06-29 08:12:36,750 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2014-06-29 08:12:36,750 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=-1
2014-06-29 08:12:36,750 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Could not estimate number of reducers and no requested or default parallelism set. Defaulting to 1 reducer.
2014-06-29 08:12:36,750 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
2014-06-29 08:12:36,750 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
2014-06-29 08:12:37,347 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job1668949293984769897.jar
2014-06-29 08:12:46,909 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job1668949293984769897.jar created
2014-06-29 08:12:46,909 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.jar is deprecated. Instead, use mapreduce.job.jar
2014-06-29 08:12:46,940 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-06-29 08:12:46,987 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2014-06-29 08:12:46,987 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cache
2014-06-29 08:12:46,987 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Setting key [pig.schematuple.classes] with classes to deserialize []
2014-06-29 08:12:47,112 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2014-06-29 08:12:47,112 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.http.address is deprecated. Instead, use mapreduce.jobtracker.http.address
2014-06-29 08:12:47,112 [JobControl] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at headnodehost/100.85.198.84:9010
2014-06-29 08:12:47,206 [JobControl] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-06-29 08:12:49,043 [JobControl] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 71
2014-06-29 08:12:49,043 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 71
2014-06-29 08:12:49,090 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 12
2014-06-29 08:12:49,293 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - number of splits:12
2014-06-29 08:12:49,621 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1404029421644_0002
2014-06-29 08:12:49,621 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - Kind: mapreduce.job, Service: job_1404029421644_0001, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@5b531130)
2014-06-29 08:12:49,636 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - Kind: RM_DELEGATION_TOKEN, Service: 100.85.198.84:9010, Ident: (owner=cornac, renewer=mr token, realUser=hdp, issueDate=1404029526385, maxDate=1404634326385, sequenceNumber=1, masterKeyId=2)
2014-06-29 08:12:50,058 [JobControl] INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1404029421644_0002 to ResourceManager at headnodehost/100.85.198.84:9010
2014-06-29 08:12:50,121 [JobControl] INFO  org.apache.hadoop.mapreduce.Job - The url to track the job: https://headnodehost:9014/proxy/application_1404029421644_0002/
2014-06-29 08:12:50,121 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1404029421644_0002
2014-06-29 08:12:50,121 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases logs_date_cip,logs_date_cip_with_headers,logs_date_time_cip,logs_date_time_cip_count,logs_date_time_cip_group,logs_raw
2014-06-29 08:12:50,121 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: logs_raw[4,11],logs_date_cip_with_headers[5,29],logs_date_cip[6,16],logs_date_time_cip[7,21],logs_date_time_cip_count[9,27],logs_date_time_cip_group[8,27] C: logs_date_time_cip_count[9,27],logs_date_time_cip_group[8,27] R: logs_date_time_cip_count[9,27]
2014-06-29 08:12:50,168 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2014-06-29 08:13:33,534 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 4% complete
2014-06-29 08:13:36,784 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 11% complete
2014-06-29 08:13:39,034 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 16% complete
2014-06-29 08:13:39,566 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 20% complete
2014-06-29 08:13:41,258 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 33% complete
2014-06-29 08:13:46,533 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-06-29 08:13:46,533 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-06-29 08:13:46,533 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Reduce phase detected, estimating # of required reducers.
2014-06-29 08:13:46,533 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2014-06-29 08:13:46,533 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=-1
2014-06-29 08:13:46,533 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Could not estimate number of reducers and no requested or default parallelism set. Defaulting to 1 reducer.
2014-06-29 08:13:46,533 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
2014-06-29 08:13:47,042 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job8669508224269273799.jar
2014-06-29 08:13:56,229 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job8669508224269273799.jar created
2014-06-29 08:13:56,261 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-06-29 08:13:56,276 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2014-06-29 08:13:56,276 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cache
2014-06-29 08:13:56,276 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Setting key [pig.schematuple.classes] with classes to deserialize []
2014-06-29 08:13:56,307 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2014-06-29 08:13:56,307 [JobControl] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at headnodehost/100.85.198.84:9010
2014-06-29 08:13:56,386 [JobControl] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-06-29 08:13:57,318 [JobControl] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2014-06-29 08:13:57,318 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2014-06-29 08:13:57,318 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2014-06-29 08:13:57,505 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1
2014-06-29 08:13:57,630 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1404029421644_0003
2014-06-29 08:13:57,630 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - Kind: mapreduce.job, Service: job_1404029421644_0001, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@3c124a75)
2014-06-29 08:13:57,630 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - Kind: RM_DELEGATION_TOKEN, Service: 100.85.198.84:9010, Ident: (owner=cornac, renewer=mr token, realUser=hdp, issueDate=1404029526385, maxDate=1404634326385, sequenceNumber=1, masterKeyId=2)
2014-06-29 08:13:57,755 [JobControl] INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1404029421644_0003 to ResourceManager at headnodehost/100.85.198.84:9010
2014-06-29 08:13:57,771 [JobControl] INFO  org.apache.hadoop.mapreduce.Job - The url to track the job: https://headnodehost:9014/proxy/application_1404029421644_0003/
2014-06-29 08:13:57,771 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1404029421644_0003
2014-06-29 08:13:57,771 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases logs_date_time_cip_count_ordered
2014-06-29 08:13:57,771 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: logs_date_time_cip_count_ordered[10,35] C:  R: 
2014-06-29 08:14:16,338 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2014-06-29 08:14:24,487 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 66% complete
2014-06-29 08:14:28,544 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-06-29 08:14:28,544 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-06-29 08:14:28,544 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Reduce phase detected, estimating # of required reducers.
2014-06-29 08:14:28,544 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
2014-06-29 08:14:29,021 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job3067576297837929164.jar
2014-06-29 08:14:38,100 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job3067576297837929164.jar created
2014-06-29 08:14:38,115 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-06-29 08:14:38,115 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2014-06-29 08:14:38,115 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cache
2014-06-29 08:14:38,115 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Setting key [pig.schematuple.classes] with classes to deserialize []
2014-06-29 08:14:38,162 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2014-06-29 08:14:38,193 [JobControl] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at headnodehost/100.85.198.84:9010
2014-06-29 08:14:38,256 [JobControl] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-06-29 08:14:39,129 [JobControl] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2014-06-29 08:14:39,129 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2014-06-29 08:14:39,145 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2014-06-29 08:14:39,333 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1
2014-06-29 08:14:39,442 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1404029421644_0004
2014-06-29 08:14:39,442 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - Kind: mapreduce.job, Service: job_1404029421644_0001, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@6791a1a0)
2014-06-29 08:14:39,442 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - Kind: RM_DELEGATION_TOKEN, Service: 100.85.198.84:9010, Ident: (owner=cornac, renewer=mr token, realUser=hdp, issueDate=1404029526385, maxDate=1404634326385, sequenceNumber=1, masterKeyId=2)
2014-06-29 08:14:40,604 [JobControl] INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1404029421644_0004 to ResourceManager at headnodehost/100.85.198.84:9010
2014-06-29 08:14:40,610 [JobControl] INFO  org.apache.hadoop.mapreduce.Job - The url to track the job: https://headnodehost:9014/proxy/application_1404029421644_0004/
2014-06-29 08:14:40,610 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1404029421644_0004
2014-06-29 08:14:40,610 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases logs_date_time_cip_count_ordered
2014-06-29 08:14:40,610 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: logs_date_time_cip_count_ordered[10,35] C:  R: 
2014-06-29 08:14:55,976 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 83% complete
2014-06-29 08:15:11,394 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2014-06-29 08:15:11,597 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: 
HadoopVersion    PigVersion    UserId    StartedAt    FinishedAt    Features
2.2.0.2.0.9.0-1686    0.12.0.2.0.9.0-1686    hdp    2014-06-29 08:12:36    2014-06-29 08:15:11    GROUP_BY,ORDER_BY,FILTER
Success!
Job Stats (time in seconds):
JobId    Maps    Reduces    MaxMapTime    MinMapTIme    AvgMapTime    MedianMapTime    MaxReduceTime    MinReduceTime    AvgReduceTime    MedianReducetime    Alias    Feature    Outputs
job_1404029421644_0002    12    1    33    16    29    31    20    20    20    20    logs_date_cip,logs_date_cip_with_headers,logs_date_time_cip,logs_date_time_cip_count,logs_date_time_cip_group,logs_raw    GROUP_BY,COMBINER    
job_1404029421644_0003    1    1    7    7    7    7    5    5    5    5    logs_date_time_cip_count_ordered    SAMPLER    
job_1404029421644_0004    1    1    5    5    5    5    7    7    7    7    logs_date_time_cip_count_ordered    ORDER_BY    /wasbwork/iislogs-client-ip,
Input(s):
Successfully read 592234 records from: "wasb://demo@monstockageazure.blob.core.windows.net/data/iislogs"
Output(s):
Successfully stored 712 records in: "/wasbwork/iislogs-client-ip"
Counters:
Total records written : 712
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_1404029421644_0002    ->    job_1404029421644_0003,
job_1404029421644_0003    ->    job_1404029421644_0004,
job_1404029421644_0004
2014-06-29 08:15:31,611 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: workernode0/100.85.210.91:49416. Already tried 0 time(s); maxRetries=3
2014-06-29 08:15:51,619 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: workernode0/100.85.210.91:49416. Already tried 1 time(s); maxRetries=3
2014-06-29 08:16:11,631 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: workernode0/100.85.210.91:49416. Already tried 2 time(s); maxRetries=3
2014-06-29 08:16:31,768 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2014-06-29 08:16:54,015 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: workernode1/100.85.164.84:49580. Already tried 0 time(s); maxRetries=3
2014-06-29 08:17:14,021 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: workernode1/100.85.164.84:49580. Already tried 1 time(s); maxRetries=3
2014-06-29 08:17:34,027 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: workernode1/100.85.164.84:49580. Already tried 2 time(s); maxRetries=3
2014-06-29 08:17:54,161 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2014-06-29 08:17:56,525 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: workernode2/100.85.206.85:49658. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1 SECONDS)
2014-06-29 08:17:58,543 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: workernode2/100.85.206.85:49658. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1 SECONDS)
2014-06-29 08:18:00,583 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: workernode2/100.85.206.85:49658. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1 SECONDS)
2014-06-29 08:18:01,705 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2014-06-29 08:18:01,908 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning ACCESSING_NON_EXISTENT_FIELD 330 time(s).
2014-06-29 08:18:01,908 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
job done
will remove cluster
removed cluster
done

Benjamin (@benjguin)

I regulalry need to check if my Azure virtual machines are turned off before leaving. Besides virtual machines, I also want to know in which pricing level my web sites, Azure SQL Databases, etc are running. I want to know if I have an HDInsight cluster running.

So I created an Azure automation job that checks the subscriptions at 6pm every day.

Here is how it looks:

How to start

You can start using Azure automation by following the instructions available here:

https://azure.microsoft.com/en-us/documentation/articles/automation-create-runbook-from-samples/

Credentials

The script will need to get access to the subscriptions.

So I created a management certificate. One way to do so is explained in this blog post by Keith Mayer.

In my case, here is how my environment looks:

In Azure automation, the same certificate is declared in the assets:

The script

Here is how the script itself:

workflow Inventory
{ 
    # Get the Azure management certificate that is used to connect to this subscription
    $Certificate = Get-AutomationCertificate -Name 'azure-admin.3-4.fr'
    if ($Certificate -eq $null)
    {
        throw "Could not retrieve '$AzureConn.AutomationCertificateName' certificate asset. Check that you created this first in the Automation service."
    }

    InlineScript
    {
        $Certificate = $using:Certificate

        $subscriptions = (('Azdem169A44055X','0fa8xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'), 
        ('Azure bengui','b4edxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'), 
        ('APVX','0ec1xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'), 
        ('demos-frazurete','4b57xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'))
    
        foreach ($subscription in $subscriptions)
        {
            $subscriptionName = $subscription[0]
            $subscriptionId = $subscription[1]
            
            echo "------- Subscription $subscriptionName ----------"

            # Set the Azure subscription configuration
            Set-AzureSubscription -SubscriptionName $subscriptionName -SubscriptionId $subscriptionId -Certificate $Certificate
            Select-AzureSubscription -Current $subscriptionName 

            $vms = @()
            foreach ($s in Get-AzureService)
            {
                $vms += Get-AzureVm -ServiceName $s.ServiceName
            }
            echo "--- Virtual Machines ---"
            $vms | select servicename, Name, PowerState | format-table
            
            $hclusters=Get-AzureHDInsightCluster
            echo "--- HDInsight clusters ---"
            $hclusters | format-table
        
            $webs=Get-AzureWebSite
            echo "--- Web Sites ---"
            $webs | select Name, SiteMode |  sort Name | format-table
            
            $dbs = @()
            foreach ($s in Get-AzureSqlDatabaseServer)
            {
                $dbs += Get-AzureSqlDatabase -ServerName $s.ServerName
            }
            echo "--- SQL Databases ---"
            $dbs | select Name, Edition, MaxSizeinGb | format-table    
        }
    }
}

NB: I have obfuscated the subscription ids.

Make it your own

You can change the script for your own usage. You would need to change the certificate name (mine is azure-admin.3-4.fr), the names and ids of your subscriptions).

Benjamin (@benjguin)

The drawback of the latter is that you use streaming between Hive and Python. In Hadoop streaming is just a way to call stdin/stdout inter process communication. So if you just do simple operations like string concatenations between two fields in Python it may be slow. the good things is that hive has user defined functions and also standard ones that help do all the simple things (like string concatenation).

There’s a way to use Python language without using streaming: just run Python in the JVM (remember Hadoop is written in Java). Python in the JVM is Jython. And Pig (an equivalent to Hive that has its own scripting language, instead of using SQL) can call Jython scripts.

With HDInsight 3.0 which became generally available recently, you can use that kind of feature. Here’s how. In order to launch the job here, I use a script from Linux that leverages WebHCat / Templeton REST API from a Linux machine. Here is how.

The Python script that launches the job is the following:

import requests #https://pypi.python.org/pypi/requests

clusterName='monclusterhadoop'
clusterAdmin='cornac'
clusterPassword='ChangeWithY0urs!'

#get WebHCat status
webHCatUrl='https://' + clusterName + '.azurehdinsight.net/templeton/v1/status'

r = requests.get(webHCatUrl, auth=(clusterAdmin, clusterPassword))

print r.status_code
print r.json()

#submit a pig job:
# https://docs.hortonworks.com/HDPDocuments/HDP1/HDP-Win-1.3.0/ds_HCatalog/pig.html

webHCatUrl='https://' + clusterName + '.azurehdinsight.net/templeton/v1/pig'

hive_params={'user.name':clusterAdmin,
             'file':'wasb://demo@monstockageazure.blob.core.windows.net/scripts/pig_python/pig_python.pig',
             'statusdir': '/wasbwork/pig_from_python'}

r = requests.post(webHCatUrl, auth=(clusterAdmin, clusterPassword), data=hive_params)
print r.status_code
print r.json()

the pig job looks like this:

Register 'wasb://demo@monstockageazure.blob.core.windows.net/scripts/pig_python/pig_python.py' using jython as myfuncs;

a = load 'wasb://demo@monstockageazure.blob.core.windows.net/data/ref_villes' using PigStorage(' ') as (ville:chararray);
b = foreach a generate ville, myfuncs.helloworld(), myfuncs.square(3);
store b into '/wasbwork/pigresult';

the Python script called by Pig and defines a few sample basic functions is the following:

#!/usr/bin/python

@outputSchema("word:chararray")
def helloworld():
    return ('Hello, World')
 
@outputSchema("t:(word:chararray,num:long)")
def complex(word):
    return (str(word),long(word)*long(word))

@outputSchemaFunction("squareSchema")
def square(num):   
    return ((num)*(num))   

@schemaFunction("squareSchema") 
def squareSchema(input):   
    return input   

# No decorator - bytearray 
def concat(str):   
    return str+str

Source data (ref_villes) looks like this (first lines) :

paris
marseille
lyon
toulouse
nice
nantes
strasbourg
montpellier
bordeaux
lille
rennes
reims
le havre
saint-etienne
toulon
grenoble

the output (part-m-00000) looks like this

paris    Hello, World    9
marseille    Hello, World    9
lyon    Hello, World    9
toulouse    Hello, World    9
nice    Hello, World    9
nantes    Hello, World    9
strasbourg    Hello, World    9
montpellier    Hello, World    9
bordeaux    Hello, World    9
lille    Hello, World    9
rennes    Hello, World    9
reims    Hello, World    9
le    Hello, World    9
saint-etienne    Hello, World    9
toulon    Hello, World    9
grenoble    Hello, World    9

the execution report looks like this (stderr):

2014-03-21 11:50:59,951 [main] INFO  org.apache.pig.Main - Apache Pig version 0.12.0.2.0.7.0-1551 (r: unknown) compiled Feb 19 2014, 11:47:04
2014-03-21 11:50:59,951 [main] INFO  org.apache.pig.Main - Logging error messages to: C:\apps\dist\hadoop-2.2.0.2.0.7.0-1551\logs\pig_1395402659935.log
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/C:/apps/dist/hadoop-2.2.0.2.0.7.0-1551/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/C:/apps/dist/pig-0.12.0.2.0.7.0-1551/pig-0.12.0.2.0.7.0-1551.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See https://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2014-03-21 11:51:00,810 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file D:\Users\hdp/.pigbootup not found
2014-03-21 11:51:00,997 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-03-21 11:51:00,997 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-03-21 11:51:00,997 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: wasb://monclusterhadoop@monstockageazure.blob.core.windows.net
2014-03-21 11:51:01,451 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-03-21 11:51:01,841 [main] INFO  org.apache.pig.scripting.jython.JythonScriptEngine - created tmp python.cachedir=D:\Users\hdp\AppData\Local\Temp\pig_jython_5196260548692206718
2014-03-21 11:51:03,951 [main] WARN  org.apache.pig.scripting.jython.JythonScriptEngine - pig.cmd.args.remainders is empty. This is not expected unless on testing.
2014-03-21 11:51:04,560 [main] INFO  org.apache.pig.scripting.jython.JythonScriptEngine - Register scripting UDF: myfuncs.complex
2014-03-21 11:51:04,560 [main] INFO  org.apache.pig.scripting.jython.JythonScriptEngine - Register scripting UDF: myfuncs.square
2014-03-21 11:51:04,576 [main] INFO  org.apache.pig.scripting.jython.JythonScriptEngine - Register scripting UDF: myfuncs.helloworld
2014-03-21 11:51:04,576 [main] INFO  org.apache.pig.scripting.jython.JythonScriptEngine - Register scripting UDF: myfuncs.concat
2014-03-21 11:51:04,701 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-03-21 11:51:04,951 [main] INFO  org.apache.pig.scripting.jython.JythonFunction - Schema 'word:chararray' defined for func helloworld
2014-03-21 11:51:05,232 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2014-03-21 11:51:05,326 [main] INFO  org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, DuplicateForEachColumnRewrite, GroupByConstParallelSetter, ImplicitSplitInserter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, NewPartitionFilterOptimizer, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]}
2014-03-21 11:51:05,482 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.textoutputformat.separator is deprecated. Instead, use mapreduce.output.textoutputformat.separator
2014-03-21 11:51:05,763 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-03-21 11:51:05,810 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-03-21 11:51:05,810 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-03-21 11:51:06,091 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at headnode0/100.86.204.54:9010
2014-03-21 11:51:06,263 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-03-21 11:51:06,263 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent
2014-03-21 11:51:06,263 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-03-21 11:51:06,263 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2014-03-21 11:51:06,279 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job608857099241848139.jar
2014-03-21 11:51:14,294 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job608857099241848139.jar created
2014-03-21 11:51:14,294 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.jar is deprecated. Instead, use mapreduce.job.jar
2014-03-21 11:51:14,341 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-03-21 11:51:14,341 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2014-03-21 11:51:14,341 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cache
2014-03-21 11:51:14,341 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Setting key [pig.schematuple.classes] with classes to deserialize []
2014-03-21 11:51:14,404 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2014-03-21 11:51:14,404 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.http.address is deprecated. Instead, use mapreduce.jobtracker.http.address
2014-03-21 11:51:14,404 [JobControl] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at headnode0/100.86.204.54:9010
2014-03-21 11:51:14,513 [JobControl] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-03-21 11:51:16,154 [JobControl] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 2
2014-03-21 11:51:16,154 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 2
2014-03-21 11:51:16,185 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2014-03-21 11:51:16,435 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1
2014-03-21 11:51:16,732 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1395391185318_0006
2014-03-21 11:51:16,732 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - Kind: mapreduce.job, Service: job_1395391185318_0005, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@45d45314)
2014-03-21 11:51:16,763 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - Kind: RM_DELEGATION_TOKEN, Service: 100.86.204.54:9010, Ident: (owner=cornac, renewer=mr token, realUser=hdp, issueDate=1395402643673, maxDate=1396007443673, sequenceNumber=5, masterKeyId=2)
2014-03-21 11:51:17,154 [JobControl] INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1395391185318_0006 to ResourceManager at headnode0/100.86.204.54:9010
2014-03-21 11:51:17,232 [JobControl] INFO  org.apache.hadoop.mapreduce.Job - The url to track the job: https://headnode0:9014/proxy/application_1395391185318_0006/
2014-03-21 11:51:17,232 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1395391185318_0006
2014-03-21 11:51:17,232 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases a,b
2014-03-21 11:51:17,232 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: a[3,4],b[-1,-1] C:  R: 
2014-03-21 11:51:17,279 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2014-03-21 11:51:34,575 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2014-03-21 11:51:37,981 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
2014-03-21 11:51:38,028 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2014-03-21 11:51:38,028 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: 

HadoopVersion    PigVersion    UserId    StartedAt    FinishedAt    Features
2.2.0.2.0.7.0-1551    0.12.0.2.0.7.0-1551    hdp    2014-03-21 11:51:06    2014-03-21 11:51:38    UNKNOWN

Success!

Job Stats (time in seconds):
JobId    Maps    Reduces    MaxMapTime    MinMapTIme    AvgMapTime    MedianMapTime    MaxReduceTime    MinReduceTime    AvgReduceTime    MedianReducetime    Alias    Feature    Outputs
job_1395391185318_0006    1    0    6    6    6    6    n/a    n/a    n/a    n/a    a,b    MAP_ONLY    /wasbwork/pigresult,

Input(s):
Successfully read 260 records from: "wasb://demo@monstockageazure.blob.core.windows.net/data/ref_villes"

Output(s):
Successfully stored 260 records in: "/wasbwork/pigresult"

Counters:
Total records written : 260
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_1395391185318_0006


2014-03-21 11:51:38,278 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

Benjamin (@benjguin)

Je mets ici les diapos et les vidéos (les vidéos sont les vidéos de secours que j’avais, et non la présentation live qui a été faite, mais c’est évidemment très proche).

Cela permet d’avoir une première vision rapide de ce qu’on peut faire avec un cluster HDInsight. C’est un moyen très abordable (autant en termes financiers que technique) de démarrer avec Hadoop.

Les diapos complètes sont disponibles sur OneDrive.

La problématique:

Si vous voulez tester par vous-même, vous pouvez aller à https://aka.ms/tester-mon-azure où vous aurez 150 € de ressources Windows Azure pour tester pendant 1 mois.

Voici les vidéos:

Benjamin (@benjguin)

Storage account <mystorage> has container(s) which have an active image and/or disk artifacts. Ensure those artifacts are removed from the image repository before deleting this storage account.

Here is what you may want to check. In the management portal https://manage.windowsazure.com, Storage, <mystorage>, Containers, check the content of your containers, especially the “vhds” one which contains virtual hard disks by default. Here is an example of the portal with stockageazure3 instead of <mystorage>

By clicking on the arrow near vhds (or each any other container), you’ll find a list of the blobs inside the container. VHD are good candidates for lock, and we’ll see why in a minute.

Select a .vhd blob and click EDIT at the bottom of the screen

this will show you the lock:

So where do I unlock?

This is related to the way Virtual Machines Work in Windows Azure. The OS disk and data disks live in Windows Azure blob storage. Here is an image of that (here with Windows VMs, but this is very similar with Linux VMs):

So VHD blobs are virtual hard disks that may be used by virtual machines; Windows Azure doesn’t want you to remove a virtual machine disk without knwoing about it! The locks are handled by images, and disks, that you can find here:

for an image or a disk, you can see the referenced blob in the LOCATION column. Here is an example:

in this example, myCentoOSImage references the bueearwy.zbn201304031550350920.vhd blob in the vhds container of the northeurope2affstorage storage account. Thus the URI of https://northeurope2affstorage.blob.core.windows.net/vhds/bueearwy.zbn201304031550350920.vhd.

At the bottom of the screen, you can remove the image and optionally also remove the associated VHD:

Same for DISKS

A disk or an image may itself be locked by a virtual machine instance. In such a case, you may have to stop and remove the virtual machiane first.

For example, in the following screen shot, the benjguinu1 virtual machine holds the benjguinmisc-benjguinu1-0-201310042100440782 disk which locks itself the https://stockageazure2.blob.core.windows.net/vhds/benjguinmisc-benjguinu1-2013-10-04.vhd blob.

So we have

and you cannot remove the referenced blobs without removing the disks, images and VM involved in that chain.

Benjamin (@benjguin)