Mapred-site.xml

Core-site.xml

HDFS-site.xml

Hadoop Configuration on Data Node

HADOOP_HOME=c:\apps\dist\hadoop-1.1.0-SNAPSHOT

HADOOP_OPTS= -Dfile.encoding=UTF-8 -Dhadoop.home.dir=c:\apps\dist\hadoop-1.1.0-SNAPSHOT -Dhadoop.root.logger=INFO,console,DRFA -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.log.dir=c:\apps\dist\hadoop-1.1.0-SNAPSHOT\logs -Dhadoop.log.file=hadoop-hive-RD00155D43B7BE.log

HIVE_CONF_DIR=c:\apps\dist\hive-0.9.0\conf

HIVE_HOME=c:\apps\dist\hive-0.9.0

HIVE_LIB_DIR=c:\apps\dist\hive-0.9.0\lib

HIVE_OPTS= -hiveconf hive.querylog.location=c:\apps\dist\hive-0.9.0\logs\history -hiveconf hive.log.dir=c:\apps\dist\hive-0.9.0\logs

Hadoop User Log:

C:\apps\dist\hadoop-1.1.0-SNAPSHOT\logs\userlogs

MapReduce Task Tracker User Logs:

C:\hdfs\mapred\local\userlogs

Step 1: Submitting WordCount MapReduce Job to 4 node HDInsight cluster:

c:\apps\dist\hadoop-1.1.0-SNAPSHOT\bin\hadoop.cmd jar c:\apps\Jobs\templates\635000448534317551.hadoop-examples.jar wordcount /user/admin/DaVinci.txt /user/admin/outcount

The results are stored @ /user/admin/outcount

Verify the results at Interactive Shell:

js> #ls /user/admin/outcount

Found 2 items

-rwxrwxrwx 1 admin supergroup 0 2013-03-28 05:22 /user/admin/outcount/_SUCCESS

-rwxrwxrwx 1 admin supergroup 337623 2013-03-28 05:22 /user/admin/outcount/part-r-00000

Step 2:  loading /user/admin/outcount/part-r-00000 results in the Pig: 

First we are storing the flat text file data as words, wordCount format as below:

Grunt>mydata = load '/user/admin/output/part-r-00000′ using PigStorage('\t') as (words:chararray, wordCount:int);

Grunt>first10 = LIMIT mydata 10;

Grunt>dump first10;

Note: This shows results for the words with frequency 1.  We need to reorder to results on descending order to get words with top frequency.

Grunt>mydatadsc = order mydata by wordCount DESC;

Grunt>first10 = LIMIT mydatadsc 10;

Grunt>dump first10;

Now we have got the result as expected. Lets stored the results into a file at HDFS.

Grunt>Store first10 into '/user/avkash/myresults10' ;

Step 3:  Filtering Pig Results  in to Hive Table: 

First we will create a table in Hive using the same format (words and  wordcount separated by comma)

hive> create table wordslist10(words string, wordscount int) row format delimited fields terminated by ',' lines terminated by '\n';

Now once table is created we will load the hive store file '/user/admin/myresults10/part-r-00000′ into wordslist10 table we just created:

hive> load data inpath '/user/admin/myresults10/part-r-00000′ overwrite into table wordslist10;

That's all as you can see the results now in table:

hive> select * from wordslist10;

KeyWords: Apache Hadoop, MapReduce, Pig, Hive, HDInsight, BigData

To enable debugging you just need to set $DebugPreference as below:

$DebugPreference ='Continue'

To disable debugging you just need to set $DebugPreference as below:

$DebugPreference ='SilentlyContinue'

Example:

PS> $DebugPreference ='Continue'

PS>Get-AzureService –ServiceName $tempServiceName

DEBUG: ============================ HTTP REQUEST ============================
 
 HTTP Method:
 GET
 
 Absolute Uri:
 https://management.core.windows.net/****************************************/services/hostedservices/avkashwin8
 
 Headers:
 x-ms-version : 2012-03-01
 User-Agent : Windows Azure Powershell/v.0.6.10
 
 Body:
 
 
 DEBUG: ============================ HTTP RESPONSE ============================
 
 Status Code:
 OK
 
 Headers:
 x-ms-request-id : 4ede3883c5e74513af7af675dbe6****
 Content-Length : 623
 Cache-Control : no-cache
 Content-Type : application/xml; charset=utf-8
 Date : Wed, 13 Feb 2013 07:38:19 GMT
 Server : 33.0.6198.5 (rd_rdfe_stable.130128-1959) Microsoft-HTTPAPI/2.0
 
 Body:
 <HostedService xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
 
 <Url>https://management.core.windows.net/*************************/services/hostedservices/avkashwin8</Url>
 <ServiceName>avkashwin8</ServiceName>
 <HostedServiceProperties>
 <Description>Implicitly created hosted service2013-01-17 19:00</Description>
 <Location>West US</Location>
 <Label>*********</Label>
 <Status>Created</Status>
 <DateCreated>2013-01-17T19:00:39Z</DateCreated>
 <DateLastModified>2013-01-21T18:38:35Z</DateLastModified>
 <ExtendedProperties />
 </HostedServiceProperties>
 </HostedService>
 
 DEBUG: ============================ HTTP REQUEST ============================
 
 HTTP Method:
 GET
 
 Absolute Uri:
 https://management.core.windows.net/********************************/operations/4ede3883c5e74513af7af675dbe6****
 
 Headers:
 x-ms-version : 2012-03-01
 User-Agent : Windows Azure Powershell/v.0.6.10
 
 Body:
 
 
 DEBUG: ============================ HTTP RESPONSE ============================
 
 Status Code:
 OK
 
 Headers:
 x-ms-request-id : c15bbbcdd709451ca3699ec8e43957dc
 Content-Length : 232
 Cache-Control : no-cache
 Content-Type : application/xml; charset=utf-8
 Date : Wed, 13 Feb 2013 07:38:20 GMT
 Server : 33.0.6198.5 (rd_rdfe_stable.130128-1959) Microsoft-HTTPAPI/2.0
 
 Body:
 <Operation xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
 <ID>4ede3883-c5e7-4513-af7a-f675dbe6caf5</ID>
 <Status>Succeeded</Status>
 <HttpStatusCode>200</HttpStatusCode>
 </Operation>
 
 Label : avkashwin8
 Description : Implicitly created hosted service2013-01-17 19:00
 AffinityGroup :
 Location : West US
 Status : Created
 DateModified : 2013-01-21T18:38:35Z
 DateCreated : 2013-01-17T19:00:39Z
 Url : https://management.core.windows.net/*************************/services/hostedservices/avkashwin8
 ServiceName : avkashwin8
 OperationDescription : Get-AzureService
 OperationId : 4ede3883-c5e7-4513-af7a-f675dbe6caf5
 OperationStatus : Succeeded

PS>help azure

 PS>help php-dev

PS>help node-dev

PS>help python-dev

To enable debugging you just need to set $DebugPreference as below:

$DebugPreference ='Continue'

To disable debugging you just need to set $DebugPreference as below:

$DebugPreference ='SilentlyContinue'

Example:

PS> $DebugPreference ='Continue'

PS>Get-AzureService –ServiceName $tempServiceName

DEBUG: ============================ HTTP REQUEST ============================
 
 HTTP Method:
 GET
 
 Absolute Uri:
 https://management.core.windows.net/****************************************/services/hostedservices/avkashwin8
 
 Headers:
 x-ms-version : 2012-03-01
 User-Agent : Windows Azure Powershell/v.0.6.10
 
 Body:
 
 
 DEBUG: ============================ HTTP RESPONSE ============================
 
 Status Code:
 OK
 
 Headers:
 x-ms-request-id : 4ede3883c5e74513af7af675dbe6****
 Content-Length : 623
 Cache-Control : no-cache
 Content-Type : application/xml; charset=utf-8
 Date : Wed, 13 Feb 2013 07:38:19 GMT
 Server : 33.0.6198.5 (rd_rdfe_stable.130128-1959) Microsoft-HTTPAPI/2.0
 
 Body:
 <HostedService xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
 
 <Url>https://management.core.windows.net/*************************/services/hostedservices/avkashwin8</Url>
 <ServiceName>avkashwin8</ServiceName>
 <HostedServiceProperties>
 <Description>Implicitly created hosted service2013-01-17 19:00</Description>
 <Location>West US</Location>
 <Label>*********</Label>
 <Status>Created</Status>
 <DateCreated>2013-01-17T19:00:39Z</DateCreated>
 <DateLastModified>2013-01-21T18:38:35Z</DateLastModified>
 <ExtendedProperties />
 </HostedServiceProperties>
 </HostedService>
 
 DEBUG: ============================ HTTP REQUEST ============================
 
 HTTP Method:
 GET
 
 Absolute Uri:
 https://management.core.windows.net/********************************/operations/4ede3883c5e74513af7af675dbe6****
 
 Headers:
 x-ms-version : 2012-03-01
 User-Agent : Windows Azure Powershell/v.0.6.10
 
 Body:
 
 
 DEBUG: ============================ HTTP RESPONSE ============================
 
 Status Code:
 OK
 
 Headers:
 x-ms-request-id : c15bbbcdd709451ca3699ec8e43957dc
 Content-Length : 232
 Cache-Control : no-cache
 Content-Type : application/xml; charset=utf-8
 Date : Wed, 13 Feb 2013 07:38:20 GMT
 Server : 33.0.6198.5 (rd_rdfe_stable.130128-1959) Microsoft-HTTPAPI/2.0
 
 Body:
 <Operation xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
 <ID>4ede3883-c5e7-4513-af7a-f675dbe6caf5</ID>
 <Status>Succeeded</Status>
 <HttpStatusCode>200</HttpStatusCode>
 </Operation>
 
 Label : avkashwin8
 Description : Implicitly created hosted service2013-01-17 19:00
 AffinityGroup :
 Location : West US
 Status : Created
 DateModified : 2013-01-21T18:38:35Z
 DateCreated : 2013-01-17T19:00:39Z
 Url : https://management.core.windows.net/*************************/services/hostedservices/avkashwin8
 ServiceName : avkashwin8
 OperationDescription : Get-AzureService
 OperationId : 4ede3883-c5e7-4513-af7a-f675dbe6caf5
 OperationStatus : Succeeded

PS>help azure

 PS>help php-dev

PS>help node-dev

PS>help python-dev

PS D:\Users\avkash> Add-PSSnapin Microsoft.WindowsAzure.ServiceRuntime

 Add-PSSnapin : Cannot add Windows PowerShell snap-in Microsoft.WindowsAzure.ServiceRuntime because it is already added.
 Verify the name of the snap-in and try again.
 At line:1 char:13
 + Add-PSSnapin <<<< Microsoft.WindowsAzure.ServiceRuntime
 + CategoryInfo : InvalidArgument: (Microsoft.WindowsAzure.ServiceRuntime:String) [Add-PSSnapin], PSArgume ntException
 + FullyQualifiedErrorId : AddPSSnapInRead,Microsoft.PowerShell.Commands.AddPSSnapinCommand

I have create a Powershell script which auto detect the Windows Azure SDK version runtime in the Azure VM and then  fix this problem in registry and set the Role Busy as below:

[xml] $RoleModelXml = Get-Content "E:\RoleModel.xml"
 $sdkVersion = $RoleModelXml.RoleModel.Version
 Write-Output ("---------------------------------------")
 Write-Output("Your SDK Version is: " + $sdkVersion)
 Write-Output("---------------------------------------")
 
 Write-Output("")
 Write-Output("Now Setting SDK Version in Registry: " + $sdkVersion.substring(0,3))
 Write-Output("---------------------------------------")
 
 $finalSDKVersion = $sdkVersion.substring(0,3)
 $fixedAssemblyName = "Microsoft.WindowsAzure.ServiceRuntime.Commands, Version=" + $finalSDKVersion + ".0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
 
 $Registry_Key = “HKLM:\Software\Microsoft\PowerShell\1\PowerShellSnapIns\Microsoft.WindowsAzure.ServiceRuntime\"
 Set-ItemProperty -path $Registry_Key -name Version -value $finalSDKVersion
 Set-ItemProperty -path $Registry_Key -name PowerShellVersion -value 2.0
 Set-ItemProperty -path $Registry_Key -name AssemblyName -value $fixedAssemblyName
 
 
 #Validate the registry
 Write-Output("*** --- Validating the registry change *** ----")
 Get-ItemProperty -path $Registry_Key -name Version
 Get-ItemProperty -path $Registry_Key -name PowerShellVersion
 Get-ItemProperty -path $Registry_Key -name AssemblyName
 
 
 $AzureSnapIn = "Microsoft.WindowsAzure.ServiceRuntime"
 
 if ( (Get-PSSnapin -Name $AzureSnapIn -ErrorAction SilentlyContinue) -eq $null )
 {
 
 Add-PsSnapin $AzureSnapIn
 }
 else
 {
 $message = "PsSnapin " + $AzureSnapIn + " already loaded!!"
 Write-Output($message); 
 }
 
 
 #Settings the Role busy
 Write-Output("*** --- Setting Role Busy *** ----")
 Set-RoleInstanceStatus -busy
 

Step 1: First you would need to create using MAKECERT (Use VS2012 developer command prompt)

makecert -r -pe -n "CN=Avkash Azure Management Certificate" -a sha1 -ss My -len 2048 -sy 24 -b 09/01/2011 -e 01/01/2018

Step 2: Once certificate is created it will be listed in your Current User > Personal (My) store as below:

(Launch > Certmgr.msc to open the certificate mmc in Windows)

Step 3: Get the certificate Thumbprint ID and Serial Number (used in Step #13 to verify) from the certificate as below:

Thumbprint:  ‎55c96e885764055d9beccec34dcd1ea82e601d4b

Serial Number: ‎85928750c5d9229d437287103ee08a79

Step 4: Now export this certificate to BASE 64 encoded certificate as below an save as CER file locally:

Step 5: Now upload your above created certificate (avkashmgmtBase64.cer) to your Windows Azure Management Portal. Be sure that the same certificate is listed as below:

Step 6: Be sure to have your Fiddler setting configure to decrypt HTTPS traffic as described here:

 http://www.fiddler2.com/Fiddler/help/httpsdecryption.asp

Step 7:  I would assume that you already have Fiddler installed in your machine. Now create a new copy of avkashmgmtBase64.cer as ClientCertificate.cer.

Copy this certificate @ C:\Users\<Your_User_Name>\Documents\Fiddler2\ ClientCertificate.cer

This is the certificate will be used by Fiddler to decrypt the HTTPS traffic. This is very important step. 

Step 8: Now if you have already used Azure Powershell before with publishsettings then you would need to clear those settings. These files are generated every time Windows Azure Powershell connects to Management Portal with different credentials.

Go to the following folder and remove all the files here:

C:\Users\<Your_user_name>\AppData\Roaming\Windows Azure Powershell

Note: if you have Powershell settings based on previous publishsettings configuration this step is must for you.

Step 9: Now create a powershell script and using your certificate Thumbprint which you have collected in step  #3 above:

Step 10: Run the above powershell script without without Fiddler running and verify that it is working.

Step 11: Once step #10 is verified, start Fiddler and check HTTP decryption is enabled.

Step 12: Run the powershell again and you will see that HTTPS traffic shown in the Fiddler shows decrypted data.

Step 13: To verify that you are using the correct certificate with Fiddler,  what you can do is to open the first connect URL (Tunnel to -> management.core.windows.net:443) and select its properties. In the new windows you can verify that

X-CLIENT_CERT is using the same certificate name which you have created and its serial number match which you have collected in step #3.

Many thanks to  Bin Du, Phil Hoff, Daniel Wang (AZURE) for helping me to get it working. 

Tags: Windows Azure, Powershell, Fiddler, HTTPS, HTTPSDecrypt

PS D:\Users\avkash> Add-PSSnapin Microsoft.WindowsAzure.ServiceRuntime

Add-PSSnapin : Cannot add Windows PowerShell snap-in Microsoft.WindowsAzure.ServiceRuntime because it is already added.
 Verify the name of the snap-in and try again.
 At line:1 char:13
 + Add-PSSnapin <<<< Microsoft.WindowsAzure.ServiceRuntime
 + CategoryInfo : InvalidArgument: (Microsoft.WindowsAzure.ServiceRuntime:String) [Add-PSSnapin], PSArgume ntException
 + FullyQualifiedErrorId : AddPSSnapInRead,Microsoft.PowerShell.Commands.AddPSSnapinCommand

The reason for this exception is conflict between Powershell version required by Windows Azure runtime components. AS you can see below the PS version is incorrectly set in the Azure role instance VM:

To solve this problem you would need to change the PowerShell version in Windows Azure Instance according to your Windows Azure application SDK. For example if your application is based on Windows Azure SDK 1.8 then you will need to change Powershell setting to use SDK 1.8 as below:

Version : 1.8.0.0

That's all you need to get powershell working,

When you want to reuse the OS disk vhd you might encounter the following problems:

1. You can see that OS Disk is still showing attached to Virtual Machine as shown below.

2. You can deleted the OS VHD from storage as while deleting VHD you get the following error:

Error deleting blob '/vhds/avkashsql2012-avkashsql2012-2012-07-18.vhd': details

There is currently a lease on the blob and no lease ID was specified in the request.

3. If you decide to use the same OS VHD to create an OS image for creating your Virtual Machine, you will get error as below:

         The VHD http://portalvhds63*.blob.core.windows.net/vhds/avkashsql2012-avkashsql2012-2012-07-18.vhd is already registered with image repository as the resource with ID avkashsql2012-avkashsql2012-0-20120718183454.

The bottom line is OS VHD is there but you cannot use it for any purpose.

Root cause:

-          The root cause of this problem is that the VHD blob is still in lease due to some code issue and locked up in a way that it is not re-usable until you break the lease so it is free to use.

Solution:

You cannot remove the blob lease directly at portal so you would need to use the PowerShell script as described below:

• http://social.msdn.microsoft.com/Forums/en-US/WAVirtualMachinesforWindows/thread/7381ea0e-0443-4b33-aa12-ba39df003409

To delete OS VHD blob:

-          If you have to delete the blob the you must need to break the least first and only then you would be able to delete

To reuse the OS VHD blob to create OS Image:

If you are looking for an alternative solution using the OS VHD to create an OS image and then use OS image to quickly create new Virtual Machine here is the solution.

-          Download CloudXplorer 

-          Configured you Azure Blob Stroage where your locked OS VHD is stored. You can get this info from portal.

-          Now access the OS VHD you want to reuse:

-          Try renaming the VHD which will return ERROR as below:

-          However even after the error you will see the rename did worked as below:

-          This new OS VHD blob is ready to use as it is brand new.

-          Now go back to Windows Azure Portal and access

• Virtual Machines -> Images -> Create Image (+)
• Route to your vhd container and then select your newly created VHD as shown below:

-          You can now see that a new OS Image is created as below in Virtual Machines > Images section as below:

-          Finally now you use this newly created OS Image to create a new Virtual Machines as below:

-       

With License Mobility through Software Assurance, you can deploy certain server application licenses purchased under your Volume Licensing agreement in an Authorized Mobility Partner’s datacenter.

• Extend the value of your server application license by deploying them on-premises or in the cloud.
• Take advantage of the lowest cost computing infrastructure for changing business priorities.

http://www.microsoft.com/licensing/software-assurance/license-mobility.aspx

Microsoft License Mobility through Software Assurance for Microsoft Windows Server Applications on AWS:

 Microsoft has announced a new benefit, called License Mobility through Software Assurance, which allows Microsoft customers to more easily move current on-premises Microsoft Server application workloads to Amazon Web Services (AWS), without any additional Microsoft software license fees.

• Microsoft Exchange Server
• Microsoft SharePoint Server
• Microsoft SQL Server Standard Edition
• Microsoft SQL Server Enterprise Edition
• Microsoft Lync Server
• Microsoft System Center Server
• Microsoft Dynamics CRM Server

http://aws.amazon.com/windows/mslicensemobility/

Support policy for Microsoft software running in non-Microsoft hardware virtualization software:

• http://support.microsoft.com/kb/897615 ?

Support policy for Microsoft SQL Server products that are running in a hardware virtualization environment

• http://support.microsoft.com/kb/956893

This includes:

• Running SQL Server in Microsoft Hyper-V based virtualized environment
• Running SQL Server in partner supported virtualized environment

Microsoft server software support for Windows Azure Virtual Machines:

http://support.microsoft.com/kb/2721672

1.    Microsoft SharePoint Server

Microsoft SharePoint Server 2010 and later is supported on Windows Azure Virtual Machines.

Support and licensing for Windows Azure in SharePoint 2013

-          http://technet.microsoft.com/en-us/library/jj154957

2.    Microsoft SQL Server

64-bit versions of Microsoft SQL Server 2008 and later are supported.

http://support.microsoft.com/?id=956893.

3.    Windows Server

Windows Server 2008 R2 and later versions are supported for the following roles:

• Active Directory Domain Services
• Active Directory Federation Services
• Active Directory Lightweight Directory Services
• Application Server
• DNS Server
• Fax Server
• Network Policy and Access Services
• Print and Document Services
• Web Server (IIS)
• Windows Deployment Services
• Windows Server Update Services
• File Services

The following features are not supported on Windows Azure Virtual Machines:

1. BitLocker
2. Failover Clustering
3. Network Load Balancing.

To download the Windows Azure PowerShell you can use the following command in PowerShell

PS C:\> Get-AzurePublishSettingsFile

The above command launches the following URL which allows you to download the .publishsettings on your local machine.

https://windows.azure.com/download/publishprofile.aspx?wa=wsignin1.0

Based on number of subscription(s) associated with your Live ID you will see that many certificate added in your management certificate sections as shows below:

Once you save the .publishsettings file locally you can use it with PowerShell as below:

PS C:\> Import-AzurePublishSettingsFile WindowsAzureInternal-11-20-2012-credentials.publishsettings

Setting: Windows Azure Internal Consumption - ******* as the default and current subscription. To view other subscriptions use Get-AzureSubscription

When above command runs it creates configuration specific setting the local machine located as below:

C:\Users\<Windows_Login_User_Name>\AppData\Roaming\Windows Azure Powershell

•  config.json
• DefaultSubscriptionData.xml
• publishSettings.xml

If you open publishsettings.xml you will see the Certificate Thumb print as below:

 <?xml version="1.0"?>
 <PublishData xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
 <PublishProfile PublishMethod="AzureServiceManagementAPI" Url="https://management.core.windows.net/" ManagementCertificate="********************************************">
 <Subscription Id="GUID_1" Name="Windows Azure Internal Consumption *" />
 <Subscription Id="GUID_2" Name="Windows Azure Internal Consumption ****" />
 </PublishProfile>
 </PublishData>
 

In some case you may get an error as below while importing PublishSettings using Import-AzurePublishSettingsFile:

 Import-AzurePublishSettingsFile : Value can not be null.
 Parameter name: s
 At line:1 char:1
 

If that is the case, try the following solution:

1. Go ahead and delete all the files in above location (C:\Users\<Windows_Login_User_Name>\AppData\Roaming\Windows Azure Powershell)
2. Try again using the same command. This will re-generate the configuration files again the publish settings will be imported.