• https://www.nuget.org/packages/Microsoft.Azure.Management.HDInsight
• https://www.nuget.org/packages/Microsoft.Azure.Management.HDInsight.Job

I am an engineer who loves automation because once you do it right once and you reap the benefits daily. That's why when someone tells me they deployed a HDInsight cluster from Azure Portal and keep managing by going there back and forth again, well, that does not impress me. As someone on the team who has written lots of test frameworks, automation tools in the past, I can't be productive if I have to type numerous things over and over again when using portal or PowerShell CmdLets. 

In this post, I will try to cover on how to start automating using the new Microsoft.Azure.Management.HDInsight SDK and provide you a tool that solves some of tasks related to cluster management (creation, deletion etc). I will not be covering the Microsoft.Azure.Management.HDInsight.Job SDK in this post.

Tools & Scripts:

Time to introduce my new GitHub repository where I will post all the tools and code related to HDInsight that I will talk on this blog: https://github.com/rtandonmsft/hdinsight-helper-tools

First tool: HDInsightManagementCLI - https://github.com/rtandonmsft/hdinsight-helper-tools/tree/master/src/HDInsightManagementCLI - this command line tool that allows you to manage your HDInsight clusters using the new SDK, get your subscription capabilities, resize your cluster etc. If you following the link from the GitHub repository to AppVeyor, you should be able to download the compiled binaries from latest build's artifacts: https://ci.appveyor.com/project/rtandonmsft/hdinsight-helper-tools

Automation Scripts can also be created using Azure PowerShell. The first version of these scripts used by our Storm examples that manage HDInsight clusters via the Azure Service Management (ASM) can be found here: https://github.com/hdinsight/hdinsight-storm-examples/tree/master/scripts/azure/HDInsight. I will be working on getting newer version of the PowerShell scripts up that are compatible with Azure Resource Manager (ARM).

Things to note:

• Azure HDInsight has moved to the Azure Resource Manager model, starting with the launch of Linux offerings of different cluster types: Hadoop, Storm, HBase, etc. This is partly because the Linux clusters are deployed as IaaS VMs compared to the Windows based PaaS VMs of the past. You will now treat an HDInsight cluster as a resource, managed through Azure Resource Manager.

static void CreateResourceGroup(string azureResourceManagementUri, string resourceGroupName, string location)
{
var resourceClient = new ResourceManagementClient(tokenCloudCredentials, new Uri(azureResourceManagementUri));
resourceClient.ResourceGroups.CreateOrUpdate(
resourceGroupName,
new ResourceGroup
{
Location = location,
});

resourceClient.Providers.Register("Microsoft.HDInsight");
}

• Certificate-based credentials won't work anymore, due to the new emphasis on Azure Active Directory authentication for all services in the Azure Resource Manager framework. This was the something I had to get used to: The move to Azure Active Directory for authentication. Don't worry, I now have code for you to use. I still don't like the pop-up for credentials. (note-to-self: figure out completely unattended mode)

AuthenticationContext ac = new AuthenticationContext("https://login.windows.net/" + config.AzureActiveDirectoryTenantId, true);
var token = ac.AcquireToken("https://management.core.windows.net/", config.AzureActiveDirectoryClientId, new Uri("urn:ietf:wg:oauth:2.0:oob"), PromptBehavior.Auto);
tokenCloudCredentials = new TokenCloudCredentials(config.SubscriptionId, token.AccessToken);
hdInsightManagementClient = new HDInsightManagementClient(tokenCloudCredentials, new Uri(https://management.azure.com/));

• If you are looking to deploy any new clusters (including Linux clusters) through the new SDK, new PowerShell cmdlets, or new portal, then you will need to switch to the new SDK, new PowerShell cmdlets, and/or new portal to continue to manage the cluster.

Workflow:

There are two parts to understanding this workflow, the first part is how you as the client/user communicates with Azure to create your resources (clusters) and the second part is how HDInsight works with Azure to deploy the clusters.

You can use Azure Resource Manager (ARM / New) or Azure Service Management (ASM / Old) to communicate with HDInsight cluster management APIs. HDInsight communicates with Azure Compute to procure the right type of VMs for your cluster: PaaS for Windows (via RDFE), IaaS for Linux (via ARM using template deployment).

Compatibility Table:

To conclude, if you created your cluster with the old SDK, old PowerShell, or old portal, you can continue using the old SDK. If you are going to create new clusters, including Linux, through the new tools, you should invest your time in moving to the new SDK.

I also have another management tool like HDInsightManagementCLI (now posted on GitHub) which uses the old SDK and talks to Azure Service Management to manage HDInsight clusters. If there is enough interest, I will publish that too.

Cheers
Ravi

Next post: HDInsight Cluster Logs Downloader

Read more about Azure Resource Manager (ARM) here:

• https://azure.microsoft.com/en-us/documentation/articles/resource-group-overview/
• https://msdn.microsoft.com/en-us/library/azure/dn790568.aspx
• https://azure.microsoft.com/en-us/documentation/articles/powershell-azure-resource-manager/
• https://azure.microsoft.com/en-us/documentation/articles/fundamentals-application-models/#CloudServices

In the last 3 years with HDInsight I have worked on multiple sections of the service:

• the deployment service (resource provider) - the service that creates your clusters
• engineering systems - continuous integration, push-to-cloud build systems
• and now I work on Apache Storm - as a PaaS service on both Windows & Linux - additionally providing C# .Net interfaces via SCP.Net

Why the blog now?

Each team in Microsoft operates differently. I used to blog a bit earlier but they were mostly on team blogs as most of the stuff we worked on was proprietary. I am engineer who likes to make things work, so I would naturally have to scrub a lot of information when it came to product code.

Moving to Microsoft HDInsight, things have changed. Now that I work more on the OSS side of things: bring Apache Hadoop, Apache Storm, Apache HBase to Azure, I have started to establish a connection with the community, which I would like to build on.

What to expect?

Earlier this year we stood up a repository to host Apache Storm examples that work on Microsoft HDInsight at https://github.com/hdinsight-storm-examples. The examples are mostly one-command runnable. I would like that repository to mature and provide more scenarios, tools on things that you can accomplish with HDInsight & Storm.

If there are any particular areas of interest where you want to know more about using HDInsight or Apache Storm, feel free to let me know. My goal has been to share as much scenarios, automation scripts, tools that make my development life easy and hopefully can make the community's life easier too.

Stay tuned.

-Ravi