Like many HPC applications, this one utilizes an SMB file share to enable all of the compute nodes to access common data and executable code.  At first, I simply created a file share on the (on-premises) HPC head node and set-up an Azure Connect group to allow all of the compute nodes to access it.  However, the performance was not acceptable.  So, I decided to create a file share on Azure to remove the Azure Connect bottleneck.  

It is possible to set up an SMB file share using an Azure worker role as described in this article:

https://blogs.msdn.com/b/windowsazurestorage/archive/2011/04/16/using-smb-to-share-a-windows-azure-drive-among-multiple-role-instances.aspx

However, I found that I could not use these guidelines for HPC managed Azure compute nodes, because the .csdef and .cscfg files are not accessible for customization.  The purpose of this blog entry is to document an approach for configuring one of the HPC Azure compute nodes to function as an SMB file share for all the others.

NOTE: There is nothing persistent about this file share (hence the name, “scratch”).  In fact, it may not even last the duration of a long running job, depending on the whim of the Azure fabric controller. But if you like to live dangerously, this will work.

The Azure based compute nodes that are provisioned by the on-premises HPC head node already have the File Services Role and the File Server Role Service installed, and port 445 is open by default on the internal endpoint.  So, all that is necessary is to run the following commands:

If the on-premises head node does not need to access the file share, then this is all that is necessary.  If the head node does need to access the file share, as was the case for this application, then a couple of other steps are required.  First, you need to make sure that Azure Connect is configured between the Head node and the Azure file server node.  Since most applications use a UNC path to access the file share (\\AzureCN-0006\scratch), all of the nodes need to be able to resolve the name to an IP Address.  The AzureCN-xxxx node names are managed by HPC and are maintained in the hosts files on each of the compute nodes.  However, since the head node communicates with the Azure compute nodes through a proxy, it does not have an entry in either DNS or its hosts file which references the AzureCN-xxxx name.  So, we need to add an entry in the hosts file on the head node which maps the IPv6 address that Azure Connect established to the AzureCN-xxxx name.   You can do that manually by using the "clusrun /nodes:AuzreCN-xxxx ipconfig" command to discover the IPv6 address for the PPP connection. 
Then, you need to add a row to the c:\windows\system32\drivers\etc\hosts file for that address and hostname combination.

When I worked through this process the first several times, I entered all of the commands manually, but then, I realized it would be simple to automate the entire process by submitting a job to the HPC scheduler.  So, I wrote this PowerShell script:

$an = Read-Host "Please enter the name of the Azure node to use for the file server"

$job = New-HpcJob -Name "Create a File Share" -RequestedNodes $an

$job | Add-HpcTask -Name "Create Folder" -Command "mkdir c:\scratch"

$job | Add-HpcTask -Name "Enable Guest User" -Command "net user guest /active:yes"

$job | Add-HpcTask -Name "Enable Firewall" -Command "netsh firewall set service type=fileandprint mode=enable scope=all"

$job | Add-HpcTask -Name "Grant Full Permissions to Guest" -Command "icacls c:\scratch /grant guest:F /t"

$job | Add-HpcTask -Name "Share Folder" -Command "net share scratch=c:\scratch /grant:everyone,FULL"

submit-hpcJob -Job $job

# Get the IPv6 address of the node

$IPConfiginfo = clusrun /nodes:$an ipconfig

$Addressline = $IPConfiginfo -match "IPv6"

$Addressline[1] -match
"[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}"

$hostsfileentry = $matches[0] + "   " + $an

# Insert new row into hosts file with AzureCN name

$hosts = "c:\windows\System32\Drivers\etc\hosts"

(gc $hosts) -replace "true", "$&`r`n$hostsfileentry" | sc $hosts

If you try this and have any problems, please let me know.

Recently, I was helping one of my clients build a VHD image for use as a VM Role on Windows Azure, and we discovered that the VM Role would not start.  We had decided to use a combination of base + differencing disk as per the recommendation from section 4.6 on this TechNet article:

If you will be frequently updating an application running on the VM nodes or changing the configuration of the operating system, you may want to upload a base VHD to Windows Azure on which the application is not installed. Then, you can use Hyper-V Manager and the base VHD to create a differencing VHD that contains the application updates and other configuration changes. When you need to update the Windows Azure VM nodes, you can upload the differencing VHD. For step-by-step procedures, see How to Change a Server Image for a VM Role by Using a Differencing VHD (https://go.microsoft.com/fwlink/p/?LinkId=217131).

Initially, we tried to keep the image as small as possible (to reduce the upload time), because we assumed that we would be able to expand it later if necessary.  However, when we ran out of space and ran the "Edit Disk" wizard from the Hyper-V management console, we discovered this warning message:

Do not edit a virtual hard disk when it is used by a virtual machine that has snapshots, or when it is associated with a differencing virtual hard disk.  Otherwise, data loss may occur.

So, we had to throw away all of the updates and applications we had installed on the differencing disk, expand the parent disk, create a new differencing disk, and re-do all the application installation and configuration work.  When we ran out of space, we noticed that even with the differencing disk, the size of the C:\ drive on the Virtual Machine was still the same as on the parent VHD, so we assumed that the differencing disk did not increase the size of the disk.  We didn't want to run out of space and have to repeat that process again, so we decided to expand the parent disk 65GB which is the maximum size specified for Medium, Large, and Extra Large Azure instances.  We ran sysprep and used the csupload utility to upload the parent VHD image (65GB) to the Azure storage.  We then created a differencing disk, installed all the applications, etc., sysprepped, and uploaded that image (9GB).

We then ran a PowerShell script to provision several VM Roles based on the differencing disk.  After about 30 minutes, the status message on the Azure portal said , "Failed to Start".  There was no indication of what the problem was.

So, we changed the .cscfg file to specify the parent VHD instead of the differencing VHD and ran the provisioning script again.  This time, the nodes all provisioned successfully with no problems. Because this worked, we went back to the Hyper-V "Edit Disk" and ran the Merge command to incorporate all of the changes from the differencing disk onto the parent.  We then uploaded the new merged image and were able to successfully provision the VM Roles.

I have seen VM Roles which were based on differencing VHD's get provisioned successfully, but it will not work if the parent VHD is already at the maximum size limit.

Cannot find the VM Role Agent.
Failed the Verification tests.
Cannot prepare VHD e:\VMRole.vhd

Thinking that I might have installed the wrong Azure Integration Components, I fired up the Virtual Machine, uninstalled the Azure Integration Components, and then re-installed them after double checking that I was using
the latest (v1.6) version.  I ran sysprep again and tried the csupload again but received the same error.

Next, I decided to try uploading the VHD using the HPC management console rather than the csupload utility. The upload was successful, but I was not able to provision a VMRole using that VHD.  After spinning for an
hour, the Azure Management Portal displayed an error message indicating that it could not connect and that the integration components might be missing.  So, the csupload utility actually did a better job verifying the image than the HPC image upload wizard did.

At this point, I was ready to give up on trying to use VMRoles altogether.  However, in my research, I discovered a blurb on this website which stated:

Window Azure Integration Components incompatible with Web Server Edition of Windows Server 2008 R2

Applies to: Windows Azure SDK 1.3 and later

Symptom: VM Roles based in Window Web Server Edition are stuck with a status “Setting Up” in the Windows Azure Portal.

Cause: The Windows Azure Integration Components depend on a system assembly which is not available in Windows Web Server 2008 R2

Resolution: Use Windows Server 2008 R2 Enterprise instead of Windows Web Server 2008 R2.

I had not used Web Server Edition when I built my VHD, I had used Standard Edition.  But since the resolution advice did not recommend the Standard edition, I wondered if there might be an incompatibility there as
well, even though on this site it says Standard should work.

So, I started from scratch and created a new VHD using the Enterprise edition of the OS and the integration components from the v1.6 Azure SDK.

The new image passed the validation step, uploaded and successfully spun up as a VMRole in Azure.

This may not be a new condition.  I have deleted my old VHD’s so I cannot say for sure whether the old images that worked in the past were built on Standard or Enterprise.  The bottom line is that there seems to be an incompatibility between the v1.6 Azure SDK and the Standard edition of the OS, so I will be using the Enterprise Edition as the base for all of my VMRoles in the future.

I realized that the problem was probably the mismatch between SP2 on the VMRole nodes and SP3 on the HPC head node. So, I created a differencing VHD and fired up a new Virtual Machine, uninstalled the HPC Pack, and installed the HPC Pack with SP3. After running sysprep, I tried to upload the new VHD to Azure storage using the csupload utility, but I encountered this error:

Cannot contact server Could not establish secure channel for SSL/TLS with authority 'management.core.windows.net'.

After scratching my head for some time, it dawned on me that I might have a problem with the certificate, even though the connection parameter specified the same certificate that I had used successfully in the past.

I have also been working on some of the learning exercises in the Windows HPC with Burst to Windows Azure Training Course available here:

https://msdn.microsoft.com/en-us/HPCAzureBurstTrainingCourse

A couple of those examples leverage the REST API's for submitting jobs to the HPC scheduler. As part of that setup, there is a step where I was given a choice to select a certificate for binding to port 443, and I can’t remember which certificate I chose, but it was obviously not the same one that I had been using for the Azure connections.

So,I opened a command prompt and ran this command to determine which certificate was bound to port 443:

netsh http show sslcert

I then opened up the MMC console, and learned a new trick.  I added the Certificate snap-in twice – once for the Local Machine, and once for the Current User.  This saved me having to do an export from one and import to the other as I described in my earlier blog post.  After I located the certificate in the Trusted Root section of the Local Machine store, I exported it to the desktop so that I could add it to my Azure Management certificates then simply dragged it to the
Personal section of the Current User store.

Once I added it to my Management Certificates through the Azure Management Portal, I ran the

csupload Set-Connection

command to specify the new certificate, then I was able to upload the new VHD with no other problems.

The moral to the story is – Make sure that the same certificate is being used for both the REST SSL port binding and the csupload connection.

I set out to deploy a Windows Azure VM node in Windows HPC Server 2008 R2.  I followed the Step-by-Step Guide in this TechNet article:

https://technet.microsoft.com/en-us/library/hh184325(WS.10).aspx 

Roadblock #1 - Could not import the VHD to HPC image store

Everything worked as expected until I got to Step 5 - Upload a VHD to Windows Azure.  Before uploading the file to Azure, the HPC process requires that the image be imported into the HPC image repository.  I started this process, but it never finished.  There were no error messages - it just ran forever.  My research revealed that the root cause of the problem was that there was not enough space on the HPC head node c: drive.  Since HPC requires that all the images that it manages reside on the c: drive, if there is not enough space, there is no work around right now. 

So, I gave up trying to use the HPC tools for the upload and turned instead to the Azure command line utility, csupload.  I found a link which describes How to Upload a VHD to Windows Azure using this utility:

https://msdn.microsoft.com/en-us/library/windowsazure/gg465385.aspx

I had already sucessfully configured the management certificate to enable HPC to provision Worker Roles on Azure by exporting the Default Microsoft HPC Azure Management certificate (which gets automatically added to the Trusted Root Certification Authorities store when the HPC Pack is installed) and adding it to my Azure subscription.  Consequently, when I read the note on step 3 that states, "The certificate that you use must be installed in the certificate store", I assumed that it already was. 

Roadblock #2 - Error with certificate

But when I tried to run the Set-Connection command, I received this error message:

Cannot access the certificate specified in the connection string. Verify that the certificate is installed and accessible.
The syntax for the connection string 'SubscriptionId=abcdef01-2345-6789-abcd-ef0123456789;CertificateThumbprint=abcdef0123456789abcdef0123456789abcdef01'
is not  valid.

This was very confusing since I had already used this combination of Subscription ID and Certificate in a node template to add Azure nodes to the HPC cluster with no problems.  After much research, I discovered a discussion forum where it was explained that the certificate had to be in the User Account as well as the Computer Account.  So, I opened up the MMC and added the Certificates snap-in, specified the Computer Account and navigated to the Trusted Root Certification Authorities store.  I did a right-click on the Default Microsoft HPC Azure Management certificate and selected All Tasks, then Export.  I saved the .cer file to the desktop.  I then closed the MMC without saving the settings, and reopened it again.  This time, when I added the Certificates snap-in, I specified My User Account.  I imported the .cer file from the desktop to both the Personal and the Trusted Root Certification Authorities stores.  I probably didn't need to do both, but I wanted to make sure I covered all the options.  Then, I retried the Set-Connection command and it ran with no errors.

Roadblock #3 - Cut and Paste from web site resulted in errors

For each of the steps in the upload process, the web site has code snippets which can be copied to the clip board for pasting into the command shell.  When I tried this on step 4, I received the following error message:

Too many arguments for command Add-VMImage expects 0 argument(s) found 6.
Found: '-LiteralPath', 'VMRole.vhd', '-Location', 'SouthCentral US', '-Name', 'VMrole.vhd'
Named parameter "-LiteralPath <string>" missing for command Add-VMImage.

This one was especially difficult to debug, because everything was specified exactly as it should have been.  I tried several times to modify the command with fewer parameters to see what the different error messages would say.  I finally decided to type in the command rather than paste it from the web site.  That fixed the problem.  The command worked and I was able to upload the VHD.

I hope these insights will help others to work around these roadblocks without killing as much time as I did.