Scenario

Two IaaS VMs will be configured as web servers and put in a Load Balanced set. Custom HTTP probe will be configured on a non-standard port. A custom ASP.Net MVC application will be deployed on each VM that will listen on the probe-port and simulate the process of monitoring the health of the VM. It will respond either with HTTP status 200 indicating good health, or with HTTP status 404, in case of errors etc. Finally, ACL will be specified on the public load-balanced endpoint to accept traffic only from designated remote address space.

Walkthrough

For the example scenario, the following steps were carried out.

Provision VMs in Cloud Service

Two IaaS VM’s named as vmlbcustprobe1 and vmlbcustprobe2 were provisioned in a Cloud Service named as cslbcustprobe.cloudapp.net. These used the Azure gallery images for Windows Server 2012 R2 Operating System.

Figure 1: IaaS VM’s provisioned in a Cloud Service

Create LB set with Custom probes

From the Azure portal UI, navigate to the page for Endpoints on the first VM (vmlbcustprobe1), and add a “Stand-alone endpoint” for HTTP on port 80. Also, select the option for Custom probes and supply the parameters as:

• Probe protocol: HTTP
• Probe path: “/”
• Probe port: “1001”. The portal UI will give a warning about port mismatch which can be ignored.
• Probe interval: 15 and Number of probes: 2

To complete the LB set configuration, navigate to the Endpoints screen on the second VM (vmlbcustprobe2) and select the option for “Add an endpoint to an Existing Load balanced set”. Once configured, the endpoint configuration screens should look as in Figure 2.

Figure 2: LB set with custom probes defined

Prepare individual VM’s for probe

The VM’s in this example will be used as members of a web farm. Also, the custom probe for the LB set is using HTTP. Both of these will use IIS. Connect to each VM using RDP and from Server Manager --> Role and Features, install IIS. Make sure the “Application Development” features are selected. See Figure 3.

Figure 3: Install IIS from Role and Features on each VM

Create a file with name “Default.htm” and drop it into the “c:\inetpub\wwwroot” folder of each VM. The contents of the Default.htm file will simply display the name of the VM that is being accessed; see Figure 4.

Figure 4: Default.htm on each VM displays the local hostname

In a real deployment, there will be full blown web application deployed on the Default web site (HTTP port 80)..

Create a custom application for Custom Probe

For the present example, the Probe protocol is HTTP and the Probe-port is 1001 – note that this is different from the port 80. The web site for the actual application will be deployed on the port 80, whereas port 1001 will be used for custom probing of VM for health check. I prefer to keep the two functionalities separate.

For the custom HTTP probing, an ASP.Net MVC application is used in the present example, as that is convenient, but strictly speaking one could have any custom process listening for HTTP requests on port 1001. Using Visual Studio in your development environment, create a new ASP.Net MVC project and select all default options.

Figure 5: Create a custom application for custom probing

Edit the code for HomeController to simulate some custom error checking. In the present case, one VM raises a 404 exception if the minutes in the current timestamp are even and the other VM raises exception if the minutes are odd; see Figure 6.

Figure 6: HomeController code modified to simulate custom health check

The code has been modified to force only one VM to return HTTP status 200 at a time, so that one can see the web farm responding from different VM’s at different times. The custom application, when tested in the development environment should show the default Welcome page when indicating a healthy VM, and should show 404 error message when simulating some issue; see Figure 7.

Figure 7: Custom health check application in normal and error states

Deploy custom probe application to VM 

The custom application for the present example is ready to be deployed to the VM’s in the web farm. The next set of steps outlines the process.

Copy the custom health check application

Copy the application binaries and other relevant files to some convenient folder on each VM; for the present example, the files were copied to “C:\web\MvcProbeApp”.

Create New web site for Custom probe

Using Remote Desktop connect to each IaaS VM in the web farm, and launch IIS Manager. Right-click the Sites node and select “Add Website” to create a new web site; see Figure 8.

Figure 8: Create a new Web Site for the custom probe application

The new site creation form will gather the configuration information as in Figure 9. Notice that port 1001 is specified to distinguish this site from the main web site which would be on port 80. The Physical path points to the location where the custom application binaries were copied.

Figure 9: Configuration information for Custom probe Web site

The site by itself can be tested from within the VM itself by browsing the Url https://localhost:1001/.

Open Probe-port in Firewall

The port 1001 chosen as the Probe-port for the present example needs to be opened in the firewall on each VM. This port is not part of the load balanced endpoint; rather it is used by the host agent internally on the private IP address. Opening this port does not pose any risks as it is not exposed through the Cloud Service public IP address.

The web farm should now be fully operational and accessible through Url https://cslbcustprobe.cloudapp.net . The way the custom probe has been defined, only one VM at a time will return a healthy status (200). Thus, the web site page should cycle between VM1 and VM2 on repeated hits.

Protect LB set Endpoint with ACL

The public endpoint exposed through the LB set can be restricted for access from a few designated address spaces, if necessary. It will depend upon whether the hosted web site is intended to be accessed by everyone on the internet, or only from a restricted pool of remote IP addresses. For the purpose of illustration, 175.1.0.3 was designated as the only remote IP address authorized to access the web site. ACL can be applied using Powershell (see the blog sites mentioned before) or using UI from the Azure portal. To use the Azure portal, navigate to any one of the VM instances in the LB set; for example vmlbcustprobe1 in the present example. Go to the page for Endpoints and after selecting the load balanced endpoint, click on “Manage ACL” at the bottom of the page. This will bring up the UI for specifying ACL as shown in Figure 10.

Figure 10: Specify ACL on LB set endpoint

Note that since this is an endpoint on a LB set, the ACL needs to be specified precisely once, on any one of the VM’s in the set. The endpoint on other VM’s will automatically reflect the ACL.

Both features – the Custom probes and the ACL’s – have been around for some time and are well documented officially as well as in blogs. My personal favorites are the blogs by Michael Washam at Creating Highly Available Workloads with Windows Azure and Network Access Control List Capability in Windows Azure Powershell. Despite this, I have found myself answering questions on the mechanics of using these features. The purpose of the present note is to have a fresh take at Custom probes and ACL’s using a walkthrough of a scenario, and revisit some of the questions that I have come across.

Assorted facts

Before showing a walkthrough, an assortment of questions that usually come up in discussions, are addressed in this section.

LB sets and Custom probes

• What exactly is the purpose of the Custom probes in LB sets? IaaS VM’s in a Cloud Service can in fact be added to a Load Balanced set with or without a custom probe. The Load Balanced endpoint however needs to know which VM’s within the set are active. This information is provided by an agent that runs on the host node which monitors the VM’s located there; see Figure 1. If there is no custom probe, the best that the agent can do is check if the VM’s are alive. With custom probes, the application developer gets a chance to implement some custom logic within the VM’s to respond to health check.

Figure 1: VM’s in an LB set with Custom probes for health check

• Custom probes are completely independent of the type of O/S (Windows or Linux) running within the VM.
• There are two types of probe protocols: TCP, and HTTP, though the HTTP is the more popular.
• The port designated for probe can be any port. It does not have to be port 80 for HTTP Probe-protocol. In fact, I prefer a port other than port 80 to keep things separate – the main website on port 80 and the custom probe on some other port. However, the Azure portal displays a “Port mismatch” warning if, while configuring the LB set, the probe port is chosen to be different from the port on the endpoint. The warning can be ignored.
• The custom logic can be implemented in any process listening on the designated Probe-port. It does not have to be IIS, although IIS is very convenient.
• Load Balanced sets with custom probes can be created in Azure portal through UI or alternatively using Powershell: Set-AzureLoadBalancedEndpoint. Also, see the blog post referenced above.
• The Load Balanced set discussed here is for VM’s in a single Cloud Service in a single datacenter. If HA/DR capability across datacenter is needed, consider Windows Azure Traffic Manager.

ACLs for Endpoints

The steps for configuring a Point-to-Site VPN are already well documented here. The official documentation should be read carefully as that is the definitive source of information. However, as well documented as it is, I like to see a walkthrough of the configuration steps with screenshots and actual addresses/values etc. as that is a very useful when one is playing with a new feature for the first time. I did not see anything relevant so decided to take screenshots of my own set up. Hopefully, folks will find it useful to see a full walkthrough.

Planning

There are three areas of planning needed before starting to configure the VNet, viz.

• Address space for the VNet and the subnets
• One or more DNS servers
• Address space on the client side

Address space for VNet

I chose 10.1.0.0/16 as the overall address space for the VNet from which to carve out the subnets. A small subnet dedicated to DNS was carved as 10.1.1.0/29 which provides three usable addresses 10.1.1.4 through 10.1.1.6 – to be discussed a bit more, below. Two subnets, 10.1.2.0/24 and 10.1.3.0/24 were carved out for locating my application related VM’s. Finally, a subnet for the VPN Gateway was again carved out as 10.1.4.0/29. These are shown in the screenshot, below. Note that the subnet for Gateway gets added automatically by the Portal UI when you click on the link labeled as “add gateway subnet”.

DNS Servers for VNet

I intend to deploy several Cloud Services in the VNet, and therefore need to deploy my own DNS server for name resolution across the Cloud Service boundary. There is an interesting conundrum though – I have not even created the VM that will eventually be configured as a DNS server; and yet, the Portal UI forces me to specify the DNS Server’s IP address right away while configuring the VNet itself. Actually, to be accurate, a recent change in the behavior of VNet now does permit specifying the DNS server later on when the VNet is already fully configured and is in use. But this new feature is still not available through the Azure Portal UI; you have to add the DNS server information using Powershell. Thus, for now, I continue to use a trick that is described, next.

Specify a subnet for DNS as 10.1.1.0/29 as shown in the screenshot, above. In theory, this subnet will have 32 – 29 = 3 bits to use toward the available addresses, which amounts to 2^3 = 8 possibilities. But, 000 and 111 are always reserved (for whole subnet and broadcast), and it is my understanding that Azure further reserves 001, 010, and 011. Thus, the addresses that are really available are those corresponding to 100, 101 and 110 (i.e. 10.1.1.4, 10.1.1.5, and 10.1.1.6). Eventually, when I do create a VM and add it to this subnet, it would be quite safe to assume that this VM will get assigned the IP address 10.1.1.4. Thus, having created a subnet for DNS like this, I can specify my DNS server during the VNet configuration even though the DNS server machine does not even exist, yet.

Address space on the Client side

In order to set up the VPN to this VNet, you have to specify the range of IP addresses that your client machines will be located in; i.e. the address space of your laptop, workstations etc. This address space must be a private address space and must not overlap with the address space of the VNet (in the present walkthrough the 10.1.0.0/16). At home, I have a router-plus-NAT combo device which also dispenses DHCP addresses in the range 192.168.1.0/24. Thus, this is the address space I picked for VPN clients.

Configure Virtual Network

At this point we have all the information needed to configure the VNet. In the portal UI, navigate to the section where new Network items can be created and choose “Custom Create.”

Specify the name of the VNet and the name of an Affinity group.

On the DNS Server and VPN Connectivity page, specify the DNS IP address as 10.1.1.4 (as discussed, above) and give it some suitable name. Also, select the checkbox for Point-to-Site VPN Connectivity.

Next, we need to specify the client side address space on the Point-to-Site Connectivity page. Use the 192.168.1.0/24 as discussed, above.

Next, on the Virtual Network Address space page, enter the Address space as 10.1.0.0/16 and then specify the various Subnets as planned, before. Finally, click on the link for “add gateway subnet” to add the Gateway subnet as 10.1.4.0/29.

Click through to the end and save the Virtual Network.

Create VPN Gateway

Click the link for the newly create VNet and navigate to the dashboard page. It should look like the following:

At the bottom of the screen, click on the Create gateway icon to start provisioning the Gateway VM’s in the Virtual Network.

This will take some time as the gateway VM’s are provisioned and configured. Once this step completes, the Gateway IP address will be displayed on the page and VPN connectivity page will look as below.

Create and upload Certificates

Certificates are used to authenticate the VPN clients. Read more about it at this link, and this link for Management certificates, in general. Using makecert utility, generate a Root certificate as shown, below. Specify a suitable name for the Root certificate.

makecert -sky exchange -r -n "CN=PRMgmtRootCert" -pe -a sha1 -len 2048 -ss My "PRMgmtRootCert.cer"

This command will create and then install a certificate in the Certificate store on your current machine. Also, a “.cer” file (without the private key) corresponding to this certificate will be put in the current file folder from where the makecert command was run. The “.cer” certificate file needs to be uploaded to the Virtual Network. It is also recommended that you export the Root certificate along with its private key (i.e. the “.pfx” file) from the Certificate store and keep it somewhere safe so that if necessary this root certificate can be installed on some other machine.

To upload the “.cer” file for the Root certificate just generated, click on the “Upload client certificate” link shown in the screenshot above and supply the path to the certificate. Once the Root certificate has been uploaded, the icons on the VNet dashboard will change to look as below:

Using the Root certificate, generate one or more client certificates and distribute to your team members. These client certificates will be used in the authentication process during the VPN session.

makecert.exe -n "CN=PRMgmtClientCert1" -pe -sky exchange -m 96 -ss My -in "PRMgmtRootCert" -is my -a sha1

Download and install the VPN package

The Virtual Network is now fully configured. In order to initiate a VPN session, you need to download and install a VPN package on your client machine. Depending upon whether your client machine is a 32 bit machine or 64 bit, pick the appropriate download link from the page shown above. An executable file will be downloaded which will be named after your VNet (in my case it was named as “vnetprvnt.exe”). Execute this VPN package on the client machine to configure the VPN settings on the local machine. Note that this VPN package is not an agent, even though it appears as if you are installing something on your local machine. Instead, the package is just a tool to configure the built-in RRAS features of your Windows Operating System.

Once the package is run successfully, your client machine is ready to VPN into the VNet. An icon should show up in the network area of your machine, as shown below for my Windows Server 2012 machine.

Click on the icon to start the connection; you may have to pick the client certificate that is chained to the Root certificate uploaded to VNet. Then, to verify that VPN connectivity is working, go to a command prompt and run ipconfig /all.

You now have a Point-to-Site VPN to Virtual Network working.

However, there are a few quirks to the way the capture process works and these are not very well documented. For example, when you capture an image from the Azure portal (as described in the documentation cited above), the user interface gives no option to specify where the image (i.e. the captured VHD file) will be located – which storage account? Which container? In fact, it automatically stores the new image file in the same storage account where the original VM used to be, and the blob container is automatically chosen to be “vhds”. Further, while we do get an option to provide a name for the image – this is the name with which the image will appear in the gallery – there is no option given for naming the vhd file, which gets assigned an auto-generated unique but nonsensical name.

There is an additional issue with the image captured as discussed above. It turns out that any new VM provisioned from that image will always have its vhd stored in the same storage account as that which contains the image; i.e. one cannot provision a VM in a different storage account. This is quite significant and if one needs to put the new VM in a different storage account, then the only way of doing so is to first create an image in this other storage account by copying the previously created image from the previous storage account.

The copying of image from one storage account to another is largely a matter of copying the underlying blob, but it is not sufficient. The newly copied VHD, which is actually just a blob, needs to be published as an image, otherwise attempts to provision VM from the copied blob/vhd would give errors. This point unfortunately is also not documented very well anywhere till you stumble upon it through trial-and-error. The purpose of this blog post is to show the steps through a Powershell script for copying and publishing a new image in a storage account different from that where the image was originally created. Further, once a decision has been made to copy the image to a different storage account, then it actually works to our advantage in removing some of the restrictions faced during the initial capture of the image; i.e. better naming and ability to specify the container where it will be stored etc. During the copy process, using Powershell, one is able to specify the container where the blob will be copied as well as one gets an opportunity to name the VHD file according to the naming conventions of one’s group. Finally, we are able to publish that image with a suitable name. The script below provides a walkthrough of the steps described above and also shows how to test the newly created image by provisioning a VM out of it.

The Powershell script is attached for ease of use; see attachment.

publish-vm-image.ps1

The trick to automating the swap space configuration at the time of VM provisioning is to use the Windows Azure Linux Agent (waagent). Most people are somewhat vaguely aware that there is an agent running in the Linux VM, but most people also find it a bit too obscure and ignore it, even though the Azure portal actually has a nice documentation on waagent, here. There is one other point that needs to be mentioned before drilling down into the details of waagent and how it may be used for the task at hand. That point is that this approach works well if you have a customized Linux VM of your own and are exporting it as a reusable image for provisioning of Linux VM’s in future. There is no way of tapping into the capabilities of the waagent when using a raw base image of Linux from the Azure gallery. This is not really a limitation since the use case scenario that I find most useful is one where I start out with a VM that is provisioned using a gallery image, and is then customized for things I like to have; for example, I like to have Standard Java instead of the open-jdk Java; or I may want to install Hadoop binaries on the VM so that the image can then be used later on toward a multi-node cluster. In such a scenario, it is just as easy to configure the waagent to do a few of the additional things that I want done automatically through the provisioning process.

As discussed in the Windows Azure Linux Agent user’s guide, the agent can be configured to do many things, among which are:

• Resource disk management
• Formatting and mounting the resource disk
• Configuring swap space

The waagent is already installed in the VM provisioned from a gallery image and one needs to simply edit its configuration file located at “/etc/waagent.conf” where the configuration looks like the following:

Change the two lines shown above in the configuration file to enable swap, by setting as follows:

• Set ResourceDisk.EnableSwap=y
• Set ResourceDisk.SwapSizeMB=5120

The overall process, therefore, is the following:

• Provision a Linux VM in IaaS as usual using one of the images in the gallery.
• Customize the VM to your liking by installing or removing software components that you need.
• Edit the “/etc/waagent.conf” file to set the swap related lines, as shown above. Adjust the size of the swap file (the above is setting it to 5 GB).
• Capture a reusable image of the VM using instructions described here.
• Provision new Linux VM’s using the image just exported. These VM’s will have the swap space automatically enabled.

While we are on the subject of Windows Azure Linux Agent, it turns out that it provides yet another interesting capability – that of executing an arbitrary, user-specified script through the Role.StateConsumer property in the same configuration file “/etc/waagent.conf”. For example, one can create a shell script “do-cfg.sh” as follows:

Then, in the configuration file set Role.StateConsumer=/home/scripts/do-cfg.sh or whatever is the path to your script. The waagent execute the script just before sending a “Ready” signal to Azure fabric when provisioning a VM. It passes a command-line argument “Ready” to the custom script which can be tested within the script, as shown above to do some custom initialization. Likewise, the waagent executes the same script at the time of VM shutdown and passes a command-line argument “Shutdown” to the script which can be tested for and some custom cleanup task can be run in the VM.

It is not difficult to guess that this had something to do with the limited resources of a medium sized VM – after all it has only 2 CPU cores and 3.5 GB of memory. But, while I would have expected the workloads to run slow, I was not expecting the whole VM to become unresponsive or start dropping connections. After discussing this issue with my friends and colleagues, we determined that the VM did not have Swap (i.e. what is called a page file on Windows) configured at all. Thus, its virtual memory system could not swap to disk when the memory pressure increased.

You can check how the system is doing memory wise by running the “free” command from the Linux shell prompt, and in particular, you can use “cat /proc/swaps” to see the state of the swap space – how much is configured and how much is in use. See the screenshot, below.

If the swap space is not configured at all, as is the default case with Linux VM’s provisioned in Azure IaaS, the “cat /proc/swaps” will return nothing, and likewise the “free” command will not show any activity in swap.

An interesting question is why doesn’t the VM provisioning using a Linux library image (i.e. from the Azure gallery) automatically configure swap space. Perhaps the thinking is that the user should decide on the size and location of the swap and do it post provisioning. However, this information is not documented anywhere, and it is quite possible that one continues to use the VM without the swap ever getting configured till processes begin to crash or the VM freezes up.

That said, once the problem has been diagnosed, the solution is a simple set steps to configure a file based swap on the resource disk; a medium sized IaaS VM in Azure comes with 135 GB of resource disk mounted as “/mnt/resource”. Given below is a walkthrough of the steps for configuring a file based swap space on the VM.

• Use the “fallocate” command to allocate a swap file of suitable size, say, 5GB on the resource disk. The syntax is: “fallocate -l 5g /mnt/resource/swap5g” where “swap5g” is the name of the file
• Change the permissions on the file using “chmod” command so that only the root user has read/write permissions on the swap file. The syntax is: “chmod 600 /mnt/resource/swap5g”
• Use the “mkswap” command to set up the file as swap area. The syntax is: “mkswap /mnt/resource/swap5g”
• Enable the use of the swap file using “swapon” command. The syntax is: “swapon /mnt/resource/swap5g”
• The swap is ready for use now, and the “cat /proc/swaps” command should confirm it now. Add an entry to the “/etc/fstab” file so that even if the VM recycles in Azure, the swap settings are retained. The syntax is: echo “/mnt/resource/swap5g   none  swap  sw  0 0” >> /etc/fstab

Here is a transcript of the above commands executed in my VM.

Acknowledgement: Thanks to my colleague Amit Srivastava for help with troubleshooting and resolving the swap issue.