Michiel van Otegem’s Blog

Measuring Latency and Throughput Between Azure Regions

Michiel van Otegem - MSFT — Mon, 20 Nov 2017 13:51:49 GMT

If you want to implement disaster recovery, latency and throughput between your primary and secondary location are key factors. These determine how quickly and how much data you can transport to the secondary location, and hence how much data will be lost in case of failure of the primary location. More formally, the latency and throughput determine the possible recovery point objective. In this post I will walk through measuring latency and throughput between Azure regions, so you can determine the best configuration for your scenario. Latency is determined by the speed of light and the number of infrastructure components between the source and destination. This means that the closer the primary and secondary location are, the lower the latency. The catch is that if the primary and secondary location are too close together, they could potentially be hit by the same (natural) disaster. When Microsoft picks locations for Azure regions, they are selected to be a safe distance apart for most major disasters (earthquake, flood, and so on).

Options for connecting virtual networks in different regions

In Azure there are three ways to connect virtual networks in different regions:

In this post I will focus on VPN and virtual network peering, but you could use the same method to measure latency and throughput in a configuration with ExpressRoute. Before I go into how you measure latency and throughput, let’s look at the architectural differences between the three options.

VPN

With VPN, you link two virtual networks together by deploying a VPN gateway in both virtual networks, and then set up a connection between the two, as shown below. Diagram 1: Virtual networks VNET1 and VNET2 in different Azure regions connected via VPN

Cross-region virtual network peering

The downside of using a VPN is the need for VPN gateways, which incur cost, add latency, and limit bandwidth. Within a region, you can use virtual network peering instead, which connects virtual networks without a gateway. Cross-region virtual network peering recently went into preview, and this allows you to securely connect virtual networks across regions, effectively stretching your network across regions in Azure, as shown below. Diagram 2: Virtual networks VNET1 and VNET2 in different Azure regions connected via virtual network peering (preview)

Express Route

ExpressRoute is a dedicated private connection from your on-premises location to Azure (and Office 365) data centers. You can link virtual networks to ExpressRoute, and with the ExpressRoute Premium feature you can link virtual networks from multiple regions, as shown below. Diagram 3: Virtual networks VNET1 and VNET2 in different Azure regions connected via ExpressRoute When you connect two virtual networks in different Azure regions with VPN or virtual network peering, the traffic between the two regions runs over the Microsoft backbone, and the traffic is routed as any normal IP-based traffic. Although we currently don’t offer an SLA on latency and throughput, the fact that the traffic stays within our backbone means latency and throughput typically stay within a certain range. When using ExpressRoute (with the Premium Add On) this is mostly the same, but there is a subtle difference: the virtual networks aren’t connected directly. Each virtual network is connected to the same Microsoft Edge your Partner Edge is connected to. In most cases that won’t make a lot of difference, but I’ll illustrate the impact through an extreme case. Suppose your on-premises location in Australia is connected via ExpressRoute, and terminates at a Microsoft Edge in Australia. If you connect virtual networks in West Europe and North Europe to this ExpressRoute, the traffic between the virtual networks will now run over the Microsoft Edge in Australia.

Creating two virtual networks connected by VPN

Creating two virtual networks in different regions that are connected through VPN is quite easy using this ARM template from the Azure QuickStarts. You can download my parameter file here, so you can see what kind of values you need to fill in. I used IP ranges for the virtual networks as shown in diagram 1. Note that I used IP address ranges that don’t overlap. Also, the VPN VIPs will be different, as these are automatically assigned. Deploying the template will take a while, mostly due to the VPN gateway configuration. You can continue with the next steps in the setup once the virtual networks have been created. You don’t have to wait for the VPN gateway configuration to finish.

Adding virtual machines

To measure latency and throughput, you need a virtual machine in both virtual networks. For the tests I’m using Windows Server 2016 on a D3_v2 virtual machine. A D3_v2 virtual machne has 4 CPUs, which may seem like overkill. However, the size of your virtual machine impacts the network bandwidth assigned to the virtual machine. By choosing a larger virtual machine I’m making sure I have some bandwidth to play with. Of course, if you are targeting a certain size machine for your workload, it makes sense to use that size to test with. Note: you could select accelerated networking when deploying the virtual machine, but that feature mainly impacts networking within a single region. Once the virtual machines are created, each has an internal IP address. If you used the subnet configuration I used, and deployed in Subnet-1 in each region, the internal IP addresses are 10.10.1.4 for the West Central US virtual network and 10.20.1.4 for the US West 2 virtual network. Go to the first virtual machine in the portal, and connect to it with remote desktop protocol (RDP). Once the VPN gateway configuration is finished, you can test connectivity by using RDP to connect from the first virtual machine to the second virtual machine on the internal IP address.

Press Ctrl+R.
Enter mstsc.
In the RDP dialog that pops up, enter the IP-address of the second virtual machine (in my case 10.20.1.4).
Click Connect.
Click Confirm.
You should now be logged into the second virtual machine. If this doesn’t work, there may be a configuration error in the connection between the virtual networks, but be sure to check if you have the right IP-address, and try connecting from the portal to the second virtual machine like you did with the first virtual machine.

Measuring latency and throughput

The easiest way to measure latency and throughput is using PsPing. You can download PsPing from here. PsPing can be used for simple ping functionality without a server, but for latency and bandwidth tests you need to setup a server.

Start the PsPing server

Open or switch to the remote desktop connection to one of the virtual machines you created.
Open a command prompt and change directory to the location where you extracted PsPing.
Type psping -f -s :, for example, psping -f -s 10.20.1.4:81 to listen on port 81 on a virtual machine that has a local IP address of 10.20.1.4.

Run the PsPing client

Open or switch to the remote desktop connection to the other virtual machine.
Open a command prompt and change directory to the location where you extracted PsPing.
Type psping -l -n -f :, for example, psping -l 8k -n 1000 -f 10.20.1.4:81 to send 1000 messages of 8 kilobytes to port 81 on the virtual machine with IP address 10.20.1.4.

This will get you a measurement over VPN, but for reference we also want a measurement over the public IP address, bypassing the VPN. To enable this, you need to modify the network security group on the virtual machine running the server, so it doesn’t block traffic over the port PsPing is listening to (port 81 in my case). It also makes sense to use different size blocks, to see how that affects the latency. To test the bandwidth, you can use the -b parameter with PsPing, as follows: Psping -b -l -n -f :, for example, psping -l 8k -n 1000 -f 10.20.1.4:81 to send 1000 messages of 8 kilobytes.

Public IP vs. VPN gateway

The table below shows the differences between going over the public IP address versus going through the VPN gateway.

	Public IP	VPN
10000 x 8k
Avg	21.48 ms	23.47 ms
Min	21.15 ms	21.93 ms
Max	31.62 ms	148.56 ms
1000x 1000k
Avg	23.70 ms	38.31 ms
Min	23.15 ms	30.07 ms
Max	45.54 ms	80.20 ms

Disclaimer: Measurement results will vary between different Azure regions, time of day, and other factors. The maximum value, in particular, varies greatly from test to test. Please test thoroughly to understand the impact on your scenario. With the small packet size the difference is only a few milliseconds on average, but the maximum latency may vary. The increased latency with a VPN gateway is caused by the additional hop of going over the gateway. The difference between connecting over the public IP address and through the VPN gateway becomes significant with larger requests. This is due to the bandwidth limitations of the VPN gateway. I used the Basic SKU, which has the least bandwidth and using a more powerful SKU (see About VPN Gateway) will likely have positive impact. Note: For maximum throughput and minimum latency you can you can turn of encryption of the traffic between two virtual networks connected using VPN gateways. Do not use this unless you understand the security implications.

Measuring with cross-region virtual network peering

With the tests over the public IP and through the VPN done, it’s time to move on to cross-region peering. You could setup two entirely new virtual networks and virtual machines, but you can also remove the connection between the VPN gateways, and then setup virtual network peering. You don’t have to delete the VPN gateways, but you can. To remove the connections, take the following steps:

Open the Azure portal.
Open the resource group you created for the environment.
Click the USWC-USW2 connection.
Click Delete.
Close the blade.
Repeat steps 3-5 for the USW2-USWC connection.

To configure virtual network peering, take the following steps:

Open the Azure portal.
Open the resource group you created for the environment.
Click the USWC-VNET connection.
Click Peerings.
Click Add.
Create a peering to the USW2-VNET, as shown below. Note that if you don’t see the virtual network in the West US 2 region, virtual network peering is not enabled yet for your subscription. You can register for the public preview here.
Provide a name and select the virtual network to peer to, the click OK.
Repeat steps 2 through 7 to create a connection from the USW2-VNET back to the USWC-VNET.

Public IP vs. cross-region virtual network peering

The table below shows the differences between going over the public IP address versus going through virtual network peering.

	Public IP	Virtual network peering
10000 x 8k
Avg	21.48 ms	21.29 ms
Min	21.15 ms	21.03 ms
Max	31.62 ms	27.68 ms
1000x 1000k
Avg	23.70 ms	23.55 ms
Min	23.15 ms	22.81 ms
Max	45.54 ms	45.40 ms

The difference between public IP and cross-region virtual network peering is only a few microseconds, and as such negligible. This makes sense, as the traffic pretty much takes the same physical route through the Microsoft backbone.

Conclusion

Using PsPing you can measure the latency and bandwidth characteristics between different regions. This will help you determine the best options for scenarios such as disaster recovery, and how you should configure synchronization between systems in different regions. Cross-region virtual network peering is a great addition to Azure networking. It provides lower latency and higher throughput than connections over VPN gateway. Once it becomes generally available, it will be the preferred method to peer networks in different regions. --- Thanks to Steffen Vorein and RoAnn Corbisier for editing. ---

Converting XML to JSON with Azure Logic Apps

Michiel van Otegem - MSFT — Thu, 06 Jul 2017 13:11:29 GMT

With Logic Apps you can quickly build small workflows to perform common tasks. An interesting feature of Logic Apps is its ability to run serverless, so you only pay for the actual compute power needed to execute your workflow. This feature is perfect for what I set out to do: convert XML files to JSON in bulk.

My Test Setup

For testing purposes I used a OneDrive folder as source, and another OneDrive folder as destination. The LogicApps trigger for OneDrive executes the workflow anytime a new file is added to the folder. Technically it works on a interval that you configure, for example every minute, at which point it scans for all the new files since the last time the trigger executed. The view below shows the general idea.

Converting to JSON

As you can see in the image above, the content of the new file is created using the json function available in Logic Apps. Based on the documentation in Workflow Definition Language schema for Azure Logics Apps, this function accepts string or XML input. In my first attempt I used the following function: json(triggerBody()), where triggerBody() is shorthand for trigger().outputs.body, which represents the file contents. To my surprise this yielded the following error:

InvalidTemplate. Unable to process template language expressions in action 'Create_file' inputs at line '1'
and column '2295': 'The template language function 'json' parameter is not valid. The provided value
'    ' cannot be
parsed: 'Unexpected character encountered while parsing value: <. Path '', line 0, position 0.'. Please see
https://aka.ms/logicexpressions#json for usage details.'.

Apparently the json function is stumbling over the '<' character (the red character in the error message), even though the documentation says it supports XML input. My first assumption was that this had something to do with an invisible character, so I tried different editing tools to create the XML, but that didn't help. Jeff Hollan on the Logic Apps team helped me understand the actual problem is the way Logic Apps interprets the file content of a file on OneDrive. Hence, Logic Apps doesn't understand you are feeding XML to the json function unless you explicitly tell it what it's dealing with. To do so, you need to use the xml function to cast the input to XML. The correct expression therefore is

json(xml(triggerBody()))

Using a different source/destination

My test setup was with OneDrive, but the expression will work with any input/trigger that doesn't stumble over XML content. You could just as easily use blob, queue, or table storage, or another method such as email, FTP, or Azure Service Bus.

Separating environments in Azure Data Catalog

Michiel van Otegem - MSFT — Mon, 24 Oct 2016 10:50:34 GMT

Azure Data Catalog is a services that enables discovery of data assets in your organization. Users can search for data sources entered into the catalog in various ways, so they can (re)use the data sources, instead of creating duplicate and potentially inconsistent sources for the same data. As explained in the Azure Data Catalog developer concepts, you can only have one Data Catalog per Azure account. If an Azure account contains multiple subscriptions, the same Data Catalog applies to all subscriptions. This means that if you use subscriptions to separate environments (e.g. dev/test/production or finance/marketing/manufacturing), then these environments still share the same Data Catalog. From the perspective of a Data Catalog that holds information on all data sources in your organization, this makes sense, but it also raises a question: how do you separate data sources from different environments? Let's look at an example situation where we have three subscriptions under the same account: dev, test, and production. As services are being developed, the same data source may exist in dev, test, and production with slight variations because of version differences. You need to ensure that a developer doesn't use wrong (version of the) data source, and inadvertently writes development data into a production system. It is important to realize that in itself this is not a discovery problem, it's a rights management problem. If the rights to a production system allow a developer to manipulate production data, this issue exists regardless of whether you have a discovery mechanism in place or not. Separating information in the Data Catalog will not solve this issue. However, making clear in the Data Catalog whether a particular data source is for dev, test, or production can help a little.

How NOT to use Data Catalog

I've added two assets to the Data Catalog, both named Contacts. When searching for these, you get a result like this:

If you look at the details of both assets, you can see they are almost similar, except for the location: For someone that is not knowledgeable about your data sources and locations, the above doesn't help. In this case you can still infer whether it's a development asset, but with more usual data source names, this is less likely.

Data Catalog Best Practices

To avoid situations like the one shown above, you should follow the guidelines below.

Use a consistent naming scheme, for example contacts.dev and contacts.prod. The friendly name can be the same or something like Contacts (Development) and Contacts (Production).
Use tags to identify the environment the asset belongs to.
If naming convention and tags are not sufficient, limit visibility to different user groups, for example only make development assets visible to people in the developers group. You can do this as follows:
1. In Azure Active Directory create the needed groups and add users.
2. Select an asset and Take Ownership*.
3. Set Visibility to Owners & These Users.
4. Click Add... to add the group(s) you want to grant visibility.

* The guidance in How to Manage Data Assets is to have at least two individuals as owner of an asset.

Hello Azure!

Michiel van Otegem - MSFT — Wed, 07 Sep 2016 13:47:43 GMT

Welcome to my new MSDN blog. I joined Microsoft a while back and from this point forward I'll used this space to blog about Microsoft technology in general and Azure specifically. My personal blog http://michiel.vanotegem.nl won't go away, but you'll likely find references to this blog there, and maybe some other non-Microsoft related stuff. So what about the name of my blog? 20 years ago I started my own little company called IT Composer. When dealing with customers and when speaking at customers I often got the question "IT Composer, is that your company name or is that your job title?" The answer was: both. At the time the term (Software) Architect wasn't widely used yet, and even though we've got processes around IT that are very similar to manufacturing, I still think IT is a creative process. IMHO the term "composer" better reflects this than "architect" (although I don't want to imply architects are not creative!). Even though my company no longer exists, I still identify with the term IT Composer. In fact, working with Azure I've found that composing your solution with the right Azure services is extremely important if you want to get the most benefits from Cloud computing and the Azure platform. Hence I've decided to stick with the IT Composer name, but with a twist because of my Azure focus.