Rumors about Azure, SQL Server & Data in the Cloud.... https://docs.microsoft.com/archive/blogs/igorpag/feed.xml Tue, 13 Nov 2018 18:17:34 GMT en-US hourly 1 Azure Availability Zones Quick Tour and Guide https://docs.microsoft.com/archive/blogs/igorpag/azure-availability-zones-quick-tour-and-guide Thu, 03 May 2018 09:07:26 GMT https://blogs.msdn.microsoft.com/igorpag/?p=2125 Where Availability Zones are sitting? For you that are not aware of what AZ is, let me include a good single sentence definition: Availability Zones (AZ) are unique physical locations with independent power, network, and cooling. Each Availability Zone is comprised of one or more datacenters and houses infrastructure to support highly available, mission critical applications. AZ are tolerant to datacenter failures through redundancy and logical isolation of services. I already authored an article introducing this feature, then you may want to read it before going forward with the current one:

Why Azure Availability Zones

https://blogs.msdn.microsoft.com/igorpag/2017/10/08/why-azure-availability-zones

Azure today is available in 50 regions worldwide, as you can see in the map below. In the same picture (live link here) you can see both announced and already available regions. You can also find out in which regions AZ feature is already available (France Central and Central US, at the time of writing this article), and where is currently in preview (East US 2, West Europe, Southeast Asia), reaching this page. But in the general Azure hierarchy, where AZ is exactly sitting? Using the map above and the page from where I pulled it down, we can describe the following hierarchy:
  • Geographies: Azure regions are organized into 4 geographies actually (Americas, Europe, Asia Pacific, Middle East and Africa). An Azure geography ensures that data residency, sovereignty, compliance, and resiliency requirements are honored within geographical boundaries.
  • Regions: A region is a set of datacenters deployed within a latency-defined perimeter and connected through a dedicated regional low-latency network. Inside each geography, there are at least two regions to provide geo disaster recovery inside the same geopolitical boundary (Brazil South is an exception).
  • Availability Zones: AZ are physically separate locations within an Azure region. Each Availability Zone is made up of one or more datacenters equipped with independent power, cooling, and networking. For each region enabled for AZ, there are three Availability Zones.
  • Datacenters: physical buildings where compute, network and storage resources are hosted, and service operated. Inside each AZ, where present, there is at least one Azure datacenter.
  • Clusters: group of racks used to organize compute, storage and network resources inside each datacenter.
  • Racks: physical steel and electronic framework that is designed to house servers, networking devices, cables. Each rack includes several physical blade servers.
I often heard this nice question from my customers and partners: why Azure provides three zones? Should be enough two zones instead? Reason is pretty simple: for any kind of software or service that requires a quorum mechanism, for example SQL Server or Cassandra, two zones are not enough to always guarantee a majority when a single one is down. Think about three nodes: if you have three zones, you can deploy one node in each zone, and still have majority if a single zone/node is down. But what happens if you have only two zones? You will have to place two nodes into the same zone, and this specific zone will be down, you will lose majority and your service will stop. A nice example for SQL Server, deployed using AlwaysOn Availability Group across AZ, is reported below:

SQL Server 2016 AlwaysOn with Managed Disks in Availability Zones

https://azure.microsoft.com/en-us/resources/templates/sql-alwayson-md-mult-subnets-zones

Why should I use Availability Zones?

AZ has been built to make Azure resiliency strategy better and more comprehensive. Let me re-use a perfect summarizing sentence from this blog post:

Azure now offers the most comprehensive resiliency strategy in the industry from mitigating rack level failures with availability sets to protecting against large scale events with failover to separate regions. Within a region, Availability Zones increase fault tolerance with physically separated locations, each with independent power, network, and cooling....  

While regions are 100s of miles distant and provide protection from wide natural disasters, AZs are much closer and then not suitable for protecting from this kind of events. If you want to build your geo-DR story, you should still use different Azure regions. On the other hand, this relative proximity should permit you to replicate synchronously (RPO = 0) your data across different physical and isolated, independent locations, protecting your application or service from datacenter wide failures. With AZ, what Azure can provide now in terms of High-Availability SLA, is slightly changed, let me recap the full spectrum using the graphic below: As you can see, now for your VMs you can have 99.99% HA SLA, compared to the lower 99.95% provided by the traditional Availability Set (AS) mechanism. AS is still supported, but cannot be used in conjunction with AZ, you have to decide which option you want to use: at VM creation time, you need to place your VM into an AZ or into an AS, and you will not allowed to move it later, at least as the technology works today. For an overview of mechanisms that you can use to manage availability of your VMs, you can read the article below:

Manage the availability of Windows virtual machines in Azure

https://docs.microsoft.com/en-us/azure/virtual-machines/windows/manage-availability

If you decide to use AZ, you will *not* formally have the concepts of Update Domains (UDs) and Fault Domains (FDs). In this case, AZ will be now your UDs an FDs, remember that these are different and independent Azure locations, from a physical and logical perspective. If you try to put your VM into both AS and AZ, you will get an error. You can realize it looking into the PowerShell cmdlet "New-AzureRmVMConfig": as you can see below, it is here that you can specify values for either "-Zone" or "-AvailabilitySetId" parameters, but not both at the same time. You will not get an error for this cmdlet execution, but later when you will try to instantiate a VM object with “New-AzureRmVM”: In the print screen above you can read the error message below:

Virtual Machine cannot be created because both Availability Zone and Availability Set were specified. Deploying an Availability Set to an Availability Zone isn’t supported.

 

Availability Zones are only for VM?

Short answer is “NO”, long answer is starting now. As I stated at the beginning, in this article I’m mainly talking about the Virtual Machine (IaaS) context but let me shortly describe where AZ is also expanding its coverage. I would be not surprised at all to learn about upcoming new Azure resources and services. This is only my opinion, but AZ is a great mechanism to improve resiliency of distributed services, and I can imagine Azure will make more of them AZ enabled.

Azure SQL Database (SQLDB)

It is still in preview, even if AZ is generally available in some regions, main article is reported below and the section interesting for us is “Zone redundant configuration”:

High-availability and Azure SQL Database

https://docs.microsoft.com/en-us/azure/sql-database/sql-database-high-availability

Important thing to keep in mind is that SQLDB with AZ is only available with the “Premium” tier. Why this? If you are interested in the details, I would recommend you read the article above. If not, let me shortly recap here: only with “Premium” SQLDB tier you have three different active SQLDB instances, deployed across all the 3 zones, database storage is local ephemeral SSD storage, and synchronously replicated. HA SLA is the same as without AZ, that is 99.99%, with RPO = 0, RTO = 30s. For automatic backups, ZRS storage is used. NOTE: You don’t have control on how many and which zones your SQLDB nodes will be placed.

Virtual Machine Scale Set (VMSS)

VMSS has nice and deep relation with AZ, let me explain why. First, with VMSS you can use AZ and at the same time you can still use Availability Set (AS). Once again, this is *not* possible with standard Azure VMs.

Create a virtual machine scale set that uses Availability Zones

https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-use-availability-zones

As explained in the article above, you have the option to deploy VMSS with "Max Spreading" or "Static 5 fault domain spreading". With first option, the scale set spreads your VMs across as many fault domains as possible within each zone. This spreading could be across greater or fewer than five fault domains per zone. With the second option, the scale set spreads your VMs across exactly five fault domains per zone. Another cool VMSS feature that can be used in conjunction with AZ is “Placement Group”. A placement group is a construct similar to an Azure AS, with its own fault domains and upgrade domains. By default, a scale set consists of a single placement group with a maximum size of 100 VMs. If a scale set is composed of a single placement group, it has a range of 0-100 VMs only, with multiple you can arrive to 1000. Last evidence of good integration between VMSS and AZ is the “load balancing” options you have. For VMSS deployed across multiple zones, you also have the option of choosing "Best effort zone balance" or "Strict zone balance". A scale set is considered "balanced" if the number of VMs in each zone is within one of the number of VMs in all other zones for the scale set. With best-effort zone balance, the scale set attempts to scale in and out while maintaining balance.

Where am I placing my resources?

I decided to group here a series of questions related to AZ selection and placement. I felt some confusion here from customers and partners, then I hope that what I’m writing will be useful.
  • Usage of AZ is mandatory? NO, you can continue deploying your VMs as usual. Then, where will my resource will be deployed? You should not be worried about this, you will only indicate the “region” in your request, and Azure will place your VM in a datacenter inside that region. NOTE: AZ related parameters in ARM templates, PowerShell cmdlets and AZ CLI are all optional.
  • Is there a default AZ if not specified? NO, apparently there is no way to set a subscription wide default AZ. If you want to use it, you must specify which one.
  • Can I specify the physical AZ? NO, you can specify only logical AZ as “1”, “2” or “3”. How logical is translated to physical zones, and datacenters, is managed automatically by Azure platform.
  • Is logical to physical mapping constant across Azure subscription? NO, I didn’t find any official statement or SLA about providing same mapping for different subscriptions even under the same Azure Active Directory tenant.
  • Which is the distance (and then the latency) between AZ? Officially, there is no indication of distance between different AZs, nor network latency SLA, only a general indication of “less than 2ms” boundary inside the same region.
  • Is it possible to check in which AZ my VM is deployed? YES, you can check in which zone your VM is deployed looking into the VM properties using the Azure Portal, PowerShell and Azure Instance Metadata Service (see JSON fragment example below, more details later in this article).

What I need to create my VM into Availability Zone?

If you want to create and access a VM in Azure, you would normally need to define first dependent “core” resources: a public IP (VIP), maybe a Load Balancer (LB) tied to this public IP, a network interface card object (NIC), a Virtual Network and subnet (VNET, Subnet) and a disk (Managed Disk). This remains the same also with AZ. In addition to these resources, you may also need an Image to deploy your custom OS, and a Snapshot taken from your VM disk for backup/recovery purposes.  With the advent of AZ, Azure team applied several underlying changes to all these core resources to make possible and efficient running Azure VMs, and other services, into AZ. Before shortly describing each of them, let me clarify a new terminology that you may find around in Azure documentation: with “zone redundant” is intended a resource that is deployed across at least two AZs and then able to survive the loss of a single zone. Instead, with “zonal” is intended a resource that is deployed into a single and specific AZ, and will not survive to zone loss. Keep in mind that this distinction is only for resources that are “AZ-aware”. There are resources that are not deployed, or deployable, into Availability Zone. For example, an Azure VM per se is a “zonal” resource because is a monolithic object that can be deployed only into a single AZ. But can be also created without mentioning AZ at all, as you have always done before this feature. Instead, Azure Load Balancer can be created either as “zonal” resource or “zone redundant”.  

Load Balancer

Azure recently introduced a new type/SKU of load balancer (LB) called Azure Standard Load Balancer that has lots of new nice features, one of these is the support for Availability Zones (AZ). Pay attention to what I have just written: I said “support” because a LB resource itself is regional. Imagine it as zone-redundant already, even without leveraging AZ capability. What is correct to say is that LB can support zonal or zone-redundant scenarios, depending on your needs and configuration. Both public and internal LB supports both scenarios, with internal and external frontend IP configurations, can direct traffic across zones as needed (cross-zone load balancing). These two scenarios are reported below as examples:

Create a public Load Balancer Standard with zonal frontend using Azure CLI

https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-standard-public-zonal-cli

 

Load balance VMs across all availability zones using Azure CLI

https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-standard-public-zone-redundant-cli

 

Public IP

Along with the Load Balancer described above, also for Public IP (VIP) we have a Standard SKU. If you want/need to work with a Standard LB for AZ, you also need to use a Standard Public IP, you cannot mix the SKUs here. With Standard LB and VIP you can now enable zone redundancy on your public and internal frontends using a single IP address, or tie their frontend IP addresses to a specific zone. This type of cross-zone load balancing can address any VM or VMSS in a region. A zone-redundant IP address is served (advertised) by the load balancer in every zone, since the data path is anycast within the region. In the unlikely chance that a zone goes down completely, the load balancer is able to serve the traffic from instances in another zone, very quickly.

Virtual Network, Subnet, NIC, NSG and UDR

You may be surprised if you are not familiar with Azure, and coming from other Cloud providers with AZ feature available. Azure VNET and related subnets are regional by design, and do not require any AZ specification when creating them. Additionally, there is no requirement on subnet placing: both VNET and subnet can include IP addresses for resources across AZs. Azure supports region wide VNETs since 2014 as originally announced here. Based on this, there is no specific impact on definition and behavior for Network Security Groups (NSGs) and User Defined Routes (UDRs) when AZ design is applied. Same for the Network Interface Card (NIC) object.

Managed Disk & Storage

A VM needs at least an OS disk to boot from, additional data disks are possible but not required. There are several types of disks available in Azure, but if you want to attach to a “zonal” VM, you need to use the Managed Disk (MD) type, either Premium or Standard, you cannot use the unmanaged legacy version. If using the Azure Portal, the “Storage” option will be immediately grayed out as soon as you will select a zone for your VM placement. Once again, Availability Zones feature requires Managed Disks for VMs.   Please remember that MD feature only supports Locally Redundant Storage (LRS) as the replication option. When you create a VM from the Azure Portal, or for example using PowerShell, a MD for OS will be implicitly created and co-located in the same AZ chosen for the placement of compute resources. MD can be also created as standalone object, syntax for creation commands/scripts has been augmented to allow AZ specification. MD is always a “zonal” object and cannot be directly moved/migrated. Be aware that you cannot attach a MD created in AZ(1) to a VM created in AZ(2).

Images & Snapshots

Images and Snapshots (for Managed Disks) are not strictly required, but you may need to use these objects. If you want to use a different image from the ones available in the Azure Gallery, you need to build your own one, store in the Azure storage as an image object, and then deploy new VMs sourcing from this new template. For Snapshots, the usage and purpose are a bit less evident: whenever you will take a snapshot on your VM Managed Disk, it will be created on ZRS storage and synchronously replicated across all storage zones. This is useful because MD storage cannot be neither ZRS nor GRS, more details in the next section. Now these two objects are “zone-redundant”.

General availability: Azure zone-redundant snapshots and images for managed disks

https://azure.microsoft.com/en-us/updates/azure-managed-snapshots-images-ga

Which kind of storage should I use for my zoned VM?

I decided to dedicate a specific section to this topic because there are some key aspects that need to be clarified. I’m not going to disclose any internal information, everything is in the Azure public documentation, but you may find difficult to connect all the dots and obtain a unique clear picture. First, let me clarify about ZRS: Zone Replicated Storage has been released in v1 long time ago. This is different from what, instead, we recently released as v2 on March 30th, 2018:

General availability: Zone-redundant storage

https://azure.microsoft.com/en-us/updates/azure-zrs

Now v1, renamed as “ZRS Classic”, is officially deprecated and will be retired on March 31, 2021. Original intent was to be used only for block blobs in general-purpose V1 (GPv1) storage accounts. ZRS Classic asynchronously replicates data across data centers within one to two regions. The new v2 instead, uses synchronous replication between all AZs in the region. Zone Redundant Storage (ZRS) synchronously replicates your data across three storage clusters in a single region. Each storage cluster is physically separated from the others and resides in its own availability zone (AZ). Each availability zone, and the ZRS cluster within it, is autonomous, with separate utilities and networking capabilities. Now that you are fully aware of the goodness of ZRS v2, you may be tempted to use it to host VM OS and/or Data disks. Since data replication is synchronous, then no data loss, in case of a zone disaster, you may want to mount the disks to another VM in another AZ. Unfortunately, ZRS v2 does *not* support disk page blobs, as stated in the article below:

ZRS currently supports standard, general-purpose v2 (GPv2) account types. ZRS is available for block blobs, non-disk page blobs, files, tables, and queues.

Zone-redundant storage (ZRS): Highly available Azure Storage applications

https://docs.microsoft.com/en-us/azure/storage/common/storage-redundancy-zrs

At this point, you may ask yourself which storage/disk options you should use for your zoned VMs. As I briefly introduced in the previous section, you need to use Managed Disks (MD). Premium will give you higher performance with SSD based storage, guaranteed IOPS and bandwidth, while Standard will be a cheaper choice. If you create a MD in AZ(1), you will have the benefit of the three local copies, but always remember that MD does not replicate data beyond the local AZ(1). Once MD will be created in a certain zone, it will be *not* possible to mount to a VM located in another zone, compute and storage here must be aligned (see error message below):

“EmptyManagedDiskMD4 cannot be attached to the VM because it is not in the same zone as the VM”

Since you are going to use VMs in each zone with local disks, how would you replicate data? Since we are in the IaaS context, it is all dependent on the application that you will install inside the VMs. Once again, Azure does not replicate the VM data in this context (i.e. data stored in disks) across zones. The customer can have benefits from AZ if their application replicates data to different zones. For example, they can use SQL Server Always-On Availability Group (AG) to host the primary replica (VM where data is written) in AZ(1) and then host secondary replicas in AZ(2), and a third one in AZ(3). SQL Always-On replicates the data from primary replica to the secondary replicas. If AZ(1) goes down, then SQL Server AG will automatically fail over to either AZ(2) or AZ(3). However, Azure automatically replicates the zone-redundant snapshots of Managed Disks. It ensures that snapshots are not affected by zone failure in a region. For example, if a snapshot is created on a disk in AZ(1) and it goes down, then the snapshot can be used to restore the VM in AZ(2) or AZ(3). Below are a couple of excellent articles to understand storage performances and replication strategies.

Azure Storage replication

https://docs.microsoft.com/en-us/azure/storage/common/storage-redundancy

Azure Storage Scalability and Performance Targets

https://docs.microsoft.com/en-us/azure/storage/common/storage-scalability-targets

Where is my VM SKU in AZ?

Not all the VM SKUs are available now for AZ zonal VM deployment, but this gap will be reduced over time and more options will be progressively available. I tried to find a central unique link, to list all SKUs availability per region and per zone, but did not find it at the moment, probably it would be too hard to manage. What you can use instead, is the Azure Portal page when creating a VM and specifying the SKU/size, or you can use the nice PowerShell cmdlet “Get-AzureRmComputeResourceSku” as shown below:

Get-AzureRmComputeResourceSku | where {$_.Locations.Contains("FranceCentral") `

                                 -and $_.ResourceType.Equals("virtualMachines") `

                                 -and $_.LocationInfo[0].Zones -ne $null }

Pay attention to the column “Restriction”: as you can see in the print screen above, currently I don’t have provisioned capacity in that zone, in that region, for my subscription. If you want to list all regions and zones where your subscription can deploy VMs in AZ, you can use a slightly modified command:

Get-AzureRmComputeResourceSku | where {$_.ResourceType.Equals("virtualMachines") `

                                 -and $_.LocationInfo[0].Zones -ne $null `

                                 -and $_.Restrictions.Count -eq 0}

If you encounter the same, you should open a Support Ticket through the Azure Portal and ask for a quota increase. Be sure to specify that you want this capacity in the AZ.  Using the Azure Portal is easy because based on your selection, available SKUs and zones will be specified for your convenience: It is worth noting that some SKUs maybe not available in all the zones, as you can see for GS SKU, available only in zones “2” and “3” but not in “1”, in the print screen above. If you scroll down the list, you will also find two additional sections: the first one will tell you for which SKUs your subscription is enabled but you don’t have enough quota, and the second one will provide information on SKU not available at all in the specific region and zone.

How can I check Zone property for my VM?

Azure Portal properties in the “Overview” pane: In PowerShell is easy to find the zone property for the VM object and all the Managed Disks in a Resource Group, look at the examples below. For now, I don’t see any PowerShell cmdlet dedicated to Availability Zones.   IMPORTANT: At the time of authoring this article, it seems that there is no way to change the zone where a VM is deployed or migrate a VM to make it “zonal”.

How much AZ will cost?

There is no direct cost associated to Availability Zones, but you will pay for network traffic as specified in the article below. There is a fee associated with data going into VM deployed in an AZ and data going out of VM deployed in an AZ.

Bandwidth Pricing Details

https://azure.microsoft.com/en-us/pricing/details/bandwidth

NOTE: Availability Zones are generally available, but data transfer billing will start on July 1, 2018. Usage prior to July 1, 2018 will not be billed.

Thank You!

Simple Azure Portal guide, and step-by-step code sample in PowerShell are available at the links below:

Create a Windows virtual machine in an availability zone with PowerShell

https://docs.microsoft.com/en-us/azure/virtual-machines/windows/create-powershell-availability-zone

Create a Windows virtual machine in an availability zone with the Azure portal

https://docs.microsoft.com/en-us/azure/virtual-machines/windows/create-portal-availability-zone

Hope this content has been useful and interesting for you, let me know your feedbacks. You can always follow me on Twitter (@igorpag), enjoy with new Azure Availability Zones (AZ) feature!                ]]> Why Azure Availability Zones https://docs.microsoft.com/archive/blogs/igorpag/why-azure-availability-zones Sun, 08 Oct 2017 04:49:11 GMT https://blogs.msdn.microsoft.com/igorpag/?p=1786 To fill this gap, Microsoft recently announced public preview of Azure Availability Zones (AZ):

Introducing Azure Availability Zones for resiliency and high availability

https://azure.microsoft.com/en-us/blog/introducing-azure-availability-zones-for-resiliency-and-high-availability

Availability Zones (AZ) are fault-isolated locations, within an Azure region, providing redundant power, cooling, and networking. AZs allow your customers to run mission critical applications with high availability and fault tolerance to datacenter failures. Customers can now deploy VMs and single-zone or multi-zone VMSS, Managed Disks, Public IPs, and Load Balancers, initially in East US 2 and West Europe, with 3 AZs each. More regions with 3 AZ support will come soon. It is worth noting that some other cloud providers with similar functionality, are not offering such high number of zones. With the addition of AZ, Microsoft is now able to present a unique combination of high-availability scenarios and HA SLAs:
  • 99,90% on single instance VMs with premium storage for an easier lift and shift; no other cloud provider so far provides this.
  • 99,95% VM uptime SLA for Availability Sets (AS) to protect for failures within a datacenter.
  • 99,99% VM uptime SLA through Availability Zones with loss protection from fire, power, cooling disruption.
Availability Zones and Availability Sets cannot be used together: when creating a Virtual Machine (VM), you will have to specify an AS, or AZ assignment, you cannot do both. The Azure Virtual Machine official HA SLA is available here, and will be updated with AZ once the feature will be finally released for general availability. Additionally, Azure paired regions within data residency boundaries, but hundreds of miles apart, can protect from larger scale events that may impact an entire region. In order to participate in the Availability Zones public preview, self-service enablement is available, per subscription, using this link.  

Available Services and Resources

During this initial phase of the public preview, these resources are available to be pinned to specific AZs: Virtual Machine (VM), VM Scale Sets (VMSS), Managed disks (MD), Virtual IP (VIP). In addition, there is also a new enhanced version of the Azure Load Balancer that is now zone-resilient. Since there is no Azure Service Management (ASM) API support planned, AZs cannot be used for legacy Cloud Services (Worker and Web roles). From a user perspective, Availability Zones are visible at IaaS level. It is here that you can choose to have control over your basic VM resources (compute, storage, networking). Regarding PaaS, Microsoft will progressively move main services to AZ, but this will be transparent for customers. One of the first service to be moved will be Azure SQLDB, and it makes perfect sense. Azure SQLDB is already architected to have a primary instance and two secondary instances (replicas), all participating into synchronous database replication with quorum voting, zero data loss and automatic failover. Imaging this architecture in 3 Availability Zones is immediate and intuitive. Regarding storage, Azure already has ZRS (Zone Replicated Storage), but is limited to block blob and does not fit in the actual AZ design and implementation. For this reason, it is going under redesign, and once available, will support page Blobs, Tables, Queues and Files. Additionally, each write to the storage, through a single logical endpoint, will be replicated synchronously across all the zones. This would mean zero data loss in case of a zone failure (RPO = 0) and automatic failover of reads towards the other zones. Going back to fundamental IaaS resources, here is the list of changes introduced to adapt to AZ:
  • Virtual Machines (VM) - For a VM object, will be possible to specify a logical "Zone" property value, thus locating the VM in a datacenter supporting that specific AZ.
  • Load Balancer (LB) - Azure Load Balancer now offers two SKUs, Basic and Standard. The Basic SKU is what you have always seen so far in Azure. The Standard SKU is a new enhancement and provides zone redundancy for even higher availability. You can now deploy a Standard Load Balancer that can distribute traffic across all VMs in any AZ in the region, actual scale limit is 1000 VMs. There are many additional advantages on using the new Standard SKU, see here for more details.
  • Virtual IP (VIP) - As for LB, now in Azure you can create Public IP using two SKUs, Basic and Standard. Basic is what you have used so far in Azure, while the new Standard SKU IP enables you to build highly scalable and reliable architectures, in conjunction with Standard SKU LB. This can be zone-redundant, or zonal. You can see below the differences (in order) in how to deploy using ARM templates:
"apiVersion": "2017-08-01", "type": "Microsoft.Network/publicIPAddresses", "name": "public_ip_standard", "location": "region", "sku":             {  "name": "Standard" }
"apiVersion": "2017-08-01", "type": "Microsoft.Network/publicIPAddresses", "name": "public_ip_standard", "location": "region",       "zones": [ "1" ],     "sku":             { "name": "Standard" }
  • VM Scale Set (VMSS) - Can be now deployed in two different flavors (see picture below). First one consists in 3 different Scale Sets deployed 1 in each zone as zonal resources, then load-balanced by a cross-zone resilient Load Balancer (Standard SKU). The second possibility (bottom part of the picture), instead, requires the creation of a single ScaleSet that is cross-region deployed and then resilient. This option will be soon available in preview, you can subscribe here if not already publicly available.
  • Custom Images and Snapshots - You can now create VMs in AZ from a custom VM image, previously there was a limitation applied and only Azure Gallery images were permitted. Additionally, you can now take snapshots of VMs in an AZ. Snapshots are also global objects across all the zones in a region, now they are not "zone-resilient", work is in progress to fix it.
IMPORTANT: Azure Virtual Networks (VNETs) and Subnets, are and will remain "regional" entities. Once you will define in a region, they will be visible and usable across all the AZ since Network Resource Manager in Azure is region-wide. There is no AZ specification for these objects, VNETs and Subnets can cross AZ.

Tools and API

At both the API and Portal levels, changes applied to include AZ are really minimal. In the Azure Portal, there is only one place where you will find AZ, that is Step [3] in the standard creation workflow: Be sure to select a Region and a VM SKU enabled for AZ, otherwise you will not see the edit-box above. See here to check what and where is available. Before proceeding, give a look to the settings for the "Public IP Address", it is interesting: there is a new IP SKU called "Standard", as explained previously. The "Basic" type is the one used so far in Azure. For the Standard, you can see it has "Availability Zone" property but value is fixed, you cannot change, must be in the same zone as the compute resource. This type of IP is zoned (or zonal) and will not survive its home zone failure, not that bad since the VM will behave in the same way. On the other hand, if you want to, for example, unbind this VIP from a VM in Zone [1], and bind to a VM in a different zone, then you must use Standard SKU for the VIP. This kind of IP is zone resilient and valid across all zones, allocation method can be only static. Be careful since for the same VM you cannot mix different IP SKUs, must be same in case of additional IPs assigned. If you want to check details of the subnet and VNET that will be used or created, you will see that there is no "Zone" property: once again, these objects are region-wide, valid across all the zones and can include VMs from all the zones. There is no need, for example, to create separate subnets for different AZs. In the ARM template definitions, pay attention to the additional "zones" property: it is an array of strings, because potentially more than one AZ can be specified for a VMSS. For REST API usage, pay attention to the API version used when creating the VM, the Managed Disks, and the Public IP Address:
  • "type": "Microsoft.Compute/disks", "computeResouresApiVersion" : "2017-03-30"
  • "type": "Microsoft.Compute/virtualMachines", "computeResouresApiVersion" : "2017-03-30"
  • "type": "Microsoft.Network/publicIPAddresses", "apiVersion": "2017-08-01"
List of all the necessary REST APIs for ARM are reported below:

NOTE: At the time of writing this post, REST documentation for VM and Managed Disks must still be fixed to include some details. This should be fixed soon.

Availability Zones (AZ) are already available also in version 1.3 of the Azure Management Libraries for Java, you can find it here on GitHub. Availability Zones management has been included also in the v1.3 release of Azure Management Libraries for .NET that you can find at this link. AZs support is also available using the Azure CLI tool: if you use Cloud Shell, no worry... you are already good to go. Instead, if you install Azure CLI tool locally, be sure to use version 2.0.17 or greater.

az vm create --resource-group myResourceGroupVM --name myVM --image UbuntuLTS --generate-ssh-keys --zone 1

az network public-ip create --resource-group myResourceGroup --name myPublicIp --zone 1 --location westeurope

az vm disk attach -g myResourceGroup --vm-name myVM --disk myDataDisk --new --size-gb 50

Please note that in the last command above, where a new Managed Disk is created, there is no possibility to specify any "Zone": the disk is automatically created in the same zone as the VM.

Step-by-Step Example using PowerShell

Here on GitHub, you can find an easy example, using PowerShell, of how to deploy a Virtual Machine as "zonal" resource, along with a Public IP and a Managed Disk. Additionally, I will show you where the "Zones" attribute will show up, and how to use Snapshots to create/copy and attach Managed Disks in a different AZ. Finally, I will highlight some of the errors and mistakes that you could face trying your first approach. Do not run the entire script at once, it is structured in sections to drive you through the creation process, do it in steps and read before executing. For PowerShell, AZ support is included starting with version 4.4.0. Make sure that you have installed the latest Azure PowerShell module. If you need to install or upgrade, see Install Azure PowerShell module. These are the specific modules versions that you need to have:
  • 3.4.0   AzureRM.Compute  Microsoft Corporation 3.0
  • 4.4.0   AzureRM.Network  Microsoft Corporation 3.0
  • 3.4.0   AzureRM.Storage  Microsoft Corporation 3.0
I tried to explore all my well-known cmdlets, but I was not able to find any one to check if a region is enabled for Availability Zone, and if yes, which are the available zones. I will keep searching, please let me know if you will find it before me. If you will select an Azure region where AZ are not enabled, or you will use an incorrect value, PowerShell commands will return error messages like the one below: To show the "Zones" property of a Managed Disk, let's try to resize the disk. This is the output you will obtain at the end (truncated): In the code sample used, if you look at the section where a Snapshot of the Managed Disk is created, you will see that there is no way, in the used cmdlets, to specify any value for "Zones" attribute: Snapshots are region-wide objects, then valid across all regions. In the main VM object creation step, please note the absence of Availability Set (AS): if you want to use AZ, you cannot include that VM into AS at the same time. If the "Zones" property value will be *not* aligned for VM, Managed Disk and IP, then you will receive a similar error: If the VM SKU used is not one in the supported list for the AZ, you will receive an error like the one below:   If you try to attach a Managed Disk located in a zone different from the one where the VM resides, you will have this error: Finally, with cmdlet "Get-AzureRmVM" you can now see the "Zone" attribute value listed along with the other VM properties:

Thank You!

Hope this content has been useful and interesting for you, let me know your feedbacks. You can always follow me on Twitter (@igorpag), enjoy with new Azure Availability Zones (AZ) feature!

Resources

Overview of Availability Zones in Azure (Preview) https://docs.microsoft.com/en-us/azure/availability-zones/az-overview   Regions and availability for virtual machines in Azure https://docs.microsoft.com/en-us/azure/virtual-machines/windows/regions-and-availability   Azure Load Balancer Standard overview (Preview) https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-standard-overview#why-use-load-balancer-standard   Very simple deployment of a VM in an Availability Zone https://github.com/Azure/azure-quickstart-templates/tree/master/101-vm-simple-zones   Create Multiple Virtual Machines in Different Availability Zones and Configure NAT Rules through the Standard Load balancer https://github.com/wkasdorp/azure-quickstart-templates/tree/master/201-multi-vm-lb-zones   Simple deployment of a VM Scale Set of Linux VMs within an Availability Zone behind a load balancer with NAT rules https://github.com/Azure/azure-quickstart-templates/tree/master/201-vmss-linux-nat-zones   Virtual Machine ScaleSet distributed across Availability Zones with Load Balancer https://github.com/Azure/azure-quickstart-templates/tree/master/301-multi-vmss-linux-lb-zones  ]]> Azure SQL Data Sync Test Drive and First Impressions https://docs.microsoft.com/archive/blogs/igorpag/azure-sql-data-sync-test-drive-and-first-impressions Thu, 06 Jul 2017 13:23:48 GMT https://blogs.msdn.microsoft.com/igorpag/?p=1555 Azure SQL Data Sync is a service of Azure SQL Database that enables you to synchronize relational data you select across multiple cloud databases and SQL Server instances, in uni-directional or bi-directional way. This service is already in public preview, then you don't need to enable, it is already here but you will pay for the databases you will create. Version covered in this post is 2.0, last important refresh of previous long incubated V1 version. The reason I decided to test this feature, and that's why is attractive for me, is that in the Microsoft SQL Server universe, both on-premises and cloud, there is no multi-master concept: doesn't matter how many SQL Server database cluster nodes, replicas, Availability Group instances you have, you still have to write data to a single instance that is your master. There is only one exception using SQL Server box product, that is Data Replication feature, using Merge or Peer-to-Peer topologies. In these limited cases, you can submit your transactions to different multiple writable masters that will (slowly) replicate data each other. It is important to understand that all these technologies, including Azure SQL Data Sync, do not provide strong consistency, they do not provide distributed synchronous transactions between multiple writable masters. Instead, data replication is asynchronous, will have some latency, it does not follow original transaction order, conflicts may happen, and application must be able to tolerate this or designed to work with it. Then, in which scenarios Azure SQL Data Sync would be useful? Whenever you need to replicate or synchronize data between different Azure SQL databases and SQL Server data sources, requiring multiple writable masters, where application does not rely on strong consistency and synchronous replication, you may want to consider this feature. Distributing data across regions and/or environments, for example in different Azure datacenters, or between on-premises and the cloud, are also valid scenarios. Reference data like list of customers, product catalogs and object directories are good real examples. Conversely, Azure SQL Data Sync is not appropriate if your application requires strong data consistency, is not appropriate to replicate a multi-terabyte OLTP database with millions of transactions per second, is not appropriate to build disaster recovery architectures, or to support migrations.

Limitations

Currently, this feature comes with some limitations you need to be aware, full details are reported in the article below, I reported here the main ones for your convenience:

Sync data across multiple cloud and on-premises databases with SQL Data Sync

https://docs.microsoft.com/en-us/azure/sql-database/sql-database-sync-data

 
  • Replication time: there is no SLA on replication latency.
  • Limitations on service and database dimensions: max number of endpoints in a single sync group is 30, max 500 tables in a Sync Group, max 100 columns per table, max row size is 24MB, maximum replication frequency is 5 minutes.
  • Performance Impact: Data Sync uses insert, update, and delete triggers to track changes. It creates side tables in the user database. These activities have an impact on your database workload, so assess your service tier and upgrade if needed.
  • Eventual Consistency: since Data Sync is async, transactional consistency is not guaranteed. Microsoft guarantees that all changes are made eventually, and that Data Sync does not cause data loss.
  • Unsupported data types: FileStream, SQL/CLR UDT, XMLSchemaCollection (XML supported), Cursor, Timestamp, HierarchyID.
Since this feature is still under development, additional capabilities will be available in the near future, top desired ones from my test are:
  • Higher sync frequency, less than actual 5 minutes.
  • Automatic management of table schema changes.
  • Custom script (or procedures) to manage data synchronization conflicts.
  • Include a graphical map, in the "Sync to other databases" page in the Azure Portal, reporting the Members and Hub in the "Sync Group", healthy status and some replication performance numbers.

Overview

Before starting my drive test, we first need to introduce the basic concept of "Sync Group", better to start with a nice picture: "Sync Group" is a logical object that describe your synchronization topology, then including the list of your databases and tables, and common settings that are shared across all members in the group. Replication path follow a "Hub & Spoke" model where each "Spoke" (Member) synchronize separately with the "Hub". Now let's go through a complete end-to-end provisioning experience, explaining details in each step. First artifact you need to create is a kind of configuration database called "Sync Metadata Database": it will contain all necessary metadata to maintain your "Sync Group" (can be used for more than one), for a specific subscription, in a specific region. When enabling database sync, if not existing, you will be asked to create a new one. Microsoft recommends that you create a new, empty database to use: setup creates tables in this database and runs a frequent workload, there is no minimum recommendation for the service tier and size, but you need to be careful especially if you will use the same one for many databases and "Sync Group". NOTE: As you can see, support in the old Azure Portal has been just retired from July 1st, 2017. If you have older version of Azure SQL Data Sync, please refer to the article below for migration as soon possible.

Migrating to Azure SQL Data Sync 2.0

https://azure.microsoft.com/en-us/blog/migrating-to-azure-data-sync-2-0

The way you access this feature is clearly telling you that Azure SQLDB is the central hub, you can also have "Spoke" from on-premises, SQL Server installed in Azure VMs (or from other Cloud providers) or other Azure SQL DB databases. Then, to start you need to open the new Azure Portal and point to one of your Azure SQL DB database, then click on "Sync to other databases": as you can see in the picture above, if there is no existing "Sync Group", you will be asked to create a new one. In the first blade of the procedure, there are the most important settings for your new "Sync Group":
  • "Sync Metadata Database": as explained before, you can create brand new or re-use an existing one in the same region. For convenience, and since this is a test, I will use the same Azure SQL DB database also for the "Hub" These are the requirements for the Sync database:
    • Azure SQL Database of any service tier
    • Same region as the "Hub" database of a "Sync Group"
    • Same subscription as "Sync Group"
    • One per region in which you have a "Sync Group" ("Hub" Database)
  • "Automatic Sync": you can decide here if you want automatic system initiated data sync activity, or if you prefer manual triggering instead. In the former case, you will need to select a schedule, and then a frequency; in the latter case instead, you will need to use API to start the synchronization on-demand. It is worth noting that the maximum frequency you can set is 5 minutes, but the UI is showing you the possibility to select "Seconds" unit measure. I can only guess here that in the future more frequent syncs will be possible.
  • "Conflict Resolution": this is very important because will determine which data update your application can afford to lose in case of conflict: which update you want to discard if, inside the same sync cycle, the same record (determined by table Primary Key) is updated on both the "Hub" and the "Member"? Please note that this parameter is at the entire "Sync Group" level and cannot be changed without destroying and re-creating the entire structure.
If you are asking about custom script, or stored procedure, to resolve replication conflict, as happens in SQL Server Replication feature, then the answer is NO: at the moment, this is not possible, but it has been asked already to the owning Product Group, then stay tuned. Now, let us move to the second step, as you can see in the picture below. First, you can add "Members" database from other subscriptions (and logical servers) but must be under the same Azure Active Directory (AAD) tenant, then also from other regions if you want. Second, the interface here does not allow you to create new empty databases on the fly, you have to have already created them. Third, and probably most important, what here is called "Sync Directions": remember, the data synchronization architecture and scheme here is "Hub & Spokes", and is directional, then you need to decide how data will spread inside the "Sync Group". It is important to keep in mind that synchronization tasks happen separately, and in parallel, for each pair formed by "Hub" and "Member" databases. Direct consequence is that you cannot assume the order by which each "Member" will be updated. In case of bi-directional replication, first data is downloaded from the Hub, then data is uploaded from the member to the hub. IMPORTANT: A single database can be a "Member" in different "Sync Group", but in this case, you need to pay attention and avoid creating circular references as shown in the picture below. Currently Azure SQL Data Sync does *not* automatically manage this, and then you may end up in an infinite replication loop that will never terminate.

Adding On-Premises Members

Procedure to add a SQL Server instance and databases, from on-premises or from a VM in Azure, is almost identical from an Azure SQL Data Sync perspective: you have to download an agent, called "Sync Agent Gateway", and install inside the network environment. While is possible to install the agent locally on the SQL Server box, it is not recommended for two reasons: first, you may not want to install something directly on the production machine: secondly, the agent is not cluster-aware then if using SQL Server Failover Cluster Instance (FCI), it is not supported nor will work in case of node failure. SQL Server 2005 SP2 and higher versions are supported. Regarding the connectivity and security, is the agent that will initiate the outbound connection to the Azure SQLDB "Hub" and "Metadata Sync" databases, then there is no need to open firewall port TCP 1433 to allow inbound traffic to your protected network. Now, you can copy the URL to download the agent and use it to download and install locally to your network (or Azure Virtual Network). It is worth remembering that inside Azure VNET, by default a Virtual Machine has full open outbound connectivity to the Internet: if you plan to use Network Security Groups (NSG) to restrict, please be aware of this port requirement.

SQL Azure Data Sync Agent

https://www.microsoft.com/en-us/download/details.aspx?id=27693

      Be aware of the following pre-requisites before installing the agent, and please note that the 32bit version is explicitly required. In this case, installation order matters also. Once installed you must run the configuration tool before being able to use it: provide the key generated in the Azure Portal, then supply the credentials to access the Sync Metadata Database. Once registered the SQL Server instance and database, using either SQL Authentication or Windows Integrated, you should see something like the picture below. You can test the connection to the Azure SQLDB hub, to the local SQL Server instance, you can change the Agent Key and upgrade eventually the agent version. You can use the same agent to sync multiple SQL Server local instances. Finally, you can go back to the Azure Portal and complete the member database configuration.

My Test Configuration

At the end, the Sync Group configuration I created is reported below:
  • Hub
    • Logical Azure SQLDB server igor-sqlsrv1 in West Europe, hub database name is igor-synchub-we, created in Subscription1.
  • Members
    • Logical Azure SQLDB server igor-sqlsrv1 is in West Europe, member database name igor-sqldb1, created in Subscription1.
    • Logical Azure SQLDB server igor-sqlsrv2 is in North Europe, member database name igor-sqldb2, created in Subscription1
    • Logical Azure SQLDB server igor-sqlsrv3 is in West Europe, member database name igor-sqldb3, created in Subscription2
    • igor-sql2016vm1 is a VM with SQL Server 2016 installed in West Europe, member database name is igor-sqldb4, created in Subscription1
  • Conflict Detection: Hub wins
  • Data Directions & Conflict Detection
    • Bi-directional Sync for member database igor-sqldb1
    • From the Hub for member database igor-sqldb2
    • To the he Hub for member database igor-sqldb3
    • From the Hub for member database igor-sqldb4
Here is the picture from the Azure Portal related to the member list: Enable Data Replication Now is finally arrived the time to select which tables to include in our Sync Group replication topology. You can start clicking on the big Tables icon in the Azure Portal: Depending on the replication direction, initial database to choose for selecting table may vary, in this case let us proceed from the Hub Database where I created a simple table called Table1. In order to be considered as a valid selection, a table *must* have a Primary Key (PK), this is required to locate changed items and manage conflicts. I also created another table called Table2 but as you can see, is not suitable for Azure SQL Data Sync replica since does not contain a PK. You are allowed to select specific columns you want to replicate, by default everything is selected, PK *must* be included. NOTE: I tried to find a way to horizontally select the rows to sync, but apparently there is no way to include a WHERE filter or insert a custom SELECT to specify which rows to include in the mechanism. I also tried to create a VIEW with a WHERE clause on the base table, but the GUI interface did not allow me to select. Finally, with at least one table added into the Sync Group, data flow is now active. Once confirmed the tables that you want to include, Azure SQL Data Sync will do immediately create the database tables in *all* database Members, if not existing already (see later for more details) despite the replication direction you selected. For example, for member database igor-sqldb3, as you can read in the configuration section above, I selected To the he Hub, then from the Member upward, but the table has been created also here. What is important noting is that only the schema has been replicated, no data has been pulled down from the Hub database. Once schema is applied to all Member databases, you will see that the button Sync is enabled: data replication will happen based on the frequency you specified, and that can be changed in the Sync Group properties, but you can always force manual start if desired. This operation can be obviously also invoked using REST API and future PowerShell module.

Schema Changes

At this point, would be interesting to understand if this mechanism is tolerant to schema changes of the tables included in the Sync Group. Let us try to add, for example, a new (NULL-able) column named Value2 to Table1, insert a new row, fix (NULL) values with new default:

USE [igor-synchub-we]

GO

ALTER TABLE [dbo].[Table1] ADD [Value2] BIGINT DEFAULT (0)

GO

INSERT [dbo].[Table1] VALUES (2,'Added Column 2',0)

GO

UPDATE [dbo].[Table1] SET [Value2] = 0 WHERE [Value2] IS NULL

GO

SELECT * FROM [dbo].[Table1]

GO

Looking to the logs in the Azure Portal, this modification has been safely processed, two changes in rows have been detected: Looking to all Member databases, row changes are replicated (Download: 2 changes applied) but now the schema change, all the existing rows do not have the additional column we introduced, but the mechanism continue to work without errors. My experiment is not finished yet, let us try to introduce the schema change using the Azure Portal interface and see what will happen: Apparently, the operation is succeeded, but looking at the error log, several error messages raised:

Database re-provisioning failed with the exception SqlException Error Code: -2146232060 - SqlError Number:207, Message: Invalid column name 'Value2'. SqlError Number:207, Message: Invalid column name 'Value2'. For more information, provide tracing ID ‘d94e2fcf-eed2-4f92-8ae3-88721d956cc1’ to customer support.

Ok, schema modification will not be propagated by the Azure SQL Data Sync, then the only remaining option is to remove the table from the Sync Group, apply the schema change to all Members, then adding back the table, including the additional new column. Please remember that this feature is still in preview. Regarding error log, in the main blade for Data Sync Group, there is no way to select or download error messages, you should use general Azure Activity Log to achieve this task: Finally, what happens if a table with the same name exists already, in the Member target database, when included in the sync set? In this case, synchronization mechanism will *not* overwrite it, but will apply necessary changes, for example will create TRIGGERS to manage data replication. This is powerful since you may want to customize the table structure with additional columns, constraints, indexes or triggers: be careful because you have this flexibility, but these changes must not interfere with the synchronization logic. During my tests, I verified that TRIGGER, FOREIGN KEY and CHECK constraints are not replicated downstream to the other databases in the Sync Group, then if you want or need them, you have to modify the target DBs manually.

Database Internals

Azure SQL Data Sync modifies the schema of your database adding object schemas, tables, triggers and stored procedures. This is very important to consider before enabling this mechanism because will alter your database structure and may conflict with your application. Let us take a small journey inside these modifications. If you have minimal SQL Server experience, you can look very easily at the tables created by Azure SQL Data Sync setup inside the Hub Database, under the new dss and TaskHosting objects schemas. Obviously, you can look and do your read-only queries, but you should not alter in any way any content of these tables, otherwise you may destroy your Sync Group. If you try to do the same inspection on the Member databases, you will see similar tables added to support data synchronization, but a different smaller set under DataSync schema, you can observe in the red rectangle below. Those tables are always present, but the DataSync.Table1_dss_tracking is special: this kind of shadow table is created for every user table (in this case Table1) enabled for data sync, purpose is to accumulate changes in the rows of the original table. How this table is populated with data? If you give a look to the structure of Table1 and dependencies, you will discover this: There are stored procedures defined for DELETE, INSERT and UPDATE and bulk-logged versions, and there are triggers (after-trigger type) defined for DELETE, INSERT and UPDATE operations. If you examine the code for INSERT trigger Table1_dss_insert_trigger you will see that the core is a T-SQL MERGE statement that will detect inserted row, then will upsert data in the shadow table [DataSync].[Table1_dss_tracking]: The INSERT clause is very interesting: as you can see above, the shadow table is *not* accumulating the entire row (all column values), but only the Primary Key (PK) along with some metadata information, including timestamp of the last modification. If you are a DBA, you will recognize here something similar to Change Tracking, not really Change Data Capture, well known SQL Server box product features. The tracking records will not be deleted immediately. Tracking records for deleted rows will be removed after a retention period of 45 days. There are optimizations that can avoid using the tracking tables during initial sync, after the initial sync all the changes are going to be tracked in the tracking tables. Index fragmentation may happen, you can refer to this article to reorganize the index. It is worth noting that in case of multiple updates to the same row, there will be only one record in the shadow table including the PK and timestamp of the latest update. Finally, table [dss].[configuration] is very interesting since it contains main configuration parameters for the current Data Sync Group. I found very interesting, in the Hub Database, the table [dss].[configuration], specifically parameter SqlSyncProviderBatchSizeInMB with value (24): it seems that Azure SQL Data Sync max batch size is 24MB and max 50,000 rows per batch, when applying data modifications to synched tables. What I expected to find here is something indicating how long a database can be disconnected before being marked ad offline, that is not synchronized anymore. I do not see in the list above, but downloading the full documentation in PDF format from here, I found that the limit is 45 days:

SQL Data Sync (Preview) removes databases that have been offline 45 days or more (as counted from the time the database went offline) from the service. If a database is offline for 45 days or more and then comes back online its status is set to Out-of-Date. 

What is totally missing also from the online documentation, at the moment, is more details on the retry mechanism and how long it will last.

Programmability

Today, you can play with Azure SQL Data Sync using the new Azure Portal interface. You can find the REST API specifications at the link below; PowerShell support is coming soon and will be available for the general availability of this feature. Download the SQL Data Sync REST API documentation

https://github.com/Microsoft/sql-server-samples/raw/master/samples/features/sql-data-sync/Data_Sync_Preview_full_documentation.pdf?raw=true

Final Remarks

For the latest announcement on Azure SQL Data Sync, see the blog post below, more will come when Product Group team will announce general availability:

Azure SQL Data Sync Refresh

https://azure.microsoft.com/en-us/blog/azure-sql-data-sync-refresh

Regarding pricing, it has not disclosed yet, but you will pay for sure for all your databases, including Sync Metadata Database and Hub Database, depending on the service tier you selected. If replicating data across Azure regions, you will be charged for this traffic. If you have also on-premises members, only egress traffic exiting from Azure will be counted, igress traffic will not. Full Azure SQL Data Sync documentation can be downloaded from here in PDF format. If you want to post feedbacks and ideas for this feature, you can use the link below, Product Group will look at it and post a response.

How can we improve SQL Data Sync?

https://feedback.azure.com/forums/217535-sql-data-sync

Thanks for your patience, hope you appreciated this content. You can follow me on Twitter using @igorpag. Regards.  ]]> Using PowerShell as an Azure ARM REST API Client https://docs.microsoft.com/archive/blogs/igorpag/using-powershell-as-an-azure-arm-rest-api-client Wed, 28 Jun 2017 14:14:37 GMT https://blogs.msdn.microsoft.com/igorpag/?p=1255 Introduction and Goals Just mentioned above that there are already several tools available on the Internet that will permit you to use ARM REST API directly and making web requests, let me list here first the most popular and my favorites:
  • Postman: you can download from here versions for Windows, Linux, MacOS and as a Chrome extension. Nice tool, available in free basic version and pro paid. powerful GUI platform to make your API development faster & easier, from building API requests through testing, documentation and sharing.
  • HTTP Master: grab your free basic or professional paid version from here. in addition to basic features, it provides almost complete coverage for development and testing of API applications and services, including HTTP tool to simulate client acitivity.
  • ARM Client: it is not an official Microsoft tool, instead is an OSS Project you can find on GitHub or can install via Chocolatey. Nice article from David Ebbo if you want to know more.
There are also many articles that will explain you how to use Azure ARM REST API, then why I decided to publish this post focused on PowerShell? Long story short, I recently worked on two different projects aiming the same kind of final result: write everything necessary to create and manage a series of Azure resources, using plain ARM REST API calls. In the first case, my partner wanted then to include these REST calls in their Java application. In the second case, my other partner wanted to include REST API in their deployment engines using different components written in different languages (Python, .NET, PowerShell), then requiring a common denominator to avoid re-writing code. Another important aspect for the first project was the requirement to have access to new Azure features immediately, or at least as soon as possible once something new feature is announced.

How new Azure API are published?

This is an important first detail to be aware of: practically, there is no new Azure feature if there is no REST API available to use it, and obviously you need to have API reference and documentation to use. What happen normally, when Microsoft release a new feature, is that REST API are built at the very first, and described  here on GitHub: What you will find in that repository are the REST API full details for all Azure things, decorated with Swagger specs. If you don't know what is Swagger and want to know more, you can go here, or take my short and trivial scholastic definition: Swagger is a specification for machine-readable interface files for describing, producing, consuming, and visualizing RESTful Web services. Finally, what Microsoft internally does is using a tool called AUTOREST (and something more called extensions) to automatically generate client SDKs and make available to you for consume. You can find it here on GitHub:

Swagger (OpenAPI) Specification code generator featuring C# and Razor templates.

Supports C#, Java, Node.js, TypeScript, Python and Ruby.

Let me emphasize this: official ARM REST API documentation should be updated to reflect the change soon, but it is worth nothing that this would require some time since human intervention is necessary to write, review and publish content:

Azure Resource Manager

https://docs.microsoft.com/en-us/rest/api/resources

Then, if you want to know as quickly as possible about how to use a new Azure feature, you should go to GitHub and look at REST API specs. Documentation will follow soon. NOTE: At the time of writing, there are no Swagger specs for Azure Storage API. If you are a Visual Studio 2017 user, and interested in building Web API using Swagger, you may want to review the article below:

Visual Studio 2017 and Swagger: Building and Documenting Web APIs

https://www.simple-talk.com/dotnet/net-development/visual-studio-2017-swagger-building-documenting-web-apis

PowerShell Pre-Requisites

Now that you are a bit familiar with Azure REST API and how they are published, and heard why I used PowerShell to consume these APIs, let me start showing you a practical example on how to build REST API calls, invoke them and manage the results. But before going into details, you need something to prepare at the beginning of your PowerShell script, here is the list of logical steps:
  • Initialize your script with common variables values you will use all over the script. This will eventually include Subscriptions details (ID and Name), Azure region, create common objects like a default Resource Group, you Azure Active Directory (AAD) tenant.
  • Authenticate to your Azure subscription using Azure Active Directory (AAD) account: this step will use interactive authentication (Login-AzureRmAccount) just to create Application and Service Principal objects in AAD. This is required only one time: once created the necessary objects for application logon (see later), your application/script will be able to logon to AAD programmatically without any user intervention.
  • Create an Application object in AAD (New-AzureRmADApplication). Azure offer developers the possibility to build applications that can be integrated with Azure Active Directory (AAD) to provide secure sign in and authorization for their services. To integrate an application or service with AAD, a developer must first register the details about their application with Azure AD. This can be done using the Azure Portal or using PowerShell script as in my code sample below. The important piece of information you will need to obtain from this step is ApplicationID that you will use as a surrogate User Name for creating logon credentials.

Use portal to create an Azure Active Directory application and service principal that can access resources

https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal

 
  • Once created the Application object in AAD, and obtained its ApplicationID, you need to create a Service Principal object using  New-AzureRmADServicePrincipal cmdlet, associated to the ApplicationID. Why you need this object? It is important to understand the difference between Application and Service Principal. You can read more at the link below, but in short, you can consider the application object as the global representation of your application for use across all tenants, and the service principal as the local representation for use in a specific tenant.

Application and service principal objects in Azure Active Directory (Azure AD)

https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-application-objects

 
  • With the Service Principal object in your hands, you can use now use to assign permissions in the current AAD tenant. In the example below you can see both scopes supported: at the entire Azure subscription level, or scoped to a particular Resource Group (recommended). You can use both default built-in AAD roles or create new custom ones, depending on your requirements.
  • Next step consists in creating a new Key in AAD, associated to your just created Application, and then store for later use in your script. You can read here instructions to do this using the Azure Portal (key chars scrambled).
  • Using nice PowerShell mechanism, as you can see in the example below, you can now create encrypted credential set (user name and password) to pass to your subsequent cmdlets. Be aware that "ApplicationID" is your username and must be used in the form of "username@domain-name", where the domain-name looks like "<something>.onmicrosoft.com". Last piece of information you need to retrieve is AAD Tenant Name for your subscription, you can easily retrieve using last PowerShell script line shown below.
  • From now and going forward, you can logon using just created Application identity and secret, you don't need interactive logon anymore. After execution of the new instance of Login-AzureRmAccount with application parameters, you will be now authenticated as an application in the script.
We are now at the final preparation step, probably the most important one, that is getting a Bearer authentication token from AAD endpoint (https://login.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/token) using Oauth2 (key chars scrambled): As you can see above, I used "Invoke-RestMethod" PowerShell cmdlet to send HTTPS requests to Representational State Transfer (REST) web services, in this case a PUT to Azure AAD Oauth2 passing ApplicationID and Key, for the ARM resource endpoint (https://management.core.windows.net) . Token as a form similar to the one below (text is scrambled). Be aware that the default life time for the token is 60 minutes (3600 seconds), after that you will need to generate another request and acquire a new one.   Azure recently introduced the possibility to change the default lifetime of an AAD token feature is still in preview and I did not personally use it yet, but you may want to read the article below:

Configurable token lifetimes in Azure Active Directory (Public Preview)

https://docs.microsoft.com/en-us/azure/active-directory/active-directory-configurable-token-lifetimes

Building and Executing REST calls

Now we have a valid AAD token and we can finally do something interesting with Azure ARM REST API. It is important to highlight that every REST call must include authentication token in the header, but I am sure will be clear once you will see below the first trivial example, that is how to retrieve details about your Azure subscription with a simple GET verb: Please be aware that the result is not JSON formatted by default, you need to explicitly convert, then you will see a similar output as the one in the picture below. It is also worth mentioning that specifying the API version is mandatory. Finally, you can see there is no Body part defined in this simple GET request. The sample above is pretty simple, at the end is a GET request getting synchronous answer. But what happen if I want to write (PUT for example) something? Look at the example below used to create a storage account: NOTE: in the code snippet above, I started using Invoke-WebRequest instead of Invoke-RestMethod because I had problems in correctly parsing and processing the response header, I normally prefer the former. In the response content above there are several interesting things: first, the status code returned in StatusCode is (202), that is Accepted (StatusDescription) in HTTP response code vocabulary. It is indicating that the request you submitted is a long running asynchronous operation, then the system returned your code a result, but the backend operation is still running in the Azure ARM infrastructure. For synchronous operations like typical GET methods to read objects and resources, you normally receive 200 (OK).

Microsoft REST API Guidelines

https://github.com/Microsoft/api-guidelines/blob/master/Guidelines.md

According the excellent documentation above, "Retry-After" HTTP header indicating the minimum number of seconds that clients SHOULD wait before attempting the operation again. Why this is important? Azure REST API infrastructure tends to protect itself from being overloaded, then this value is returned in all async responses. But, if you do not  want to obey to this hint, there is another safe limit that Azure enforces, you can guess looking to x-ms-ratelimit-remaining-subscription-writes element. For each subscription and tenant, Azure Resource Manager limits read requests to 15,000 per hour and write requests to 1,200 per hour. When you reach the limit, you receive the HTTP status code (429), that is "Too many requests". You can see all the limits in the article below along with additional details.

Throttling Resource Manager requests

https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-request-limits

Now, if you want to check the status of your running async operation, you need to execute some code like the example below:   And here is the content of the variable $URIAsyncCheck (single line scrambled): General rule to track async operation, for Azure ARM REST API, requires you to check a specific dynamically crafted URL for the status: this URL is generated depending on the ARM Resource Provider (RP) you previously sent the async request to, in this case the Microsoft.Storage RP, and the OperationID of that request. You can see these elements highlighted in red in the above text. Unfortunately, there is no single place you can retrieve that OperationID: as you can see in the code snippet above, you have first to check for the existence of value for Location attribute in the response header, then if nothing is in there, you need to check Azure-AsyncOperation.  With this knowledge in mind, let us check the long running operation using the code snippet below: This is the output you should see: As you can see, this is a long running async operation (Response Code = 202 Accepted). You can also note that until the Response Code will not change to 200 - OK, there is no body content returned (Content Length = 0). You should then wait in a loop and check periodically for this condition, once happened, you should check for provisioningState attribute value returned: if, and only if, the operation is completed, then possible values are "Succeeded", "Failed" or "Canceled". Different values could be returned, depending on the Resource Provider you may see different values, but this indicates that the request is not completed yet. Some operations that you could think as being asynchronous, in reality they are not, as in the example below related to the deletion of a storage account (Sample[5] in the sample code): Here the only difference is the Method = 'Delete' part, and result HTTP code will be 200 - OK (if the storage account exists and you have permissions). No attribute "Location" or "Azure-AsyncOperation" will be generated to check for async operation. You can see details for this particular REST API at the location below:

Storage Accounts ARM REST API

https://docs.microsoft.com/en-us/rest/api/storagerp/storageaccounts

Going through the examples, I refined and sometimes modified the logic to track async long-running operations, a sample (extracted from the GitHub code) is below: Building REST API request Header and Body can become difficult for very large objects as Virtual Machine. You can see an example below, please note that only the minimum required parameters are used, there are many more optional that I did not include. If you want to work using this approach, you need to keep in your hands the Azure ARM REST API reference pages below, and the -Debug switch in PowerShell, you can read more details on the latter in the next section.

Azure REST API Reference

https://docs.microsoft.com/en-us/rest/api

 

Code Samples on GitHub

Here on GitHub, you can find the code and how to deploy, end- to-end, an Azure Virtual Machine, including NSG, VNET and IPs, in a new Resource Group all using REST API direct call in PowerShell. Code for Application and Service Principal creations in Azure Active Directory is also provided. Here is the sample list divided in PowerShell regions: Purpose of this work is to show how to work directly with Azure ARM REST API using PowerShell as an ARM client. Code is highly de-normalized to make each section pretty autonomous and self-contained to facilitate reuse. At the end, what you should be able to obtain is the following list of resources, all of them built using plain REST API built and executed in PowerShell.

PowerShell Hidden Gems

I have been surprised many times why most people don't know the power of Azure cmdlets when adding -Debug switch at the end of the command string. Azure PowerShell module is a wrapper built on top of Azure ARM (in this case) REST API, but if you add it, you will see the raw underlying REST API request built for you, and the correspondent result:

Get-AzureRmStorageAccount -ResourceGroupName $rgname -Debug

Then, if you want to work at the pure REST API level, but want your life a bit easier, you could use higher-level PowerShell cmdlets and then reverse-engineer them using the -Debug switch. Another very useful feature you can use in PowerShell is Stop Watch. Starting Sample[6] in my GitHub code example, I used it extensively to monitor execution time of my REST API: below you can find a very simple general structure of its usage.

Java Sample

If you want a Java version of something similar I have shown you here, I would encourage you to look at Silvano Coriani (from AzureCAT) sample work on GitHub here, very nice if this is your favorite development language.

https://github.com/scoriani/ServiceBuilder/blob/master/src/com/microsoft/azure/provisioningengine/ProvisioningEngine.java

References

GitHub Code Sample Repository: https://github.com/igorpag/PSasRESTclientAzure Azure REST API specs: https://github.com/Azure/azure-rest-api-specs Azure Resource Manager: https://docs.microsoft.com/en-us/rest/api/resources Azure REST API Reference: https://docs.microsoft.com/en-us/rest/api Track asynchronous Azure operations: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-async-operations Throttling Resource Manager requests: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-request-limits Resource providers and types: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-supported-services Use portal to create an Azure Active Directory application and service principal that can access resources: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal Use Azure PowerShell to create a service principal to access resources: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-authenticate-service-principal Use Resource Manager authentication API to access subscriptions https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-api-authentication  ]]> My personal Azure FAQ on Azure Networking (v3) https://docs.microsoft.com/archive/blogs/igorpag/my-personal-azure-faq-on-azure-networking-v3 Thu, 06 Apr 2017 08:00:52 GMT https://blogs.msdn.microsoft.com/igorpag/?p=875 Last updated: April 6th, 2017. IMPORTANT: The information I reported in this post is time sensitive since may change in the future. I will try to maintain the content updated, but you should double-check with official Azure documentation sources. This is v3 of my previously published blog posts below, I will maintain previous versions for historical reasons:

My personal Azure FAQ on Azure Networking SLAs, bandwidth, latency, performance, SLB, DNS, DMZ, VNET, IPv6 and much more (ARM)

https://blogs.msdn.microsoft.com/igorpag/2014/09/28/my-personal-azure-faq-on-azure-networking-slas-bandwidth-latency-performance-slb-dns-dmz-vnet-ipv6-and-much-more

My personal Azure FAQ on Azure Networking SLAs, bandwidth, latency, performance, SLB, DNS, DMZ, VNET, IPv6 and much more (ASM)

https://blogs.msdn.microsoft.com/igorpag/2014/09/28/my-personal-azure-faq-on-azure-networking-slas-bandwidth-latency-performance-slb-dns-dmz-vnet-ipv6-and-much-more

In the last five years, I worked extensively on Azure core IaaS, during engagements with my partners I have been asked many questions on Azure networking in general, and more specifically on Virtual Network (VNET) and VMs in new Azure Resource Manager (ARM) world. Providing adequate answers is not always easy, sometime the documentation is not clear or missing in details, while other times you can deduct yourselves but requires pretty good knowledge. Since I detected some recurring patterns in these questions, I decided to write my personal FAQ list in this blog post. Just to be clear: even if I’m a MSFT and have inner knowledge of Azure, I’m not going to unveil you any reserved or secret information. Everything you will read here can be found playing directly with Azure or retrieved by using official and public documentation. If you have additional “nice” questions requiring a non-trivial or non-easy-to-retrieve answer, fill free to insert a comment below this post. Additionally, if you have proposals or feedbacks regarding new feature that Azure networking should have, please use the link below to submit your ideas or vote existing ones:

FEEDBACK FORUM: Networking (DNS, Traffic Manager, VPN, VNET)

https://feedback.azure.com/forums/217313-networking-dns-traffic-manager-vpn-vnet/category/77469-virtual-networks-vnet

As usual, you can also follow me on Twitter at @igorpag. Regards.  

Can I take a network trace inside my Azure Virtual Network?

YES. This cool and useful feature has been recently released under the Azure “Network Watcher” service. You can now capture packet data in and out of a virtual machine. Advanced filtering options and fine-tuned controls such as being able to set time and size limitations can be applied as well. The packet data can be stored in a blob store or on the local disk in .cap format, then use tools like WireShark to analyze and display.

Introduction to variable packet capture in Azure Network Watcher

https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-packet-capture-overview

1 Additionally, packet capture provides the capability of running proactive captures based on defined network anomalies, thus triggering actions to react to network conditions. More details in the article below:

Use packet capture to do proactive network monitoring with Azure Functions

https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-alert-triggered-packet-capture

Be aware that packet capture requires a virtual machine extension to be installed inside the VM (AzureNetworkWatcherExtension). It is available for both Windows and Linux Operating Systems. This feature can be managed from the Azure Portal and PowerShell.

How can I check my network resources usage against my assigned limits?

One of the nice features of new Azure “Network Watcher”, is the possibility to check all networking related resources currently used in a specific region for a certain subscription. All Azure resource limits, including networking ones, are listed in the article below. Hard limits cannot be increased, for Soft Limits, instead, you can open an incident to Azure Support and ask to increase.

Azure subscription and service limits, quotas, and constraints (Networking)

https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits?toc=%2fazure%2fvirtual-network%2ftoc.json#networking-limits

2

Can I persist my VM MAC address?

YES. Azure now provides persistent MAC addresses generated at NIC creation time. This ID will not change over the entire NIC lifecycle, even if the VM will be stopped, crashed and recovered to a different Azure Host, resized, deallocated or unbound. Be aware that Azure assigns a MAC address to the NIC only after the NIC is attached to a VM and the VM is started the first time. You cannot specify the MAC address that Azure assigns to the NIC. The MAC address remains assigned to the NIC until the NIC is deleted or the private IP address assigned to the primary IP configuration of the primary NIC is changed.

Network interfaces

https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-network-interface

Is there any limit on VNET and subnet sizing?

YES. Inside a VNET you can create a maximum of 1000 subnets. This is a “soft” limit that can be eventually increased. The smallest subnet we support is a /29 and the largest is a /8 (using CIDR subnet definitions). Inside each subnet, Azure reserves some IP addresses within each subnet. The first and last IP addresses of the subnets are reserved for protocol conformance, along with 3 more addresses used for Azure services. Regarding IP addresses, currently Azure VNET supports 4096 IP addresses, independently from the number of NICs and VMs and from their state, stopped VM will still count toward the limit.

Azure Subscription limits for Networking

https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#networking-limits

 

Can I create a Storage Account or Azure SQLDB with restricted access from a VNET only?

While is possible to create a Virtual Machine not exposed to the Internet, for Azure Storage Accounts and Azure SQLDB logical servers there is a mandatory DNS name created that need to be exposed. During creation of these resources, a FQDN records is created and exposure cannot be avoided since this resource locator is required to access the resource. Both these resources are protected by Microsoft DDoS and Azure SQLDB can be also configured with its own firewall mechanism.

Configure an Azure SQL Database server-level firewall rule using the Azure Portal

https://azure.microsoft.com/en-us/documentation/articles/sql-database-configure-firewall-settings

Currently there is a feature request to provide an internal-only endpoint for storage and Azure SQLDB to VMs inside a VNET: they will be able to access these services even without Internet connectivity. This functionality should be available in the second half of calendar year 2017.

Create predefined NSG for Azure Datacenters IP Range https://feedback.azure.com/forums/217313-networking/suggestions/11093742-create-predefined-nsg-for-azure-datacenters-ip-ran

 

What is IP address “169.254.169.254”?

If you monitored in some way network traffic and connections for your Azure VMs, you may have been noticed IP address “169.254.169.254”: what is this? Don’t worry, it is not a malicious attempt to hack or attack your VM. This is a non-routable IP address that is accessible only from inside an Azure VM and is used to communicate with the Azure Host for some very specific scenarios and information retrieval.  

Preview: “What is about to happen to my VM?” In-VM Notification Service

https://azure.microsoft.com/en-us/blog/what-just-happened-to-my-vm-in-vm-metadata-service

Today, this special IP is used to retrieve the planned maintenance events that will touch the VM, as described in the article above, but soon will be used also to retrieve metadata information related to the VM itself. Figure below describes the high level overview of today’s communication framework. The REST Server is the only place the VM can communicate. For the metadata instance server, we use the standard Link-Local addresses i.e. 169.254/16 which is aligned with RFC3927. 3

Does Azure support IPv6?

YES. Azure now provides support for IPv6 public facing IPs, both static and dynamic. Please note that you cannot use IPv6 inside a VNET for VM-to-VM communications: using an IPv6 address as an internal VM IP (DIP) or for ILB is *not* supported. The public IPv6 addresses cannot be assigned to a VM as an instance level IP (ILPIP or PIP). They can only be assigned to the external Azure Load Balancer. It is worth mentioning that currently is not possible to use NSG with IPV6, but access can be restricted using load balancing rules.

Overview of IPv6 for Azure Load Balancer

https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-ipv6-overview

4

Can I connect Azure Virtual Networks without any Gateway?

YES, this features is called “VNET Peering“ and can connect two VNETs without any gateway. Only VNETs in the same region can be peered, mixing ASM and ARM is supported, gateways in the peered VNET can be used by the peering VNET. 5

Is it possible to have multiple NICs for an Azure VM?

YES, this feature has been available since 2014 but with some limitations. What has been recently changed here is that you are now able to add (or remove) additional NICs even after VM creation, the only restriction is that VM must be stopped and deallocated. Additionally, you can mix single and multiple NIC VMs in the same Availability Set (AS), you can assign public IPs and NSGs also to the secondary NICs. Each NIC can support up to 250 IP addresses.  Be aware that adding more NICs will not give you more bandwidth, all will share the same limit granted to the VM level. Maximum number of NICs is depending on the VM SKU and size as documented in the article below:

Sizes for Windows virtual machines in Azure

https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-sizes

6

Can I assign multiple IPs to my single Network Interface Card (NIC)?

YES. An Azure Virtual Machine (VM) has one or more network interfaces (NIC) attached to it. Any NIC can have one or more static or dynamic public and private IP addresses assigned to it. Every NIC attached to a VM has one or more IP configurations associated to it. Each configuration is assigned one static or dynamic private IP address. Each configuration may also have one public IP address resource associated to it. A public IP address resource has either a dynamic or static public IP address assigned to it. The actual limit for assigned IP per NIC is 250, there is also a limit to the subscription level, you can check the article below:

Sizes for Windows virtual machines in Azure

https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-sizes

Be aware that maximum number of NICs is dependent on the VM SKU and size. Finally, network throughput will not increase adding more NICs since they are logical constructs.

7

Is it possible to create custom TAGs for NSG?

Today, you can create a NSG containing “rules” and then assign to subnets (or single NIC). You are limited to one NSG per subnet, and you cannot create nested or linked NSG. Additionally, it is not possible to create custom TAGs containing predefined sets of VMs or IP addresses, and you cannot leverage “system” TAGs to refer to Azure services, like storage or Azure SQLDB, to restrict access. Azure Networking team heard feedbacks around these functionalities and is considering exposing system tags for STORAGE and SQL in the near term. System TAGs are also on the roadmap for future improvements.

Azure Networking Feedbacks Forum

https://feedback.azure.com/forums/217313-networking/category/77468-security-acls-firewalls-intrusion-detection

 

Which is the Public IP what a VM will use to send out network packets to Internet?

In older ASM API Azure days, a VM was always contained in a Cloud Service logical container, and a VIP has been always associated to it. In this scenario, network packets originated from the VM exited Azure network using the Cloud Service VIP as source IP. Making this VIP “reserved”, permitted users to know that IP and configure, for example, VIP white-listing on their on-premises perimeter firewalls. In ARM, there is no more mandatory Cloud Service (and related VIP). When you create a VM inside a VNET, there is no default VIP associated if you don’t specifically ask it. The net result is that if network packets originated from that VM exited Azure network, the VIP used was one from a pool of Azure managed VIPs with no possibility to predict which one and no guarantee that would remain the same. If you want your VM to send packets out to the Internet using a fixed/reserved IP, you need to assign a VIP to the VM, through the Azure Load Balancer, or assign a VIP directly to the VM (PIP or ILPIP). All scenarios are well explained in the article below:  

Understanding outbound connections in Azure

https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-outbound-connections

8

Should I care about Azure Load Balancer capacity for VM outgoing traffic?

While Azure Load Balancer “filter” incoming connections and flows to VNET contained VMs, for outgoing traffic the load balancer is bypassed and network packets reach directly the Internet. This is true independently from the type of VIP, if any, assigned to the VM. If you want to have strict control on outgoing traffic, you need use a NVA and use User Defined Routing (UDR) to force the traffic through it.  

Do Azure VPN or ExpressRoute provide traffic encryption?

Azure VPN tunnels, for Site-to-Site connectivity (S2S), are by design/default encrypted using IPSEC. Encryption is also provided for Point-to-Site (P2S) using SSTP (Secure Socket Tunneling Protocol). ExpressRoute does not provide network traffic encryption for its circuits.

VPN Gateway FAQ

https://azure.microsoft.com/en-us/documentation/articles/vpn-gateway-vpn-faq

  For Azure VPN Gateways, is not possible to customize IPSEC parameters, this is something that should be possible in the future.  

Is it possible to obtain ExpressRoute usage statistics?

YES, it is possible. Azure Portal provides only two simple counters as in the example below, but as you can see they are cumulative and it is difficult to monitor usage over time. 9 To obtain more granular Azure ExpressRoute statistics, you can use the PowerShell script below from GitHub:

Visualize Azure Express Route Statistics

https://github.com/tvuylsteke/azure-ea-powerbi/tree/master/Express%20Route%20Stats%20-%20V1

 

Does Azure Load Balancer support multiple IPs?

YES, Azure Load Balancer supports multiple IPs both in the public facing VIP scenario (SLB) and internal only scenario (ILB). In the first case, it is possible to have multiple public IP (VIPs) assigned to the same external load balancer, and then traffic forwarded to the same set of Virtual Machines included in one single Availability Set. It is also possible to combine this feature with multiple NICs per VM, as described in the article below:

Load balancing on multiple IP configurations using PowerShell

https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-multiple-ip-powershell#steps-to-load-balance-on-multiple-ip-configurations

  The configuration described above is possible since last year, what has been introduced only recently is the multiple IP support for Azure Internal Load Balancer (ILB) using internal IPs (DIPs). After initially creating an ILB with a single internal DIP, you can add more using the steps outlined in section “Example script: Add an IP address to an existing load balancer with PowerShell” in the article below. In short, each IP address requires its own load balancing rule, probe port, and front port. It is worth mentioning that thanks to this feature, it is now possible to deploy multiple SQL Server AlwaysOn Availability Groups (AG) using the same set of Azure VMs.

Configure one or more Always On Availability Group Listeners - Resource Manager

https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sql/virtual-machines-windows-portal-sql-ps-alwayson-int-listener

  NOTE: Today, you can have a load balancer with multiple VIPs backed by a single VM “Backend Pool”, but you cannot use multiple Availability Sets for a single load balancer.

Does ICMP work in Azure?

It depends from where you want to use ICMP tools and which targets you want to reach. ICMP works inside an Azure Virtual Network, between different Azure Virtual Networks connected via VPN or ExpressRoute or hybrid connections between on-premise and Azure VNET. It will *not* work if you have to cross the Azure Load Balancer (SLB). If you need ICMP for diagnostic and troubleshooting, there are valid alternatives to PING and TRACERT not based on ICMP but TCP. My favorite ones, among many others available in Internet, are PSPING from Mark Russinovich SysInternals tools and NMAP:

PsPing v2.01

https://technet.microsoft.com/en-us/sysinternals/jj729731.aspx

NMAP

https://nmap.org/download.html

Inside an Azure VNET, you can use TCP, UDP, and ICMP TCP/IP protocols. Multicast, broadcast, IP-in-IP encapsulated packets, and Generic Routing Encapsulation (GRE) packets are blocked within VNets. Remember: Azure VNET is a Layer-3 virtual network.  

What is Direct Server Return (DSR) for Azure Load Balancer? Should I use it?

“Direct Server Return” (DSR) is probably one of the most unknown and mis-understood parameter in all Azure networking objects configurations, probably because is seldom used and documentation is not good enough. In short, it is a parameter used to enable a special behavior when configuring Azure Load Balancer load-balanced rules, both external and internal ones: when enabled, Load Balancer does not perform any Network Address Translation (NAT) of the request and will pass the incoming packet to the destination VM using the Load Balancer VIP as the destination IP, instead of the VM internal IP (DIP). 10 In the above picture, please note on the right the “Dst: service VIP” label. Conversely, when DSR is disabled (default behavior), “Dst: service VIP” will be replaced by VM internal IP. It is also worth noting that the target VM will respond to the Client directly, bypassing the Load Balancer. Now the second question: should I use it? The answer is NO, if you don’t need it for specific application or service. For sure, you will need if you want to use SQL Server AlwaysOn Availability Group feature, as described in the article below:  

Configure an ILB listener for SQL Server AlwaysOn Availability Groups in Azure ARM

https://blogs.msdn.microsoft.com/igorpag/2016/01/25/configure-an-ilb-listener-for-sql-server-alwayson-availability-groups-in-azure-arm

11 In the last PowerShell command above, DSR is enabled using “-EnabledFloatingIP” parameter. If you want to use DSR for other applications, be careful because the VM will need to receive network packets where the destination IP is not the VM IP but the load balancer VIP. If you don’t have application or service specific support to enable this, as in the SQL Server case, you will need to add a loopback adapter to the VM and manually add the load balancer VIP to it.  

Does Azure support SR-IOV for accelerated networking?

Azure feature currently in preview, and soon broadly available, called “Accelerated Networking” enables “Single Root I/O Virtualization” (SR-IOV) to a virtual machine (VM), greatly improving its networking performance. This high-performance path bypasses the host from the datapath reducing latency, jitter, and CPU utilization for use with the most demanding network workloads on supported VM types.  

Accelerated networking for a virtual machine using the Azure portal

https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-accelerated-networking-portal

12

Can I use Receive Side Scaling (RSS) optimization in Azure VM?

YES. Azure Host hypervisor can support “Receive Side Scaling” (RSS) at the VM level. Azure Virtual Machines have default network settings that can be further optimized for network throughput. Following the instructions in the article below, you can optimize network throughput for both Windows and Linux Guest OS, including major distributions such as Ubuntu, CentOS and Red Hat.

Optimize network throughput for Azure virtual machines

https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-optimize-network-bandwidth

  A VM using Receive Side Scaling (RSS) can reach higher maximal throughput than a VM without RSS. RSS may be disabled by default in a Windows VM, you can easily check following the previous article. For Linux, starting January 2017 this optimization is enabled by default, but you need to use a Linux Integration Services (LIS) version equal or higher than 4.1.3.

Can I manually change protocol and NIC settings at the Guest OS level?

If not explicitly stated in any Microsoft official documentation, manual overriding of network interface or TCP settings at the Guest OS level is not recommended. This may cause loss of connectivity, or severe VM malfunctions. Some settings at the NIC level, like MTU, should be left at default values, in this case 1500. For DNS settings at the TCP level, changing are supported but from external Azure API/Portal/PowerShell/CLI-tool. Changing OS Firewall configuration is safe and supported, as long as you will not close out your remote connection. 21

Does Azure provide SLA for VM network bandwidth?

I tested VM bandwidth limits on almost all VM sizes and found the results pretty consistent over time, then in theory there is nothing preventing you from generating your measure. What Azure documentation provide is a relative measure of network capacity for each VM SKU/SIZE as you can see in the table below:

Sizes for virtual machines in Azure

https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-sizes

If you look at column “Max Network Bandwidth”, you can find tags as LOW, MODERATE, HIGH, VERY HIGH, EXTREMELY HIGH. If you found in Internet, some old tables and information related to maximum network bandwidth for various VM SKUs, you should not trust them since outdated and Azure infrastructure evolved in the meanwhile. The only exceptions are Azure HPC instances where it is exactly specified the type of network adapter provided, that is 1 NIC 32Gbit/s and 1 NIC 10Gbit/s for A8/A9, and 1 NIC 10Gbit/s for A10/A11. See at the link below for more information:

About H-series and compute-intensive A-series VMs for Windows

https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-a8-a9-a10-a11-specs

  If you want to run performance tests on network throughput, be sure to use a tool able to provide multi-threaded connections. For both Windows and Linux, I recommend using NTttcp tool mentioned in the next paragraph and iPerf3 for Linux only.

Does Azure provide SLA on VM-to-VM network latency?

During my tests and projects, Azure network Always provided excellent performances in terms of latency. At least at my knowledge, there is no Public Cloud vendor able to guarantee a maximum network latency, backed up by a formal SLA, between two different VMs. Since you don’t have much control over the Azure network infrastructure, there are few things you can try to optimize not only the latency but networking performances in general:
  • Chose the appropriate VM SKU since different ones have different networking equipment and then different underlying performances. As explained in the previous section, each VM SKU has a classification for networking capabilities, bandwidth is highlighted but also latency is different.
  • Place VMs that need to communicate frequently in the same Azure VNET: be aware that this will not guarantee single hop between all your VMs, with the introduction of regional VNETs, your virtual network can span multiple Azure Host Clusters in the same datacenter.
  • If you want to provide maximum proximity, and ensure VMs will be in the same Azure Host Cluster, you should put relevant VMs in the same Availability Set (AS): be aware that this is not what AS is intended for, this may have impacts on planned maintenance and availability of your VMs.
  • Even if still in preview, the best thing you can test is Azure “Accelerated Networking”, see previous section "Does Azure support SR-IOV for accelerated networking?" to discover how to do that.
  • Even if main improvement is for network throughput, enabling RSS at the Guest OS level, both Linux and Windows, can provide latency improvements, see previous section “Can I use Receive Side Scaling (RSS) optimization in Azure VM?” for more details.
  • For Linux, be sure to use the latest LIS driver: starting version 4.1.3, Microsoft introduced lots of performance improvements and some bug fixing, it’s highly recommended to upgrade this component.
  • If your VMs are in different VNETs connected using VPN or ExpressRoute, and are in the same region, consider using “Virtual network peering” instead.
It’s very easy to test yourself and have a good understanding of average latencies, but you need to be very careful on your test criteria. First of all, you need to test over various hours during the day including peak and non-peak hours, various days over the week including weekends and working days, then finally you should use calculate the average and consider a good percentile sampling near the 95% to eliminate peaks and occasional strange values. Additionally, don’t assume latencies will be the same in all Azure datacenters. If you want to do your own tests, for Windows, I would recommend you PSPING tool from SYSINTERNAL suite that you can find here: https://technet.microsoft.com/en-us/sysinternals/bb896649.aspx. What is nice regarding this tool, is that it will not use ICMP then no problem to traverse firewalls and load-balancers, you can decide which TCP port to use and, very important, you can test for throughput, not only for latency. If you want to do your networking tests using Linux, I would recommend also this tool, highly recommended if you want to test network throughput and performances:

NTttcp Utility: Profile and Measure Windows Networking Performance

https://gallery.technet.microsoft.com/NTttcp-Version-528-Now-f8b12769

NTTTCP-for-Linux

https://github.com/Microsoft/ntttcp-for-linux

Azure backend traffic will go through public Internet?

Azure traffic between our datacenters stays on our network and does not flow over the Internet. This includes all traffic between Microsoft services anywhere in the world. For example, within Azure, traffic between virtual machines, storage, and SQL communication traverses only the Microsoft network, regardless of the source and destination region. Intra-region VNet-to-VNet traffic, as well as cross-region VNet-to-VNet traffic, stays on the Microsoft network., see the article below, authored by Yousef Khalidi (CVP, Azure Networking) for great and comprehensive details:

 How Microsoft builds its fast and reliable global network

 https://azure.microsoft.com/en-us/blog/how-microsoft-builds-its-fast-and-reliable-global-network

Microsoft owns and operates its dark fibers to support WAN network connecting 38 Azure regions (at the time of writing this article) and 100s of datacenters. Customer traffic enters our global network through strategically placed Microsoft edge nodes, our points of presence (PoP). These edge nodes are directly interconnected to more than 2,500 unique Internet partners through thousands of connections in more than 130 locations. 13

Can I white-list Azure public IP addresses?

YES, all Azure public IP ranges are published at the link below. If you want to white-list, for security reasons, on your on-premise network you can do that selectively by region but not by service.

Microsoft Azure Datacenter IP Ranges

https://www.microsoft.com/en-us/download/details.aspx?id=41653

14 From the link above you can download an XML file, divided in sections per regions but there is no way currently to retrieve the same information using an API. Azure Networking team is working to address this issue:

Provide a rest api to access the list of Azure IP Addresses

https://feedback.azure.com/forums/217313-networking/suggestions/7438096-provide-a-rest-api-to-access-the-list-of-azure-ip

If you rely on specific Azure resources, for example an Azure SQLDB instance or blob storage account, be aware that the assigned IP you will see, resolving their specific DNS names, may change without notice. IMPORTANT: don’t rely on any geo-location service for Internet IP used by Azure since may report incorrect information, for more details see the blog post below:

Microsoft Azure’s use of non-US IPv4 address space in US regions

https://azure.microsoft.com/blog/2014/06/11/windows-azures-use-of-non-us-ipv4-address-space-in-us-regions

Be aware that the internet facing public IP (VIP) associated to your resources may change under some conditions, if you need to be sure it will never happen, you need to enable "Reserved IP" feature as described at the blog post below:

IP address types and allocation methods in Azure

https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-ip-addresses-overview-arm

If you still need to use legacy Cloud Services, see the article below on how to use Reserved IP using ASM API:

Reserved IP addresses for Cloud Services & Virtual Machines

https://azure.microsoft.com/blog/2014/05/14/reserved-ip-addresses

What is “*.internal.cloudapp.net” DNS suffix in Azure VM?

If you build an Azure VM and join to an Azure Virtual Network and try to ping it using the VM host name (legacy Netbios name to be clear), this is what you will see: 15 As you can see in the picture, there are two strange parts in the FQDN: nothing strange in “cloudapp.net”, but what are “internal” and “a3”? You can easily guess that “internal” is the default DNS sub-zone that Azure internal DNS (iDNS) will use to host records for VMs and resolve to internal VM IP (DIP). “a3” instead is a little bit complex: in short, it’s describing the network zone, inside the specific Azure datacenter, where the VM is allocated. There is no official documentation or list of these zones, then don’t ask more, but please note that it will change not only between Azure datacenters, but also inside a single datacenter. You can programmatically retrieve internal DNS setting using the simple PowerShell command below:

(Get-AzureRmNetworkInterface -Name "mynicname" -ResourceGroupName $rgname).DnsSettings.InternalFqdn

Output will be something like the string below: InternalDomainNameSuffix : 0fcdtenvgl4urm0sv5aj1xkcrc.fx.internal.cloudapp.net

 Which protocols Azure Load Balancer supports?

Azure Load Balancer, for both external and internal connections, can support only TCP (Protocol ID “6”) or UDP (Protocol ID “17”). It cannot support protocols like ICMP (Protocol ID “1”). As an example, also IPSec (and VPN using it) is not supported since you should open UDP port 500 (that is fine) and permit IP protocol numbers 50 and 51. UDP Port 500 should be opened to allow Internet Security Association and Key Management Protocol (ISAKMP) traffic to be forwarded through Azure Load Balancer. IP protocol ID 50 should be set to allow IPSec Encapsulating Security Protocol (ESP) traffic to be forwarded. Finally, IP protocol ID 51 should be set to allow Authentication Header (AH) traffic to be forwarded. As additional note, since this would be interesting for some scenarios, Azure internal load balancer (ILB) does *not* support today “any-protocol/any-traffic”

Protocol Numbers

https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml

How many connections a VM can support?

For Windows Server, a VM can support about 500k TCP connections but you need to be careful about other potential limits that may come into the game before this threshold: if you expose your VM to Internet traffic through an Azure Load Balancer endpoint, you may be limited by SLB capacity or DDOS security mechanisms. Actually, there is no public documentation on SLB or Endpoint limits, neither on the number of connection nor network bandwidth, then you should conduct your own performance tests to ensure everything will work correctly.  

Can I have multiple VNET Gateways?

Currently you can only have one virtual network Gateway per single VNET per single type, that is one for VPN and one for ExpressRoute. With VPN, if you configure multiple tunnels, they will share the same maximum network bandwidth. If you need more throughput for a VPN connection, you may want to consider using Network Virtual Appliances (NVA) or 3rd-party VPN software like OpenSwan and StrongSwan with big enough Azure VM SKUs/sizes.  

ExpressRoute and VPN can coexist in the same VNET?

YES, it is possible to have a VNET with gateways supporting ExpressRoute and VPN at the same time. The only requirement is to enable ExpressRoute before VPN. One Gateway per VNET restriction persists.

New Networking Capabilities for a Consistent, Connected and Hybrid Cloud

https://azure.microsoft.com/blog/2015/05/05/new-networking-capabilities-for-a-consistent-connected-and-hybrid-cloud

16

Do I need a Gateway to connect my VNET to ExpressRoute?

YES, if you want to use ExpressRoute (ER) to connect your VNET, you need to use and configure “Private Peering” routing domain, then you need to deploy a Virtual Network Gateway inside your Azure Virtual Network. This implies two important facts:
  • There are several Virtual Network Gateway sizes that can be used with ER circuits: be sure to use the right size (1000, 2000, 10000 Mbps) able to support your VNET traffic requirements, otherwise you will not be able to use all the bandwidth allocated to your circuit (50 Mbps, 100 Mbps, 200 Mbps, 500 Mbps, 1 Gbps, 10 Gbps).
17
  • Virtual Network Gateway, depending on the type, and ER circuit have different SLA, be sure to review this based on your requirements. We guarantee a minimum of 99.95% ExpressRoute Dedicated Circuit availability, and between 99,90% and 99,95% for the Gateway:

18

SLA for ExpressRoute

https://azure.microsoft.com/en-us/support/legal/sla/expressroute/v1_2

SLA for VPN Gateway

https://azure.microsoft.com/en-us/support/legal/sla/vpn-gateway/v1_2

Can I connect VNETs inside and outside a region?

YES, Azure provides three ways to connect VNETs: ExpressRoute, VPN and Virtual Network Peering. The last one has been recently released and permits to connect two VNETs in the same region without any gateway as for the other options. For this feature, there is a nominal fee based on the amount of GB transferred between the peered VNETs.

Virtual Network Peering

https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-peering-overview

19 With VPN, there is no boundary on the region, geography or subscription, essentially the tunnel only need Gateway VIPs to see each other. In case you need more throughput and lower latency, you can eventually use ExpressRoute technology to connect up to 100 VNETs and share circuits between them, even with different subscriptions. Before Premium add-on announced in May 2015, ExpressRoute posed limitations on geographical boundaries: now it is possible not only to connect VNETs in the same geography (US, Europe, etc.) but also across different geographies. For example, you can connect a VNET in West US to a VNET in North Europe.

New Networking Capabilities for a Consistent, Connected and Hybrid Cloud

https://azure.microsoft.com/blog/2015/05/05/new-networking-capabilities-for-a-consistent-connected-and-hybrid-cloud

Does Azure provide DDoS network protection?

YES, Azure infrastructure is designed to protect the network from DDOS originating from the Internet and also internally from other tenants VMs, you can read the details in the white-paper below:

Microsoft Azure Network Security Whitepaper version 3 is now available

https://azure.microsoft.com/en-us/blog/microsoft-azure-network-security-whitepaper-version-3-is-now-available

 Please be aware of the following important points:
  • Windows Azure’s DDoS defense system is designed not only to withstand attacks from the outside, but also from within.
  • Windows Azure monitors and detects internally initiated DDoS attacks and removes offending VMs from the network.
  • Windows Azure’s DDoS protection also benefits applications. However, it is still possible for applications to be targeted individually. As a result, customers should actively monitor their Windows Azure applications.
To augment your tenant security, you can deploy 3rd-party Network Virtual Appliances (NVA) in your Virtual Network, but be aware that custom DDoS to protect Azure public IPs (VIPs) is not supported.

Are Broadcast and Multicast supported within Azure Virtual Network?

This type of communication is not allowed inside a VNET or even across Azure Load Balancer.

Support Multicast within Virtual Networks

https://feedback.azure.com/forums/217313-networking-dns-traffic-manager-vpn-vnet/suggestions/3741233-support-multicast-within-virtual-networks

You can use TCP, UDP, and ICMP TCP/IP protocols within VNets. Multicast, broadcast, IP-in-IP encapsulated packets, and Generic Routing Encapsulation (GRE) packets are blocked within VNets.

Is it possible to run network penetration tests against my Azure VMs?

YES, you can do that but it’s highly recommended to follow the specific procedure mentioned below, before running the test, otherwise Azure monitoring and defense systems will trigger in and blacklist your connections, IPs and/or VMs. Be sure your request will be approved before starting the test, you can find the form to fill-in at the bottom of the page URL reported below:

Penetration Test Notification Process

https://security-forms.azure.com/penetration-testing/terms

20   Microsoft conducts regular penetration testing to improve Azure security controls and processes. We understand that security assessment is also an important part of our customers' application development and deployment. Therefore, we have established a policy for customers to carry out authorized penetration testing on their applications hosted in Azure. Because such testing can be indistinguishable from a real attack, it is critical that customers conduct penetration testing only after notifying Microsoft. Penetration testing must be conducted in accordance with our terms and conditions.  ]]> Azure Managed Disks Deep Dive, Lessons Learned and Benefits https://docs.microsoft.com/archive/blogs/igorpag/azure-managed-disks-deep-dive-lessons-learned-and-benefits Tue, 14 Mar 2017 11:42:02 GMT https://blogs.msdn.microsoft.com/igorpag/?p=736 [bing_translator]

Introduction

Managed Disks have been recently released for production (February 2017), and represent an important step to make Azure VM management much easier. In short, Azure Managed Disks feature is an abstraction built on top of current Standard and Premium storage disk in Azure Storage. Customers only need to specify the type (Standard or Premium) and size of disk they need in their selected Azure region, and Azure will create and manage the Disk accordingly. To be clear, Azure Storage Accounts are *not* going away yet, simply Azure will take care of their management on your behalf. Additionally, what will *not* change with MD is the maximum number of disks you can attach to your VM: same maximums listed in the article below apply, doesn’t matter if MD or Un-Managed (UMD - for brevity), what rules the ratio here is the number of cores of your VM.

Sizes for Windows virtual machines in Azure

https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-sizes

Finally, just to complete the picture of what remains the same, let me include disk size in this discussion: actually, in Azure the maximum size of disks you can create for a VM is 1TB, this is a cap originating from the maximum size of “Page Blobs” in the Azure Storage. Bigger disk sizes should come in the future, but it’s not directly tied to the MD concept and mechanism.

Benefits

Then, after the bad news related to what remains the same, what is changing and why you should care about MD? In some of my recent engagements for Microsoft Global ISVs, I had the opportunity to test in depth this new Azure feature, as a key feature requested for this project to go in production. Why lack of MD was a hard blocker for this partner? Well, MD presents some unique capabilities and important differences compared to traditional UMD. This is the prioritized list I built looking to my recent engagements:
  • Remove or relax Azure storage limits: No more space or IOPs or storage bandwidth limits.
  • Disks as first-class objects: provide CRUD operations, and security management, for disk objects independently from VM life-cycle.
  • Image Sharing: No more image copy to target storage accounts, reference images between different storage accounts.
  • Higher VM resiliency: Align Fault Domains (FD) concept for Storage with Compute.
  • Independent Snapshots: creating a disk snapshot and then delete the source disk while retaining the generated snapshot.
  • Instant copy: copy inside and across storage accounts must be instantaneous in the same region.
 I’m going to explain you in details the above key benefits and important details you will need to keep in mind, but let me start with two obvious things: MD is ARM (Azure Resource Manager) feature only, no plans to back-port for ASM (Azure Service Management), and NO… Microsoft is not retiring UMD, even if this is the default in the Azure Portal and recommended going forward. 1 Then, you don’t need to use UMD anymore? Well, not exactly. There are few situations where you may still need to use them for Azure VMs:
  •  VM Diagnostics: for diagnostic and boot logs, you still need to use “legacy” storage accounts, there is a plan to remove this in the future, but for the moment you have to do in the same old way. You can use MD for your VM OS and Data disks, but you will also need to specify a storage account where this information will be landed.
  • Storage Replication: actually, MD are Locally Replicated Storage (LRS) only, if you need GRS for example, you need to use UMD.
2 For a complete description of Storage Resiliency options, please consult the article below:

Azure Storage replication

https://docs.microsoft.com/en-us/azure/storage/storage-redundancy

NOTE: I will *not* cover here Azure Virtual Machine Scale Set (VMSS) enhancements due to Managed Disks, but if interested you should read the article below:

Azure VM scale sets and managed disks

https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-managed-disks   

  • You no longer need to pre-create and manage storage accounts to store the OS disks for the scale set VMs.
  • You can attach managed data disks to the scale set.
  • With managed disk, a scale set can have capacity as high as 1,000 VMs if based on a platform image or 100 VMs if based on a custom image.
 

Remove or Relax Azure Storage limits

Based on my partners and customer’s feedbacks, one of most annoying Azure consideration is the variety of caps we have on storage: both in ARM and ASM, we have limits on the maximum number of storage accounts, maximum size for storage accounts, IOPS and storage bandwidth limits. If you have to manage many VMs, my experience here is in the order of 1000s, you will be forced to do some calculations and build a logic in your deployment engine to keep track of disks you deployed and in which storage accounts. On Standard storage, even if you have 500TB space, you are limited to 20K IOPS, then you should not place more than 40 disks on single account, since each one will consume at the maximum 500 IOPS. This is even more difficult if you use Premium Storage since have different limits (35TBs vs. 500TBs, etc.) and recommendations on how to allocate VMs compared to Standard Storage. Here the calculation you have to do is about disk allocated size. Indeed, this complexity does not exist on some other Public Cloud providers. With MD, now all these limits are gone. Something remains but I’m sure will be reasonable for your you:
  • Maximum number of disks you can attach per VM doesn’t change, as I mentioned in the introduction, since not dependent on MD feature.
    • Max IOPS Per VM limit is still 80,000 IOPS with GS5, max throughput per VM is still 2,000 MB/s with GS5 (may change in the future)
    • Also, maximum disk size doesn’t change (yet).
  • There is a default soft-limit, that can be increased, on the maximum number of MD: it is 2000, is per subscription and per region, but can be increased opening a ticket with Azure Support.
    • In one specific case I was able to obtain 30K disks.
    • It is worth noting that this limit includes not only MD disks, but also the snapshots you create  for these.
 You can create MD using either Standard or Premium storage, as with UMD, but there is an important difference: now, with MD, including using Standard storage, you have a set of fixed sizes at disk creation time. This does not mean that you cannot create a disk of arbitrary size, it simply says that you will pay for closest rounded high tier limit for declared size: for example, if you create a 200GB Standard MD, you will pay for S20, based on the table below. This may change in the future. 3

Table 1.1

These are the two key articles to consult for more details for this context discussion:  

Azure subscription and service limits, quotas, and constraints

https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits

 

Azure Storage Scalability and Performance Targets

https://docs.microsoft.com/en-us/azure/storage/storage-scalability-targets

 

Disks as First-Class objects

Some of my partners never noticed in the past, but in ARM you were not able to create a disk as a standalone object: without MD, you cannot create, for example, an empty disk of 256GB and attach later to one of your VMs. You must import from a blob or create and attach as a unique operation at VM creation time. This is a simple code snippet that shows empty MD creation and, later on, check if deployment “ProvisioningState” is “Succeeded”:

$mdiskconfig1 = New-AzureRmDiskConfig -AccountType StandardLRS -Location $location -DiskSizeGB 256 -CreateOption Empty

$mdisk1 = New-AzureRmDisk -ResourceGroupName $rgname -Disk $mdiskconfig1 -DiskName "EmptyManagedDiskMD1"

$ref1 = (Get-AzureRmDisk -ResourceGroupName $rgname -DiskName "EmptyManagedDiskMD1")

Write-Host ('Disk Provisioning State -> [ ' + ($ref1.ProvisioningState) + ' ]')

Note that MD PowerShell cmdlets are included in the "AzureRM.Compute" module, not in the "AzureRM.Storage". This should not be a surprise if you think carefully about what is MD, that is an abstraction on top of Azure Storage. Let me add another interesting fact: you don’t specify, and you don’t see, any VHD URL. Azure MD resource provider will manage this for you, simply specify the disk you want, in terms of size and type (Standard or Premium), and Azure will create where you want it with a specified name. If you want, you can get access to the underlying URL, sometimes maybe necessary for example for copy operations, here is the API or PowerShell cmdlet you can use, secured using SAS signature mechanism: 4

Grant-AzureRmDiskAccess -ResourceGroupName $rgname -DiskName 'EmptyManagedDiskMD1' -Access 'Read' -DurationInSecond 60

OUTPUT (scrambled to hide real data):

https://md-grkhtgdfazmdr.blob.core.windows.net/ldc1j5nxwxzg/abcd?sv=2014-02-14&sr=b&si=9049d2762-524e-4911-a953-9ca72e0ed256&sig=TFl2jGt8FkkC1Xi970QcfxvLmepqW3vxORPQWU12bjI%3D

The output is also pretty interesting: storage account, blob and container names are auto (random?) generated, and the URL starts with a “md” prefix. You are still not convinced, here is the REST API to create a single MD object (I included “Snapshot” for completeness), please note the “path” used in the URL: 5 Then, just to summarize, you can still create a Managed Disk (MD) in implicit mode at VM creation time, but you can now also create it in explicit mode, using one of these methods:
  • Copy of a Platform Image Repository (PIR) image disk
  • Import of a VHD from a public storage blob
  • Creation of an empty disk
  • Copy of an existing disk or snapshot
At the end, Azure finally provides full CRUD operations for Managed Disks, you can also resize (only up!), delete and create snapshots on them:

# Resize of the disk: #

$ref1.DiskSizeGB = 512

Update-AzureRmDisk -DiskName $ref1.Name -Disk $ref1 -ResourceGroupName $rgname

$ref1 = (Get-AzureRmDisk -ResourceGroupName $rgname -DiskName "EmptyManagedDiskMD1")

$ref3.DiskSizeGB

$ref3.ProvisioningState

# Create a snapshot on the Managed Disk: #

$mdisksnapshotconfig1 = New-AzureRmSnapshotConfig -AccountType StandardLRS -DiskSizeGB 512 -Location $location -SourceUri $ref1.Id  -CreateOption Copy

$mdisksnapshot1 = New-AzureRmSnapshot -ResourceGroupName $rgname -SnapshotName "EmptyManagedDiskMD1snaphot1" -Snapshot $mdisksnapshotconfig1

Finally, let me include the diagram below, I’m sure will definitely clarify the relationship between MD main object type and the other objects that I’m going to cover later in this post, that is "Snapshots" and "Images". 6

Image Sharing

This functional aspect was very important for my partners, due to the nature of how VM images are built. What I have seen pretty frequently is the requirement to build images in a central Azure subscription, then deploy VMs in other subscriptions, using several/many storage accounts. It doesn’t matter if you use Standard or Premium storage account, you had to copy the image in the target storage account before being able to create a VM from there. Now with Managed Disks (MD) this is not necessary anymore, you can create VMs in a subscription using an image (MD-based) that can reside in any storage account of the same subscription. But what happens if you need to copy your image across subscriptions? At the moment, Managed Disks (MD) don’t provide this feature, but I would be not surprised to hear important updates in the near future. In the meanwhile, a practical workaround to use includes these simple steps: get Image underlying URL/SAS, use tools (AzCopy) or API/PowerShell to copy the blob to a target storage account in the target subscription, then create an image in the target subscription using that blob and its URL. In the example below, I realized a slightly different scenario, that is I’m starting from a VM (generalized) in a source subscription to create an image in the target subscription.  

# This is the source VM for the Image copy and other necessary parameters: #

$VMName = "<<VM Name>>"

$sourcergname = "<<source resource group>>"

$targetrgname = "<<destination resource group>>"

$ImageName = "<<image name>>"

$storageacccountname = "<<storage account name>>"

$containername = "images"

$VM = Get-AzureRmVM -ResourceGroupName $sourcergname -Name $VMName

$OSDiskName = $VMName + "-osDisk"

$OSDisk = Get-AzureRmDisk -ResourceGroupName $sourcergname -DiskName $OSDiskName

 

# Stop VM, Generalize and get access to OS Managed Disk's underlying blob URL: #

Stop-AzureRmVM -ResourceGroupName $sourcergname -Name $VMName -Force

Set-AzureRmVm -ResourceGroupName $sourcergname -Name $VMName -Generalized

$mdiskURL = Grant-AzureRmDiskAccess -ResourceGroupName $sourcergname -DiskName $VM.StorageProfile.OsDisk.Name -Access Read -DurationInSecond 3600

 

# Now copy out the Managed Disk using SAS as a "normal" VHD in a "normal" target storage account #

$storageacccountkey = Get-AzureRmStorageAccountKey -ResourceGroupName $targetrgname -Name $storageacccountname

$storagectx = New-AzureStorageContext -StorageAccountName $storageacccountname -StorageAccountKey $storageacccountkey[0].Value

$targetcontainer = New-AzureStorageContainer -Name $containername -Context $storagectx -Permission Blob

$targetcontainer = Get-AzureStorageContainer -Name "images" -Context $storagectx

$destdiskname = $VM.StorageProfile.OsDisk.Name + ".vhd";

$sourceSASurl = $mdiskURL.AccessSAS

$ops = Start-AzureStorageBlobCopy -AbsoluteUri $sourceSASurl -DestBlob $destdiskname -DestContainer $targetcontainer.Name -DestContext $storagectx

Get-AzureStorageBlobCopyState -Container $targetcontainer.Name -Blob $destdiskname -Context $storagectx -WaitForComplete

$sourceimagediskBlob = Get-AzureStorageBlob -Context $storagectx -Container $targetcontainer.Name -Blob $destdiskname

$sourceimagediskURL = ($sourceimagediskBlob.Context.BlobEndPoint) + $containername + "/" + $destdiskname;

 

# Finale step: create image from local copied VHD: #

$targetimagecfg = New-AzureRmImageConfig -Location $location

Set-AzureRmImageOsDisk -Image $targetimagecfg -OsType 'Linux' -OsState 'Generalized' -BlobUri $sourceimagediskURL;

$targetimage = New-AzureRmImage -Image $targetimagecfg -ImageName $ImageName -ResourceGroupName $targetrgname

Now that you have your “golden” image in your target subscription, and in the specific region, you can create as many VMs you want, without the need to copy into all target storage accounts. For the cross-subscription and cross-account copy, PowerShell makes available the most efficient API: “Start-AzureStorageBlobCopy” is extremely fast and support parallel copy by defaults using multiple threads. As an alternative, you may also use AzCopy tool, the underlying REST API is the same. One final note: In this specifc sample, be sure to follow generalization steps specific for your VM Guest OS (see below), then stop your VM.  

Generalize a Windows virtual machine using Sysprep

https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-generalize-vhd

How to generalize and capture a Linux virtual machine

https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-linux-capture-image

  Be careful before deciding to drop the original disk, or snapshot or VM, you used to create your image: image is a regional-scoped object, then if you want to create the image in a new region, at least today, you have to copy manually the VM, disk or snapshot to the target region, then create a new image object based on that source. Name is not mandatory to be the same, but if you want your code to run without modification, you need to use the same one. Process is pretty simple, for example using Snapshots: you need to get access to the underlying blob objects via SAS token, then copy to the target destination region as a blob, then create a new image using the same name (optional) using the copied snapshot. Code sample (not complete) is reported below:

$TargetStorageAcccountkey = Get-AzureRmStorageAccountKey -ResourceGroupName $rgname -Name $TargetStorageAcccountName

$TargetStorageCTX = New-AzureStorageContext -StorageAccountName $TargetStorageAcccountName -StorageAccountKey $TargetStorageAcccountkey[0].Value

New-AzureStorageContainer -Name $TargetContainerName -Context $TargetStorageCTX -Permission Container

$Snapshot = Get-AzureRmSnapshot -ResourceGroupName $rgname -SnapshotName $SourceSnapshotName

$SnapshotSASurl = Grant-AzureRmSnapshotAccess $rgname -SnapshotName $SourceSnapshotName -DurationInSecond 3600 -Access Read

Start-AzureStorageBlobCopy -AbsoluteUri $SnapshotSASurl.AccessSAS -DestContainer $TargetContainerName -DestContext $TargetStorageCTX -DestBlob $DestBlob

$TargetImageConfig = New-AzureRmImageConfig -Location $TargetLocation

$osDiskVhdUri = ($TargetStorageCTX.BlobEndPoint + "$TargetContainerName" + "/" + $DestBlob)

Set-AzureRmImageOsDisk -Image $TargetImageConfig -OsType 'Linux' -OsState 'Generalized' -BlobUri $osDiskVhdUri

New-AzureRmImage -Image $TargetImageConfig -ImageName $ImageName -ResourceGroupName $rgname

(Get-AzureRmImage -ResourceGroupName $rgname) | select Name,Location,ProvisioningState

7

  8  

Higher VM resiliency

This is my favorite improvement, long waited for several of my partners. To better explain this, let me start with a bit of history to set the context. In the “old” ASM days, Azure guaranteed (and do it also today) only two “Fault Domains” (FD). Only with the introduction of ARM, is possible to have guaranteed three FDs. Then, what MD provides on top of ARM? While you are reading here, I would suggest to also give a quick look at the picture below: essentially, without MD you don’t have control over placements of your disks. Traditional Un-Managed Disks (NMD) reside in storage accounts, even if you use different ones for different VMs, there is no guarantee that these accounts (and contained disks) will be placed in different Azure Storage “Storage Units” (or “Stamps”). MD resolves this problem for you: since you don’t have to care anymore about storage accounts, MD will ensure (if requested) that your VMs disks will be placed in different “Storage Units”, then maximizing the resiliency for your deployment. Let me be clear here: for what is concerning Azure VM High-Availability SLA, nothing is changing here, you still have 99.95% for multiple VMs (>1) inside a specific Availability Set, and 99.90% for single instances using Premium Storage. This is an enhancement for resiliency, not a fix for a bug of our previous infrastructure behavior. 9

NOTE: The number of managed disk fault domains varies by region - either two or three managed disk fault domains per region.

Once you created Availability Set (AS)  as “aligned”, you will be allowed to put in there only VMs that are using MD, mixing with UMD VMs will return an error. It is also worth mentioning that, at the moment, it is not possible to change AS for a VM after creation, this is something that may change in the future, but today this is the status. Be careful with MD defaults: in order to have this alignment between Compute and Storage FDs, you have to use explicitly a special crafted AS object that requires a specific parameter in its creation, this is shown in the PowerShell cmdlet below.

# Un-Managed Availability Sets (AS): #

New-AzureRmAvailabilitySet -Location "West Europe" -Name "MyNewUnManagedAS1" -ResourceGroupName $rgname -PlatformFaultDomainCount 3 -PlatformUpdateDomainCount 20

New-AzureRmAvailabilitySet -Sku "Classic" -Location "West Europe" -Name "MyNewUnManagedAS2" -ResourceGroupName $rgname -PlatformFaultDomainCount 3 -PlatformUpdateDomainCount 20

# Managed Availability Sets (AS): #

New-AzureRmAvailabilitySet -Managed -Location "West Europe" -Name "MyNewManagedAS3" -ResourceGroupName $rgname -PlatformFaultDomainCount 3 -PlatformUpdateDomainCount 20

New-AzureRmAvailabilitySet -Sku "Aligned" -Location "West Europe" -Name "MyNewManagedAS4" -ResourceGroupName $rgname -PlatformFaultDomainCount 3 -PlatformUpdateDomainCount 20

# Check if AS is "managed" or not in two different ways: #

$rg1 = (Get-AzureRmAvailabilitySet -ResourceGroupName $rgname -Name "MyNewUnManagedAS1")

Write-Host ("Is [" + $rg1.Name + "] Availability Set Managed? " + $rg1.Managed +"...")

$rg2 = (Get-AzureRmAvailabilitySet -ResourceGroupName $rgname -Name "MyNewManagedAS4")

Write-Host ("How [" + $rg2.Name + "] Availability Set is? " + $rg2.Sku +"...")

  As you can see from the code snippet above, there are multiple ways to achieve the same configuration:
  • By default, a new Availability Set (AS) is created as standard type, that is no alignment of Storage FDs with Compute FDs (first command).
  • If you want to do the same as above, but specifying explicitly, you can do specifying value “Classic” for parameter “-Sku” (second command).
  • The new type of AS can be created specifying “-Managed” flag (third command).
  • Same configuration as the previous one can be obtained specifying “Aligned” for the parameter “-Sku” (fourth command)
At the end, if you want to create a new Availability Set for VMs with Managed Disks that you want “aligned”, you can either specify “-Managed” or “-Sku Aligned”, but the latter is recommended since the former may be removed in the future. This is how it looks like now using Azure PowerShell module version 2.7.0 (February 2017) for AzureRm.Compute:

10

                    For more information on Azure Virtual Machines and availability, you can reach the URL below:

 Manage the availability of Windows virtual machines in Azure

https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-manage-availability

Higher VM resiliency – Data Disks

Same approach described for OS disks is valid also for Data Disks. You don’t have to worry about that: MD mechanism will take care of segregating storage accounts used to host Data Disks in different Storage Stamps, then aligning with Compute Fault Domains (FDs). Best practices in use before MD will remain valid: keep all Disks (OS and Data) associated with a VM in the same Storage Account, but in this case Azure will enforce it behind the scenes. It is worth mentioning that if you add a MD Data Disk later, after VM creation time, Azure will ensure to place in the correct Fault Domain (FD), eventually moving it to a different managed storage account. In this way, resiliency will not be diminished introducing a dependency on an object in a different FD. 20

Independent Snapshots

Snapshot of a MD is something totally different from what you may know already for classic UMD. Aligning the design with what other cloud providers do, Snapshots here are totally independent from the MD object: once created, you can delete the source MD and the Snapshot will survive. This new paradigm is more consistent with the concept of a read-only copy of some another object. You can create as many snapshots as you want on the same MD, I tried 500 personally, there is no hard limit as far as I know, but without noticing performance degradation of the source MD. This makes Snapshots suitable for Point-in-Time (PIT) backups of MD. Keep in mind that you will pay for the snapshot size as an allocated MD space, finally at the moment Snapshots on MD are not incremental but fully copy, this is something that may change in the future. Regarding resource allocation limit, there is no specific Snapshots limit per region per subscription, they are counted against the same cumulative quota with MD as I wrote previously. Since Snapshots are independent objects, sourced with a copy operation from MD, you can have a Snapshot created on Premium Storage originating from a MD allocated on Standard Storage, and vice-versa, if you don’t specify a storage type, Standard will be used. They also support encryption at the disk level, but not Storage Service Encryption (SSE) for Data at Rest, this will come in the future. These are the typical operations that you could do with MD Snapshots, in relationships with the other object types (VM, Image, Managed Disk, VHD/Blob): 
  • Create a Snapshot from a MD
  • Create a MD from a Snapshot
  • Create a Snapshot from a VHD/Blob
  • Create an Image from a Snapshot
  • Create a Snapshot as copy of another Snapshot
Creating Snapshots using REST API or PowerShell is extremely easy, as with all the available SDKs, also in this case the specific Resource Provider is under the Compute umbrella, not the Storage one.

$mdisksnapshotconfig1 = New-AzureRmSnapshotConfig -AccountType StandardLRS -DiskSizeGB 512 -Location $location -SourceUri $ref1.Id  -CreateOption Copy

$mdisksnapshot1 = New-AzureRmSnapshot -ResourceGroupName $rgname -SnapshotName "EmptyManagedDiskMD1snaphot1" -Snapshot $mdisksnapshotconfig1

11 Like Managed Disks, also for Snapshots you have full support for all CRUD operations, including getting access to underlying Blob/VHD that is normally hidden to users: in PowerShell, you can use Grant-AzureRmSnapshotAccess and Revoke-AzureRmSnapshotAccess to obtain SAS URL to eventually copy the object. During my tests and project work, the only time I had to use underlying URL, was when I had to explicitly copy the Snapshot across subscriptions, this is something that is not provided (yet) by the platform, this may change in the future. One final note about performances: even if a Snapshot is  full copy (read-only) of source Managed Disk (MD), copy operation is extremely fast, almost instantaneous, see next sections for details.

Instant copy

This is really a hidden gem in MD. Almost every copy operation involving MD or Snapshot is pretty instantaneous, completely different behavior compared to traditional UMD. Azure team here implemented some kind of “copy-on-read” mechanism that will asynchronously populate bits in the target object, but will return fast success acknowledge of the operation. We used this extensively during our projects, the only current limitation, not so bad for us, is tied to the locality of the copy that must happen in the same region. Again, this optimization is valid for both MD and Snapshots. Another big value of this of this mechanism, is that you can create a MD or a Snapshot based on a Blob resident in a storage account in another subscription, if the target subscription is under the same Azure Active Directory tenant. In our case this was extremely useful because Azure automatically copied in background the source blob using the super-fast “copy-on-read” feature. If you have scenarios where you build images and VMs in a “source” subscription and then you deploy running VMs in a “target” subscription, then this will be very useful for you. Remember: you can create an image from a VHD/Blob or Snapshot, not only directly from a VM.

Syntax Difference: Managed vs. Non-Managed

This is an area where you need to force yourself to pay close attention reading the code. Surprisingly, if you look at PowerShell syntax for example, there is no special cmdlet or parameter to create a VM based on a MD for the OS. Here is a code snippet to create a VM using MD for the OS disk (some parts removed for brevity): 

$sourceimage = Get-AzureRmImage -ResourceGroupName $rgname -ImageName $ImageName

$VirtualMachine = New-AzureRmVMConfig -VMName $VMName -VMSize $VMSize -AvailabilitySetId $as.Id

$VirtualMachine = Set-AzureRmVMOperatingSystem -VM $VirtualMachine -Linux -ComputerName $ComputerName -Credential $Credential

$VirtualMachine = Add-AzureRmVMNetworkInterface -VM $VirtualMachine -Id $nic.Id -Primary

$VirtualMachine = Set-AzureRmVMSourceImage -VM $VirtualMachine -Id $sourceimage.Id

$VirtualMachine  = Set-AzureRmVMOSDisk -VM $VirtualMachine -Name $OSDiskName -StorageAccountType StandardLRS -DiskSizeInGB 128 -CreateOption FromImage -Caching ReadWrite

$VM = New-AzureRmVM -ResourceGroupName $rgname -Location $Location -VM $VirtualMachine

Are you able to see where is the MD indication for this VM OS disk? The answer is not in what is here, but in what is *not* here. Here is the correspondent syntax using UMD: 

$sourceimage = Get-AzureRmImage -ResourceGroupName $rgname -ImageName $ImageName

$VirtualMachine = New-AzureRmVMConfig -VMName $VMName -VMSize $VMSize -AvailabilitySetId $as.Id

$VirtualMachine = Set-AzureRmVMOperatingSystem -VM $VirtualMachine -Linux -ComputerName $ComputerName -Credential $Credential

$VirtualMachine = Add-AzureRmVMNetworkInterface -VM $VirtualMachine -Id $nic.Id -Primary

$VirtualMachine = Set-AzureRmVMSourceImage -VM $VirtualMachine -Id $sourceimage.Id

$VirtualMachine  = Set-AzureRmVMOSDisk -VM $VirtualMachine -Name $OSDiskName -VhdUri $osDiskUri -StorageAccountType StandardLRS -DiskSizeInGB 128 -CreateOption FromImage -Caching ReadWrite

$VM = New-AzureRmVM -ResourceGroupName $rgname -Location $Location -VM $VirtualMachine 

As you can see in the highlighted lines, when you use MD, there is no “-VhdUri” parameter to supply and value to specify: simply tell Azure that you want a (managed) disk and you will get it, forget URI and everything related to storage accounts! The story is a little bit different if want to attach a data disk to a VM. In this case, there is a *single* parameter to specify, that is the ID of the previously created MD you want to attach (-ManagedDiskId) : 

$MDdiskConfig = New-AzureRmDiskConfig -AccountType $storageType -Location $location -CreateOption Empty -DiskSizeGB 128

$MDdisk1 = New-AzureRmDisk -DiskName $dataDiskName -Disk $MDdiskConfig -ResourceGroupName $rgName

$VM = Get-AzureRmVM -Name $VMName -ResourceGroupName $rgName

$VM = Add-AzureRmVMDataDisk -VM $VM -Name $dataDiskName -CreateOption Attach -ManagedDiskId $MDdisk1.Id -Lun 1

Update-AzureRmVM -VM $VM -ResourceGroupName $rgName

You can see that is necessary since managed disks are first-class objects that need, at the minimum to be referenced for use. My personal kudos to Azure Engineering team that made the API and cmdlets so clear for MD.  

Additional Things to Remember

Azure Managed Disks is integrated in the new Portal and offers also full development tool coverage including .NET, Java, Ruby and Python SDKs, PowerShell and Azure CLI, ARM templates. What I described in this blog post was directly experienced in my recent works, but there are some other important details that deserve attention, I grouped the main one in the list below:
  • Billing Model
    • With MD, costs are calculated in the same way (although still at different rates) for both Premium Storage and Standard storage.  Like UMD Premium storage – cost is based on the provisioned size of a disk rather than on current used space within a disk.   While one can still request a disk of any size up to 1TB, MD calculates costs based on a fixed set of standard sizes. For example, when one provisions a Standard MD 60GB disk, it would be billed as an S6 (64GB) disk – the smallest standard size of >= 60GB (see Table 1.1 above).
    • Regarding Snapshots on MD, at the moment they are not incremental and you will pay for the full size of the sourced MD.

Azure Managed Disks Overview- Pricing and Billing

https://docs.microsoft.com/en-us/azure/storage/storage-managed-disks-overview#pricing-and-billing

 
  • Outbound TCP port 8443
    • If you are using Managed Disks and if you use VM extensions,  you have to permit outbound only traffic over TCP port 8443, otherwise your VM deployment will fail. This port is required by the VM Agent to report VM extension status in the Azure storage as a blob. By default, outbound Internet access is permitted for Azure VMs, but if your environment requires strict traffic control and security lock-down, this may be an issue. Azure Engineering team is aware of this and working actively to remove this requirement. Based on the link below, estimated date for an update is “…by the end of May 2017.”:

Frequently Asked Questions about Azure IaaS VM Disks and managed and unmanaged premium disks

https://docs.microsoft.com/en-us/azure/storage/storage-faq-for-disks

  • Migration from Un-Managed Disks (UMD) to Managed Disks (MD)
    • Migrating from UMD to MD is possible, but will involve a service downtime for all your VMs in the specific Availability Set.
    • There is a nice PowerShell cmdlet called “ConvertTo-AzureRmVMManagedDisk” that will be very helpful in this task.
    • If your VM has disks on a storage account that is, or at any time has been, encrypted using Azure Storage Service Encryption (SSE), then direct migration is not possible. You have to copy the disk out and decrypt before.
    • During the migration from UMD to MD, is also possible to upgrade from Standard Storage to Premium Storage.
    • Is also possible to migrate disks from AWS to Azure MD, if you first export the EC2 instance to a VHD in an Amazon S3 bucket; procedure is detailed below:

Migrate from Amazon Web Services (AWS) to Azure Managed Disks

https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-aws-to-azure

Essentially, the procedure in the article below will stop all your VMs (with deallocation!) in the specified Availability Set (AS), will “convert” (i.e. = copy!) all attached disks to MD type, then will also convert AS to be “managed”.

Convert a VM from unmanaged disks to managed disks

https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-convert-unmanaged-to-managed-disks

  • Backup
    • Azure Backup Service now supports VM backups using Managed Disks:

Overview of the features in Azure Backup - Using Managed Disk VMs with Azure Backup

https://docs.microsoft.com/en-us/azure/backup/backup-introduction-to-azure-backup#using-managed-disk-vms-with-azure-backup

 
  • Azure modules for PowerShell
    • In order to use MD in PowerShell, be sure to use minimum version 2.6.0 or higher for AzureRM.Compute module:

Install-Module AzureRM.Compute -RequiredVersion 2.6.0 

   Samples and Templates

  • Azure MD obviously also support ARM "Templates", you can find a large collection in GitHub at this link.
  • Silvano Coriani from Azure CAT wrote some samples for Java, you can here:

ServiceBuilder: Demo of a provisioning Engine with a multi-step workflow

https://github.com/scoriani/ServiceBuilder

Thank you!

For the patience and great support to the work I have done on Managed Disks, great kudos to Silvano Coriani from AzureCAT (@scoriani). Hope you enjoyed the post, let me know your feedbacks, you can also follow me on Twitter (@igorpag). Best regards.  ]]> My Personal FAQ on Azure Networking and ARM https://docs.microsoft.com/archive/blogs/igorpag/my-personal-azure-faq-on-azure-networking-and-arm Sun, 24 Jul 2016 13:56:12 GMT https://blogs.msdn.microsoft.com/igorpag/?p=555 Last updated: July 24, 2016. IMPORTANT: The information I reported in this post is time sensitive since may change in the future. I will try to maintain the content updated, but you should double--check with official Azure documentation sources. This is a new article specifically focused on ARM, you can find a previous old one related to ASM at the link below:

My personal Azure FAQ on Azure Networking SLAs, bandwidth, latency, performance, SLB, DNS, DMZ, VNET, IPv6 and much more

https://blogs.msdn.microsoft.com/igorpag/2014/09/28/my-personal-azure-faq-on-azure-networking-slas-bandwidth-latency-performance-slb-dns-dmz-vnet-ipv6-and-much-more

In the last four years I worked extensively on Azure IaaS, during engagements with my partners I have been asked many questions on Azure networking in general, and more specifically on Virtual Network (VNET) and VMs in new Azure Resource Manager (ARM) world. Providing adequate answers is not always easy, sometime the documentation is not clear or missing in details, while other times you can deduct yourselves but requires pretty good knowledge. Since I detected some recurring patterns in these questions, I decided to write my personal FAQ list in this blog post. Just to be clear: even if I’m a MSFT and have inner knowledge of Azure, I’m not going to unveil you any reserved or secret information. Everything you will read here can be found playing directly with Azure or retrieved by using official and public documentation. If you have additional “nice” questions requiring a non-trivial or non-easy-to-retrieve answer, fill free to insert a comment below this post. Additionally, if you have proposals or feedbacks regarding new feature that Azure networking should have, please use the link below to submit your ideas or vote existing ones:

FEEDBACK FORUM: Networking (DNS, Traffic Manager, VPN, VNET)

https://feedback.azure.com/forums/217313-networking-dns-traffic-manager-vpn-vnet/category/77469-virtual-networks-vnet

As usual, you can also follow me on Twitter at @igorpag. Regards.    

Can I create a Storage Account or Azure SQLDB server without a public VIP?

While is possible to create a Virtual Machine not exposed to the Internet, for Azure Storage Accounts and Azure SQLDB logical servers there is a mandatory DNS name created that need to be exposed. During creation of these resources, a FQDN records is created and exposure cannot be avoided since this resource locator is required to access the resource. Both these resources are protected by Microsoft DDoS and Azure SQLDB can be also configured with its own firewall mechanism.

Configure an Azure SQL Database server-level firewall rule using the Azure Portal

https://azure.microsoft.com/en-us/documentation/articles/sql-database-configure-firewall-settings

Do Azure VPN or ExpressRoute provide traffic encryption?

Azure VPN tunnels, for Site-to-Site connectivity (S2S), are by design/default encrypted using IPSEC. Encryption is also provided for Point-to-Site (P2S) using SSTP (Secure Socket Tunneling Protocol). ExpressRoute does not provide network traffic encryption for its circuits.

VPN Gateway FAQ

https://azure.microsoft.com/en-us/documentation/articles/vpn-gateway-vpn-faq

 

Do Azure VPN or ExpressRoute provide QoS?

ExpressRoute provides network traffic QoS only for Skype for Business, while Azure VPN does not provide this kind of service. You can read more details on ExpressRoute at the link below:

ExpressRoute QoS requirements

https://azure.microsoft.com/en-us/documentation/articles/expressroute-qos

 

Do Azure VPN and ExpressRoute support BGP?

YES, now also Azure VPN supports Border Gateway Protocol (BGP) for automatic exchange of routing information when different VNETs and/or on-premises sites are connected together. ExpressRoute fully support BGP since the beginning. Transit routing is supported, but redundant tunnels between a pair of virtual networks are not supported yet. Be aware that is not possible today to filter BGP routes at the VPN level, if you need to build more advanced network topology, you should give a look to the second URL listed below.

Overview of BGP with Azure VPN Gateways

https://azure.microsoft.com/en-us/documentation/articles/vpn-gateway-bgp-overview

 

Hub&Spoke, Daisy-Chain and Full-Mesh VNET topologies in Azure ARM (V2)

https://blogs.msdn.microsoft.com/igorpag/2016/02/11/hubspoke-daisy-chain-and-full-mesh-vnet-topologies-in-azure-arm-v2

1

Can I connect two Azure Virtual Networks without any Gateway?

This feature is coming, stay tuned! It will permit to connect different Azure VNETs without VPN or ExpressRoute, then without bandwidth caps imposed by the Gateway.

Can I assign multiple IPs to my single Network Interface Card (NIC)?

This feature is coming, stay tuned!  

Is it possible to change subnet or virtual network for my existing Virtual Machine?

With some limitations, it is possible to change subnet for an existing Virtual Machine, but changing Virtual Network (VNET) is not directly possible, you will need to export VM definition/disks and recreate in the new VNET. You can read more details in the article below:  

How to change Subnet and Virtual Network for Azure Virtual Machines (ASM & ARM)

https://blogs.msdn.microsoft.com/igorpag/2016/03/13/how-to-change-subnet-and-virtual-network-for-azure-virtual-machines-asm-arm

Picture1

Can I mix ASM ARM Virtual Machines and Virtual Networks?

NO, only Virtual Machines created with Azure Resource Manager (ARM) API can join ARM Virtual Networks, similarly only Azure Service Manager (ASM) Virtual Machines can join ASM Virtual Network. Additionally, migration is possible from ASM VMs/VNETs and Azure Resource Manager resource model, but this will happen at the VNET level, then including all the contained VMs at the same time. You can read more about migration tools and procedures at the link below:

Technical deep dive on platform-supported migration from classic to Azure Resource Manager

https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-migration-classic-resource-manager-deep-dive

 

Does Azure Load Balancer support multiple IPs for the same set of Virtual Machines?

YES, now also using Azure Resource Manager (ARM), it is possible to have multiple IPs assigned to the same load balancer and then traffic forwarded to the same set of Virtual Machines included in one single Availability Set. This feature was previously available using the Azure Service Manager (ASM) model. For a code sample on how to add multiple VIP to Azure SLB, you can read the article below:

Adding additional VIP to Azure Load Balancer

https://jeffgraves.me/2016/04/13/adding-additional-vip-to-azure-load-balancer

 

Does Azure Load Balancer provide Layer-7 load-balancing?

Azure built-in load balancer is Layer-4 (TCP/UDP), but if you need Layer-7 load-balancing, you can use Azure Application Gateway or one of the Network Virtual Appliance (NVA) available in the Marketplace by 3rd-party vendors.

Azure Load Balancer Overview

https://azure.microsoft.com/en-us/documentation/articles/load-balancer-overview

2

Can I customize the Internet facing FQDN used by Azure for my public IP?

Without using vanilla domains with CNAME records, you can only partially change the FQDN associated to your public IP, doesn’t matter if reserved or not, doesn’t matter if VIP or Instance-Level IP (ILPIP, PIP). The suffix will be always in the form of “location.cloudapp.azure.com”, where “location” is the Azure region where you deployed, you can only add a “domainnamelabel” prefix and obtain “domainnamelabel .location.cloudapp.azure.com” DNS domain.

IP addresses in Azure

https://azure.microsoft.com/en-us/documentation/articles/virtual-network-ip-addresses-overview-arm

 

Can I use Receive Side Scaling (RSS) optimization in Azure VM?

YES. Azure HyperV is able to support RSS on the host side, but to leverage this functionality in Guest OS you need to manually enable it following the instructions at the link below:

Virtual Receive-side Scaling in Windows Server 2012 R2

https://technet.microsoft.com/en-us/library/dn383582.aspx

For Windows Server, Guest OS must be Windows Server 2012 R2 version or later, while for support on Linux OS, see the article below:

Linux and FreeBSD Virtual Machines on Hyper-V

https://technet.microsoft.com/en-us/library/dn531030.aspx

Does ICMP work in Azure?

It depends from where you want to use ICMP tools and which targets you want to reach. ICMP works inside an Azure Virtual Network, between different Azure Virtual Networks connected via VPN or ExpressRoute or hybrid connections between on-premise and Azure VNET. It will *not* work if you have to cross the Azure Load Balancer (SLB), for example if you want to ping a Cloud Service from another Cloud Service. If you need ICMP for diagnostic and troubleshooting, there are valid alternatives to PING and TRACERT not based on ICMP but TCP. My favorite ones, among many others available in Internet, are PSPING from Mark Russinovich SysInternals tools and NMAP:

PsPing v2.01

https://technet.microsoft.com/en-us/sysinternals/jj729731.aspx

NMAP

https://nmap.org/download.html

Does Azure provide precise numbers on VM network bandwidth?

I tested VM bandwidth limits on almost all VM sizes and found the results pretty consistent over time, then in theory there is nothing preventing you from generating your measure. What Azure documentation provide is a relative measure of network capacity for each VM SKU/SIZE as you can see in the table below:

Sizes for virtual machines in Azure

https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-sizes

If you look at column “Max Network Bandwidth”, you can find tags as LOW, MODERATE, HIGH, VERY HIGH, EXTREMELY HIGH. If you found in Internet, some old tables and information related to maximum network bandwidth for various VM SKUs, you should not trust them since outdated and Azure infrastructure evolved in the meanwhile. The only exceptions are Azure HPC instances where it is exactly specified the type of network adapter provided, that is 1 NIC 32Gbit/s and 1 NIC 10Gbit/s for A8/A9, and 1 NIC 10Gbit/s for A10/A11. See at the link below for more information:

About the A8, A9, A10, and A11 compute-intensive instances

https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-a8-a9-a10-a11-specs

Does Azure provide SLA on VM-to-VM network latency?

During my tests and projects, Azure network Always provided excellent performances in terms of latency. At least at my knowledge, there is no Cloud vendor able to guarantee a maximum network latency, backed up by a formal SLA, between two different VMs. It’s very easy to test yourself and have a good understanding of average latencies, but you need to be very careful on your test criteria. First of all, you need to test over various hours during the day including peak and non-peak hours, various days over the week including weekends and working days, then finally you should use calculate the average and consider a good percentile sampling near the 95% to eliminate peaks and occasional strange values. Additionally, don’t assume latencies will be the same in all Azure datacenters. If you want to do your own tests, I would recommend you PSPING tool from SYSINTERNAL suite that you can find here: https://technet.microsoft.com/en-us/sysinternals/bb896649.aspx. What is really nice regarding this tool, is that it will not use ICMP then no problem to traverse firewalls and load-balancers, you can decide which TCP port to use and, very important, you can test for bandwidth, not only for latency. You can read full details on using this tool to test Azure latency in one of my previous blog post:

Azure Network Latency & SQL Server Optimization

https://blogs.msdn.com/b/igorpag/archive/2013/12/15/azure-network-latency-test-and-sql-server-optimization.aspx

3 In the past, “Affinity Group” concept has been used to provide co-location of VMs and minimize latency: now this concept is officially deprecated with the new Azure Resource Manager (ARM) API and broad adoption of “Regional” VNET. Azure infrastructure continuous evolutions made network more efficient and performant, then you should not use it anymore.

Azure Datacenter-to-Datacenter traffic will go through public Internet?

Traffic between VNETs in different datacenters, connected via VPN connection will remain inside Microsoft owned network backbones, see slide [34] in the presentation below:

Introduction to Microsoft Azure Networking Technologies and What's New

https://channel9.msdn.com/Events/TechEd/Europe/2014/CDP-B227

Also in the case of VNETs connected using Express Route, traffic will not go through public Internet, private leased line will be used.

Is it possible to know Azure public IP address pools?

YES, all Azure public IP ranges are published at the link below. If you want to white-list, for security reasons, on your on-premise network you can do that selectively by region but not by service.

Microsoft Azure Datacenter IP Ranges

https://www.microsoft.com/en-us/download/details.aspx?id=41653

If you rely on specific Azure resources, for example an Azure SQLDB instance or blob storage account, be aware that the assigned IP you will see, resolving their specific DNS names, may change without notice. IMPORTANT: don’t rely on any geo-location service for Internet IP used by Azure since may report incorrect information, for more details see the blog post below:

Microsoft Azure’s use of non-US IPv4 address space in US regions

https://azure.microsoft.com/blog/2014/06/11/windows-azures-use-of-non-us-ipv4-address-space-in-us-regions

If you created your own Cloud Service, and want to be possible to validate it using reverse-DNS, you can do that enabling PTR record registration (by Azure) for your Cloud Service VIP as described at the link below:

Announcing: Reverse DNS for Azure Cloud Services

https://azure.microsoft.com/blog/2014/07/21/announcing-reverse-dns-for-azure-cloud-services

Be aware that the Virtual IP (VIP) associated to your Cloud Service may change under some conditions, if you need to be absolutely sure it will never happen, you need to enable "Reserved IP" feature as described at the blog post below:

Reserved IP addresses for Cloud Services & Virtual Machines

https://azure.microsoft.com/blog/2014/05/14/reserved-ip-addresses

5

Is there any API to retrieve Azure public IP addresses?

NO, there is no built-in API provided by the platform at the moment, instead you can download the XML file from the link below. Or you can build a simple logic to download the file and parse content as shown in the second URL listed below:

Microsoft Azure Datacenter IP Ranges

https://www.microsoft.com/en-us/download/details.aspx?id=41653

Step-by-Step: Automate Building Outbound Network Security Groups Rules via Azure Resource Manager (ARM) and PowerShell

https://blogs.technet.microsoft.com/keithmayer/2016/01/12/step-by-step-automate-building-outbound-network-security-groups-rules-via-azure-resource-manager-arm-and-powershell

Azure “Affinity Group” is officially deprecated?

YES, with the introduction of Azure Resource Manager (ARM) new API model, “Affinity Group” (AG) is officially deprecated, thus not recommended to use it.

Azure Subscription and Service Limits, Quotas, and Constraints

https://azure.microsoft.com/en-us/documentation/articles/azure-subscription-service-limits

4

Even if AG can be still used in Service Management (SM) API, but not in the legacy and new Azure portals, usage is not recommended since many Azure networking features rely on “Regional Virtual Networks”. Using an AG to pin a Virtual Network in fact, will originate a “Local Network” tied to a single Azure Cluster and will not have benefits of the new Azure scale units for compute resources.

Regional Virtual Networks

https://azure.microsoft.com/blog/2014/05/14/regional-virtual-networks

7

What is “*.internal.cloudapp.net” DNS suffix in Azure VM?

If you build an Azure VM and join to an Azure Virtual Network and try to ping it using the VM host name (legacy Netbios name to be clear), this is what you will see:

8

As you can see in the picture, there are two strange parts in the FQDN: nothing strange in “cloudapp.net”, but what are “internal” and “a3”? You can easily guess that “internal” is the default DNS sub-zone that Azure internal DNS (iDNS) will use to host records for VMs and resolve to internal VM IP (DIP). “a3” instead is a little bit complex: in short, it’s describing the network zone, inside the specific Azure datacenter, where the VM is allocated. There is no official documentation or list of these zones, then don’t ask more, but please note that it will change not only between Azure datacenters, but also inside a single datacenter. You can programmatically retrieve internal DNS setting using the simple PowerShell command below:

(Get-AzureRmNetworkInterface -Name "mynicname" -ResourceGroupName $rgname).DnsSettings.InternalFqdn

Output will be something like the string below:

InternalDomainNameSuffix : 0fcdtenvgl4urm0sv5aj1xkcrc.fx.internal.cloudapp.net

 

PING and TRACERT tools will work inside Azure Virtual Network?

YES, they will work perfectly inside a VNET, and even from on-premise to VNET via VPN connection. They will not work if you try to cross the Azure Load Balancer (SLB/ILB).  

Azure Load Balancer (SLB) uses Round-Robin policy to distribute incoming connections?

This is a very common mis-understanding on how SLB works. Actually, Azure SLB is a Layer-4 software Load Balancer and uses a 5-tuple (source IP, source port, destination IP, destination port, protocol type) to calculate the hash that is used to map traffic to the available servers behind a VIP. The hash function is chosen such that the distribution of connections to servers is fairly random then not perfect round-robin. Additionally, at least today, session affinity is not supported.

Microsoft Azure Load Balancing Services

https://azure.microsoft.com/blog/2014/04/08/microsoft-azure-load-balancing-services

More recently, Azure introduced a new distribution mode called Source IP Affinity (also known as session affinity or client IP affinity). Azure Load Balancer can be configured to use a 2 tuple (Source IP, Destination IP) or 3-tuple (Source IP, Destination IP, Protocol) to map traffic to the available servers. By using Source IP affinity, connections initiated from the same client computer goes to the same DIP endpoint.

Azure Load Balancer new distribution mode

https://azure.microsoft.com/blog/2014/10/30/azure-load-balancer-new-distribution-mode

9

Azure Load Balancer (SLB) supports SSL termination?

Azure SLB (or ILB) does not support SSL termination at the edge, but you can use Azure Application Gateway to achieve this goal:

Application Gateway Overview

https://azure.microsoft.com/en-us/documentation/articles/application-gateway-introduction

Alternatively, you can handle the termination process for HTTPS encryption and decryption within each VM or Web/Worker role instance, use some 3rd-party appliances or OSS technologies as NGINX or HAPROXY.  

Can I use my own Load Balancer and avoid Azure Load Balancer?

YES, usage of Azure SLB is not mandatory, there is a specific offer from Azure called “Basic” that will give you up to 27% lower prices, compared to Standard SKUs:

Basic Tier Virtual Machines

https://azure.microsoft.com/blog/2014/04/28/basic-tier-virtual-machines-2

Autoscaling will not be available with “Basic”, but you will still have 99,95% through “Availability Set”. Please also notes that not all the sizes, in terms of CPUs and RAM, will be available:

10

Once selected the “Basic” tier for your VM, you can then install your own Load Balancer software. Be aware that sizes available in the “Basic” tiers are very limited and all listed in the picture and article above.  

Can I expose full port range of my VM?

YES, there is an Azure specific feature called “Instance-Level Public IP Addresses” that will provide your VM a “Public Instance IP” (PIP) with full port range opened by default, without any need to create any endpoint. Using normal VIPs, you can only create/open up to 150 endpoints/ports, then if you need a large number of ports to be opened this is the feature that will satisfy your requirements.

Instance-Level Public IP Addresses

https://msdn.microsoft.com/en-us/library/azure/dn690118.aspx

11

How many connections a VM can support?

For Windows Server, a VM can support about 500k TCP connections but you need to be careful about other potential limits that may come into the game before this threshold: if you expose your VM to Internet traffic through an Azure Load Balancer endpoint, you may be limited by SLB capacity or DDOS security mechanisms. Actually, there is no public documentation on SLB or Endpoint limits, neither on the number of connection nor network bandwidth, then you should conduct your own performance tests to ensure everything will work correctly.

Can I have multiple VNET Gateways?

Currently you can only have one virtual network Gateway per single VNET per single type, that is one for VPN and one for ExpressRoute. With VPN, if you configure multiple tunnels, they will share the same maximum network bandwidth.

ExpressRoute and VPN can coexist in the same VNET?

YES, it is possible to have a VNET with gateways supporting ExpressRoute and VPN at the same time. The only requirement is to enable ExpressRoute before VPN. One Gateway per VNET restriction persists.

New Networking Capabilities for a Consistent, Connected and Hybrid Cloud

https://azure.microsoft.com/blog/2015/05/05/new-networking-capabilities-for-a-consistent-connected-and-hybrid-cloud

12

Can I connect VNETs across regions, geographies and subscriptions?

YES, Azure provides two ways to connect VNETs: ExpressRoute and VPN. With VPN, there is no boundary on the region, geography or subscription, essentially the tunnel only need Gateway VIPs to see each other. In case you need more bandwidth and lower latency, you can eventually use ExpressRoute technology to connect up to 100 VNETs (depending on bandwidth) and share circuits between them, even with different subscriptions. Before Premium add-on announced in May 2015, ExpressRoute posed limitations on geographical boundaries: now it is possible not only to connect VNETs in the same geography (US, Europe, etc.) but also across different geographies. For example, you can connect a VNET in West US to a VNET in North Europe.

New Networking Capabilities for a Consistent, Connected and Hybrid Cloud

https://azure.microsoft.com/blog/2015/05/05/new-networking-capabilities-for-a-consistent-connected-and-hybrid-cloud

Does Azure provide DDoS network protection?

YES, Azure infrastructure is designed to protect the network from DDOS originating from the Internet and also internally from other tenants VMs, you can read the details in the white-paper below:

Microsoft Azure Network Security Whitepaper version 3 is now available

https://azure.microsoft.com/en-us/blog/microsoft-azure-network-security-whitepaper-version-3-is-now-available

 Please be aware of the following important points:
  • Windows Azure’s DDoS defense system is designed not only to withstand attacks from the outside, but also from within.
  • Windows Azure monitors and detects internally initiated DDoS attacks and removes offending VMs from the network.
  • Windows Azure’s DDoS protection also benefits applications. However, it is still possible for applications to be targeted individually. As a result, customers should actively monitor their Windows Azure applications.
To augment your tenant security, you can deploy 3rd-party Network Virtual Appliances (NVA) in your Virtual Network, but be aware that custom DDoS to protect Azure public IPs (VIPs) is not supported.  

Can I use IPv6 inside an Azure Virtual Network?

IPv6 support has been confirmed to be under development as you can read at the link below:

Support IPv6 throughout the Azure Platform

https://feedback.azure.com/forums/217313-networking-dns-traffic-manager-vpn-vnet/suggestions/4992369-support-ipv6-throughout-the-azure-platform

Additional comments from Azure FAQ on https://azure.microsoft.com/en-us/pricing/faq: Microsoft has played a leading role in helping customers to smoothly transition from IPv4 to IPv6 for the past several years. To date, Microsoft has built IPv6 support into many of its products and solutions like Windows 8 and Windows Server 2012 R2. Microsoft is committed to expanding the worldwide capabilities of the Internet through IPv6 and enabling a variety of valuable and exciting scenarios, including peer-to-peer and mobile applications. The foundational work to enable IPv6 in the Azure environment is well underway. However, we are unable to share a date when IPv6 support will be generally available at this time.  

UDP broadcast and multicast are supported inside Azure Virtual Network?

This type of communication is not allowed inside a VNET or even across Azure SLB.

Support Multicast within Virtual Networks

https://feedback.azure.com/forums/217313-networking-dns-traffic-manager-vpn-vnet/suggestions/3741233-support-multicast-within-virtual-networks

 Is it possible to have multiple NICs on VM?

YES, this feature has been announced at TechEd Europe back in October 2014. Be aware that adding more NICs will not give you more bandwidth, all will share the same limit granted to the VM level. Now it's possible to have up to 4 additional NICs without any additional cost, the exact number varies depending on the VM size:
  • Large (A3) and A6: 2
  • ExtraLarge (A4) and A7: 4
  • A9: 2
  • D3: 2
  • D4: 4
  • D13: 4
  • <<ETC>>
13
  Be aware that some limitations exist, be sure to read the “Multiple NIC FAQs & Constraints” section in the blog post below:

Multiple VM NICs and Network Virtual Appliances in Azure

https://azure.microsoft.com/blog/2014/10/30/multiple-vm-nics-and-network-virtual-appliances-in-azure

Can I persist my VM network card MAC address?

Work is underway to provide this feature, that is preserving MAC addresses across Azure Host failures, VM resizes and re-deployments.

Can I avoid DHCP for my VM inside Virtual Network?

YES, when you join a VM to an Azure Virtual Network, by default it will take an internal IP (DIP) automatically assigned by Azure internal DHCP, valid in the address scope you defined and eventually in the subnet you indicated. This IP will have an infinite lease time, but under certain conditions may change, for example if you stop your VM with de-provisioning (default behavior). If you are going to mix DHCP dynamically assigned IPs and statically assigned IPs, it’s recommended to use separate subnets to avoid potential address clashes. You can read more details in the blog post below:

Static Internal IP Address for Virtual Machines

https://azure.microsoft.com/blog/2014/04/22/static-internal-ip-address-for-virtual-machines

Is it possible to run network penetration tests against my Azure VMs?

YES, you can do that but it’s highly recommended to follow the specific procedure mentioned below, before running the test, otherwise Azure monitoring and defense systems will trigger in and blacklist your connections, IPs and/or VMs. Be sure your request will be approved before starting the test, you can find the form to fill-in at the bottom of the page URL reported below:
14
Microsoft conducts regular penetration testing to improve Azure security controls and processes. We understand that security assessment is also an important part of our customers' application development and deployment. Therefore, we have established a policy for customers to carry out authorized penetration testing on their applications hosted in Azure. Because such testing can be indistinguishable from a real attack, it is critical that customers conduct penetration testing only after notifying Microsoft. Penetration testing must be conducted in accordance with our terms and conditions.

Which is the guaranteed bandwidth for Azure VPN?

There is no minimum guaranteed bandwidth for Azure VPN, the only SLA provided is about high-availability (99,90%). Azure provides an indication for maximum theoretical limit that actually is 200Mbit/sec with “High-Performance” SKU:

About VPN Gateway

https://azure.microsoft.com/en-us/documentation/articles/vpn-gateway-about-vpngateways

15
As you can see in the table above, there is a limit on the maximum number of tunnels you can create, based on the type of Gateway SKU and connectivity type (S2S VPN or Express Route).

VHD I/O will count toward VM network bandwidth cap?

I/O related to IaaS VM VHDs access will count against storage IOPS limits, not network limits. Seems a tricky question, but since persistent VM storage is networked, some customers and partners frequently ask it. Official numbers on scalability targets for Azure Storage are reported at the link below:

Azure Storage Scalability and Performance Targets

https://msdn.microsoft.com/library/azure/dn249410.aspx

If you use some special features like SQL Server 2014 Azure Blob Storage integration or Azure Files (over SMB), it will count toward your VM network bandwidth limits, in addition to the target service scalability and performance limits.

UPDATED: New White-Paper on SQL Server 2014 and Azure Blob storage integration

https://blogs.msdn.com/b/igorpag/archive/2014/06/22/update-new-white-paper-on-sql-server-2014-and-azure-blob-storage-integration.aspx

 

Introducing Microsoft Azure File Service

https://blogs.msdn.com/b/windowsazurestorage/archive/2014/05/12/introducing-microsoft-azure-file-service.aspx

 

Can I use A8/A9 VM Infiniband NIC for my application traffic?

It depends. I have been asked several times if, for example, Infiniband NIC can be used for SQL Server AlwaysOn Availability Group replication traffic, since it provides very high-bandwidth, low-latency connection and RDMA support. In this specific case, the answer is NO, this NIC does not provide general TCP/IP connectivity, as for all the applications that cannot talk over “Network Direct” interface and MPI protocol.

New High Performance Capabilities for Windows Azure

https://blogs.technet.com/b/windowshpc/archive/2014/01/30/new-high-performance-capabilities-for-windows-azure.aspx

Can I create a DMZ in Azure?

YES. With the introduction of “Network Security Groups” (NSG), Network Virtual Appliances (NVA), “Multiple NICs”, “User Defined Routes” and “IP Forwarding” it is now possible to implement a full-featured DMZ with strong segregation of traffic and full control between subnets inside a VNET. Creating a DMZ requires careful planning and architectural work, you can find a guided step-by-step tour, along with examples, at the link below:

Microsoft cloud services and network security

16
There is already a certain number of 3rd-party Network Virtual Appliances (NVAs) you can use to provide security features like firewalls/DMZ, but also IDS/IPS, you can find them in the Azure Market Place through Azure Certified Program to ensure quality and simplify deployment.

Azure ILB can be used to augment networking security?

YES, but it’s necessary to clarify what is the security benefit since there is some confusion on how this feature works. Essentially, it can help security since will let you define one (or more) load-balanced endpoints, as when using Azure SLB, not exposed to the Internet, then more secure. Conversely, it’s not an isolation mechanism since will not segregate resources inside a VNET, all the VMs will have open and full connectivity to all other VMs if not using ILB, even if ACLs will be used. For the same reason, ILB alone is not enough to build a DMZ.
17
If you want to build a real DMZ configuration, it is recommended to look at the previous “Can I create a DMZ in Azure?” section in this blog post.
]]> Azure Network Security Groups (NSG) - Best Practices and Lessons Learned https://docs.microsoft.com/archive/blogs/igorpag/azure-network-security-groups-nsg-best-practices-and-lessons-learned Sat, 14 May 2016 16:20:25 GMT https://blogs.msdn.microsoft.com/igorpag/?p=395 What is a Network Security Group (NSG)?

https://azure.microsoft.com/en-us/documentation/articles/virtual-networks-nsg

1 In one of my recent engagement with my partners, I had the opportunity to test NSG power, facing some limitations and gaining some good knowledge and experiences I would like to share with you. For some of them, I will include links to existing documentations since published before this blog post.

Think and Plan before Deploy

If you played even only minimally with NSG, you immediately realized that you need to think very carefully about your subnets and virtual network architecture: even if you can also assign NSG to the VM network interface (NIC) level, probably you will want to use subnets as your level of granularity (see below for more details). Since it is not easy and immediate to change subnet structure, if you already deployed VMs in there, my first suggestion is to design your VNET architecture before and (also) according to your NSG needs. Once you have designed your network topology, you need to think about the architecture of the boundaries you want to enforce and probably how your DMZ will look like. In this case, you need to answer at least the questions below, I fetched from an excellent article (see URL below), written by Jon Ormond, that I strongly encourage you to read:
  • Which characteristics and requirements perimeter network should have?
  • How many boundaries are needed?
  • Where are the boundaries located?
  • How are the boundaries implemented?

 Microsoft cloud services and network security

https://azure.microsoft.com/en-us/documentation/articles/best-practices-network-security

Good! Now you have VNET and subnets properly designed and implemented, and you are ready to deploy your NSG rules.

Notable differences between ASM and ARM

NSG is an Azure feature available in both ASM (Azure Service Manager or “classic”) and ARM (Azure Resource Manager) API. It is highly recommended to use ARM for new deployments, and then use ASM only when necessary to support existing environments. If you deployed an ARM VNET, you will need to use ARM API for NSG, similarly you will have to use ASM API in a VNET created in this way. If you played already with NSG in ASM, then jumped to ARM, you will be a bit “disoriented”, especially if you used PowerShell, since there are some changes that you need to be aware of.

How to create NSGs (classic) in PowerShell

https://azure.microsoft.com/en-us/documentation/articles/virtual-networks-create-nsg-classic-ps

 How to create NSGs in Resource Manager by using PowerShell

https://azure.microsoft.com/en-us/documentation/articles/virtual-networks-create-nsg-arm-ps

Essentially, there is no perfect symmetry between ASM and ARM, at least in PowerShell. NSG can be manipulated in the Azure Portal, but if you want to operate at scale and you want to “code” infrastructure, you will need to use PowerShell or SDKs/REST APIs. Also consider that if you want to do some advanced stuff, probably you will not be able to do that in Azure Portal GUI. This is what I found in my activities, be aware of the following: -          NSG granularity: While in ASM you had the possibility to link NSG to VM, NIC and subnet levels, in ARM it is slightly different: you can associate only to NIC and subnet levels. This has historical reasons since initially multiple NICs were not allowed in ASM, then VM concepts was used when NSG was first implemented. You can see details of Set-AzureNetworkSecurityGroupAssociation for ASM. -          Remove associations: While in PowerShell ASM you had cmdlets Remove-AzureNetworkSecurityGroupFromSubnet and Remove-AzureNetworkSecurityGroupAssociation to remove NSG from subnets or VMs, this is no longer the case in ARM. See section “Remove NSG association from Subnets or NICs” later in this blog post. -          List NSGs: In ASM you have a nice cmdlet named Get-AzureNetworkSecurityGroupForSubnet to retrieve NSG associated to a subnet, in ARM there is not, you essentially need to retrieve the subnet object and then navigate through the properties and see which NSG, if any, is linked. -          NSG logs: Logs and diagnostics for NSG are only available in ARM, there is no ASM coverage, see section “Enable NSG diagnostics and logs programmatically” later in this blog for details.   In general, I noticed that for many Azure objects and resources, in PowerShell recently we switched from a model where a distinct pair of cmdlets existed for everything, to a new model where many operations are now managed by “Get/Set” approach. Now you can retrieve object references and then read and/or manipulate directly property values.

Which Scope for NSG?

Which scope you should use for linking your NSG: subnet level or NIC level or both? Binding NSG to the individual VMs (by NIC) is powerful, but you may quickly lose control of the complexity of your deployment since would be hard to track and maintain. Generally, this is used for specific VMs with Network Virtual Appliances (NVAs) roles, otherwise it is recommended to link NSG to the subnet level and re-use across your VNETs and subnets. 2   Be very careful when you want to apply NSG to both VM (NIC) and subnet level at the same time: NSGs are evaluated independently, and an “allow” rule must exist at both levels otherwise traffic will not be admitted. Let me give you an example on incoming traffic on port 80: you need to have NSG at subnet level with ALLOW rule for port 80, and you also need another NSG with ALLOW rule on port 80 at the NIC level. For incoming traffic, NSG set at the subnet level is evaluated first, then the NSG set at the NIC level. For outgoing traffic is obviously the converse. The picture below should even clarify this concept more: you can see how rules are evaluated for network packets, once again remember that you need to evaluate this diagram two times: once for subnet level NSG rules, and once for NIC level NSG rules.   3

Be very careful with Defaults and Tools

When you create a Network Security Group (NSG), even completely empty without any rule, there are some defaults that come with it, you can visualize in ARM accessing the specific property of the NSG object in PowerShell:

$nsg0.DefaultSecurityRulesText

4 Notice in the output that there are three inbound and three outbound rules. Rules are assigned a priority, and while the default rules cannot be deleted, they can be overridden by rules with higher priority. In ASM you can achieve the same using the PowerShell command below with the “-Detailed” switch:

Get-AzureNetworkSecurityGroup -Name <<NSG Name>> –Detailed

Default rules will ensure that no inbound traffic will be permitted, except for “polling” from Azure load balancer, connectivity inside the VNET/subnet is not blocked, outbound traffic will be permitted including also Internet address space. Then, what is not included here? Your application/service endpoints! Suppose that you installed an IIS VM, you opened port 80 on the Guest OS firewall, you created a load-balanced or NAT rule for port 80. Everything works fine, but now you decide to further secure your environment adding a new Network Security Group: you need to explicitly add a rule for port 80 and protocol HTTP, otherwise when you will apply to the VM/NIC or subnet, your existing application/service will break. Tools are also very important and final results may differ greatly depending on your choice. If you create your VM using the Azure Portal, at a certain point you will be offered to create a new brand NSG and related rules, if you included a load balancer or Instance-Level IP (ILPIP or PIP). In this case the tool you have chosen has taken care of ensure that, since your VM is exposed to the Internet, access is restricted and secured. A new rule for RDP port, for Windows VMs, or SSH for Linux VM, will be present here. On the other side, if you use PowerShell instead, you will not be prompted or required to create NSG: it us under your responsibility to create proper NSG and rules. While for VIP no port will be opened and then no access will be permitted, even without NSG, if you decide to use ILPIP, you VM will be totally exposed to the Internet for all ports and NSG is strongly required, in addition to a very good Guest OS firewall.

Be very careful on “Deny All” outbound Internet traffic

If your intention is to harden network security of your environment, be very careful with adding NSG rules that will block everything, instead proceed incrementally in a test environment until you will be satisfied with the results. Default rules contains already “Deny All” rules for both inbound and outbound traffic, but are the lowest in priority (65500) and for outbound there is another rule that will allow connection to the Internet. This specific rule has been added to defaults to do not break previous Azure VM behaviors, I have seen many customers and partners that restricted this adding a new rule, with higher priority (lower number), to deny Internet connection partially or totally. This is legitimate, but deny all the Internet traffic maybe dangerous and cause your VM to fail if, for example, you are using VM Extensions as explained in the article below:

VM stuck in “Updating” when NSG rule restricts outbound internet connectivity

https://blogs.msdn.microsoft.com/mast/2016/04/27/vm-stuck-in-updating-when-nsg-rule-restricts-outbound-internet-connectivity

The case above is only one, but there may be other situations where the applications and services installed inside your environment may need to access, for example, other Azure services like Azure SQL DB or Azure Storage resources. Unfortunately, today there is no tag in NSG to identify Azure datacenter IP ranges, they can vary over time and by regions, then how to selectively block Internet outbound traffic without compromising Azure access? 5 My colleague Keith Mayer built a nice solution as described in the article below. Since Azure datacenter ranges are published here, you can use his work to automate NSG creation, using PowerShell, based on this piece of information that Microsoft periodically update.

Step-by-Step: Automate Building Outbound Network Security Groups Rules via Azure Resource Manager (ARM) and PowerShell

https://blogs.technet.microsoft.com/keithmayer/2016/01/12/step-by-step-automate-building-outbound-network-security-groups-rules-via-azure-resource-manager-arm-and-PowerShell

Another useful example is Azure Diagnostic: if you need to enable this feature for your VMs, you cannot deny all outbound traffic, otherwise the agent running inside the Virtual Machine will not be able to connect. The list can continue with SQL Server VM agents for automated patching and backup to Azure blob storage: if you enable these extensions, you need to permit outbound Internet access from these VMs.

Remove NSG association from Subnets or NICs

One of the problem I faced when initially playing with NSG in ARM was how to “reset” my NSG configuration, that is removing association at the NIC and subnet level. In ASM you have a nice cmdlet called Remove-AzureNetworkSecurityGroupAssociation, but there is no equivalent in ARM. After digging into the details of objects and making some experiment, I created a piece of PowerShell code to achieve that:

$vnet = Get-AzureRmVirtualNetwork -Name "Vnet1" -ResourceGroupName $rgname

$subnet = Get-AzureRmVirtualNetworkSubnetConfig -Name "Subnet1" -VirtualNetwork $vnet

$subnet.NetworkSecurityGroup = $null

Set-AzureRmVirtualNetwork -VirtualNetwork $vnet

 Please remember to call Set-AzureRmVirtualNetwork after setting to NULL the NSG property value, otherwise Azure will not commit your modification in the control plane. Approach to do the same at the NIC level is similar, you need to change the NSG property value at the NIC object level:

$nic = Get-AzureRmNetworkInterface -Name "nic2" -ResourceGroupName $rgname

$nic.NetworkSecurityGroupText

$nic.NetworkSecurityGroup = $null

Set-AzureRmNetworkInterface -NetworkInterface $nic

Similarly to the previous case, don’t forget to tell Azure to commit the NIC object configuration change you have just done, otherwise will have no effect. You can use the same technique to change the assigned NSG. Regarding the timing for the operation to be completed, in all my tests I saw the task completed in 30 seconds approximately, but be aware that there is no SLA/SLO on this kind of operations.

Enable NSG diagnostics and logs programmatically

Diagnostic and logs are an important part, especially if you need to troubleshoot unexpected behaviors that may be related to NSG mis-configurations. Remember that in ASM you don’t have this feature. In the article below you can find more information on how to enable this feature and which level of details you can have:

Log analytics for network security groups (NSGs)

https://azure.microsoft.com/en-us/documentation/articles/virtual-network-nsg-manage-log

You can use "Event logs" log to view what NSG rules are applied to VMs and instance roles based on MAC address, and "Counter logs" to view how many times each NSG rule was applied to deny or allow traffic. What is missing here is how to enable programmatically using PowerShell, you can use the sample below to achieve that:

$nsg = New-AzureRmNetworkSecurityGroup -Name testNSGforLogs -ResourceGroupName testwrg -Location westus

$sa = New-AzureRmStorageAccount -ResourceGroupName testwrg -Name nsglogs -Type Standard_LRS -Location westus

Set-AzureRmDiagnosticSetting -ResourceId $nsg.Id -StorageAccountId $sa.Id -Enabled $true

6 This will enable "Event logs" and "Counter logs" that are not enabled by default, remember this if you need to troubleshoot a potential problem with NSG. Conversely, Azure ARM Audit logs are always enabled, you can programmatically retrieve it using Get-AzureRmLog PowerShell cmdlet.

Audit operations with Resource Manager

https://azure.microsoft.com/en-us/documentation/articles/resource-group-audit

As a great add-on to NSG, you can leverage the great PowerBI visualization and analytics capabilities to analyze NSG log, everything you need is reported at the link below:

Azure Audit Logs content pack for Power BI

https://powerbi.microsoft.com/en-us/documentation/powerbi-content-pack-azure-audit-logs

An example of the log entries you can retrieve is shown below: 8

VPN and Express Route

If your Azure Virtual Network (VNET) has VPN and/or ExpressRoute gateways, you need to be very careful: applying NSG to the pre-defined “GatewaySubnet” is not supported neither recommended since you may break connectivity. For your information, few days ago, support for User Defined Routing (UDR) applied to the “GatewaySubnet” has been released and be now used.

NSG and Azure Cloud Services (PaaS)

YES, you heard it correctly, it is possible to leverage Network Security Groups (NSG) also for Azure Cloud Services (web/worker roles), if joined to an Azure (regional) Virtual Network (VNET). This possibility is not widely known because originally another feature was used for Azure PaaS v1, that is Network (or Endpoint) ACLs.

Windows Azure PaaS ACLs Are Here!

https://blogs.msdn.microsoft.com/walterm/2014/04/22/windows-azure-paas-acls-are-here

  You can consider this feature as NSG v1, then superseded by true Azure Network Security Group (NSG) as you know today in ARM. Please remember that there is no Cloud Service (web/worker roles) equivalent in ARM, this is a pure ASM concept only. Long-story short: if you still have Cloud Services, you can leverage NSG changing the “NetworkConfiguration” section of CSCFG file as in the example below:

<NetworkConfiguration>

    <VirtualNetworkSite name="VNet-BL"/>

    <AddressAssignments>

      <InstanceAddress roleName="WebRole">

        <Subnets>

          <Subnet name="TestSubnet-2"/>

        </Subnets>

      </InstanceAddress>

    </AddressAssignments>

    <NetworkSecurityGroupRefs>

      <NetworkSecurityGroupRef roleName="WebRole">

        <NetworkSecurityGroup name="BlockOutbound"/>

      </NetworkSecurityGroupRef>

    </NetworkSecurityGroupRefs>

  </NetworkConfiguration>

  In the “NetworkSecurityGroupRefs” section, you can reference already existing NSGs, you cannot define new ones here: you have to create outside and using ASM APIs, ARM will not work here. Finally, please keep in mind that NSG is not compatible with Network/Endpoint ACLs.

Nice-to-Have features and Roadmap

At least in my case, Azure NSG provided everything needed, but there are some areas that require additional work and workarounds, because the platform doesn’t provide (yet) the ideal solution. -          Built-in Tags for Azure services: As pointed out earlier in this post, there is no pre-defined tag to identify Azure services and then distinguish between Internet and Microsoft datacenter.

 Add a source tag for Azure Datacenter IPs to NSG Rules

https://feedback.azure.com/forums/217313-networking/suggestions/11716131-add-a-source-tag-for-azure-datacenter-ips-to-nsg-r

-          Effective NSG: Since you can have NSG at subnet and VM level, and many rules inside each of them, sometimes calculating the final net effect of a security configuration maybe hard: having a PowerShell cmdlet calculating the net result would be really a great enhancement.

Packet tracert functionality for Network Security Group (NSG)

https://feedback.azure.com/forums/217313-networking/suggestions/9639258-packet-tracert-functionality-for-network-security

 -          Network Security Group logging capabilities to show dropped packets:  This feature has been confirmed in NSG roadmap and will be available soon.

Network Security Group logging capabilities to show dropped packets

https://feedback.azure.com/forums/217313-networking/suggestions/6940205-network-security-group-logging-capabilities-to-sho

 -          ICMP protocol: Today, you can only specify TCP and UDP as protocols, support for ICMP is not available yet. -          NSG Nesting:  You cannot have nested NSGs. -          VM Groups: You cannot define groups of VMs as target for NSG, you have to use subnets for this purpose. I cannot commit on behalf of the Azure Networking team here, but there are many feedbacks from customers, partners and the entire community out there, and I’m pretty sure will contribute to the next NSG feature improvement. You want to provide additional feedbacks at the link below: How can we improve Azure Networking? Enter your idea

https://feedback.azure.com/forums/217313-networking

 Stay tuned and follow me also on Twitter @igorpag. Regards.]]> How to change Subnet and Virtual Network for Azure Virtual Machines (ASM & ARM) https://docs.microsoft.com/archive/blogs/igorpag/how-to-change-subnet-and-virtual-network-for-azure-virtual-machines-asm-arm Sun, 13 Mar 2016 11:00:36 GMT https://blogs.msdn.microsoft.com/igorpag/?p=353 Picture1

Change Subnet in ASM

Changing subnet in ASM is pretty simple, there is a nice PowerShell cmdlet that makes the magic:

Set-AzureSubnet https://msdn.microsoft.com/en-us/library/mt589103.aspx

You can use this same cmdlet at VM creation time to configure the subnet, but you can also use to change subnet for an existing VM, PowerShell code is very simple:

Get-AzureVM –Name $MyVMname –ServiceName $MyCloudServiceName `  | Set-AzureSubnet -SubnetNames $NewTargetSubnet `  | Update-AzureVM

IMPORTANT: There is a strict requirement here: you can only change subnet if the new one is in the same VNET as the old one.

Change Subnet in ARM

In ARM, this cmdlet does not exist anymore, but changing subnet is very simple once you know where to operate the change. What I found particularly interesting is that this change needs to happen at the NIC object, not to the VM level, but if you give a look to the picture above, you will agree that is expected.  First start in the procedure is to obtain references to your VM, subnet, VNET and NIC objects as reported in the code snippet below:

# Obtain VM reference and check Availability Set #

$VMName = "igor-vm1"

$VirtualMachine = Get-AzureRmVM -ResourceGroupName $RGname -Name $VMName

# VM created in Availability Set

$VirtualMachine.AvailabilitySetReference

# Obtain NIC, subnet and VNET references: #

$NIC = Get-AzureRmNetworkInterface -Name $NICname -ResourceGroupName $RGname

$NIC.IpConfigurations[0].PrivateIpAddress # '10.1.1.4'

$VNET = Get-AzureRmVirtualNetwork -Name $VNETname  -ResourceGroupName $RGname

$Subnet2 = Get-AzureRmVirtualNetworkSubnetConfig -VirtualNetwork $VNET -Name $Subnet2name

Now, the subnet setting you need to change resides in the “IPConfigurations” property array, you can access it with index [0] and change directly the “Subnet.Id” value to the ID of the new target subnet:

$NIC.IpConfigurations[0].Subnet.Id = $Subnet2.Id

Set-AzureRmNetworkInterface -NetworkInterface $NIC

Once you have done this operation, you need to commit the change with the Azure ARM Network Provider using the cmdlet “Set-AzureRmNetworkInterface”. Please note that the execution of this cmdlet will take about two minutes, at least in my scenario with an A3 VM type. Why not an immediate operation? Because once you will commit the change, Azure will automatically restart the VM, then you can execute this procedure while the VM is running, but at a certain point you will have a service downtime. Picture2   If you want to check the change completed, you need to re-acquire the NIC object reference and access the private IP address or subnet properties as in the example below:

$NIC = Get-AzureRmNetworkInterface -Name $NICname -ResourceGroupName $RGname

$NIC.IpConfigurations[0].PrivateIpAddress # 10.1.1.4 -> 10.1.2.4

There is a last aspect I want to mention here: what happens if you have more than one VM in an Availability Set and you want to move one of more VMs in a different subnet? No problem, I tested this scenario and you can have one (or more) VMs in one subnet S1 and one (or more) VMs in a different subnet S2, provided that S1 and S2 are in the same VNET. Picture3 Picture4

Change Virtual Network in ASM and ARM

Unfortunately, today is not possible/supported to change directly a VNET for an existing VM. the only way to operate this change is export the VM definition, drop the VM preserving disks, then re-create the VM with the same settings, except for the new VNET assignment, and attaching previous disks. As a proof in ARM, if you try to move a VM to a subnet in a different VNET, you will obtain an error like this:

Update-AzureRmVM : Subnet Subnet4 referenced by resource  /subscriptions/8e95e0bb-d7cc-4454-9443-75ca862d34c1/resourceGroups/igorrg7/providers/Microsoft.Network/networkInterfaces/nic4-igor-vm1/ipConfigurations/ipconfig1 is not in the same Virtual Network as the subnets of other VMs in the availability set.  StatusCode: 400  ReasonPhrase: Bad Request  OperationID : ''  At line:1 char:1  + Update-AzureRmVM -VM $VirtualMachine -ResourceGroupName $rgname  + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~      + CategoryInfo          : CloseError: (:) [Update-AzureRmVM], ComputeCloudException      + FullyQualifiedErrorId : Microsoft.Azure.Commands.Compute.UpdateAzureVMCommand

Another very important property you cannot change once the VM is created, is the “Availability Set”, then you should think about it carefully. Finally, please note that even if with a single VM into an “Availability Set”, trying to move to a different VNET will give you exactly the same error as above, even if there only one VM. That’s all I wanted to share with you on this topic, if you want you can also follow me on Twitter (@igorpag). Best regards.]]> Hub&Spoke, Daisy-Chain and Full-Mesh VNET topologies in Azure ARM (V2) https://docs.microsoft.com/archive/blogs/igorpag/hubspoke-daisy-chain-and-full-mesh-vnet-topologies-in-azure-arm-v2 Thu, 11 Feb 2016 06:37:00 GMT https://blogs.msdn.microsoft.com/igorpag/2016/02/11/hubspoke-daisy-chain-and-full-mesh-vnet-topologies-in-azure-arm-v2/ IMPORTANT: This is an updated version, using Azure PowerShell ARM module later than 1.0.0 release, of the original blog post below. If you need PowerShell code samples earlier than 1.0.0, please use the article below.

Hub&Spoke, Daisy-Chain and Full-Mesh VNET topologies in Azure ARM using VPN (V1)

https://blogs.msdn.com/b/igorpag/archive/2015/10/01/hub-amp-spoke-daisy-chain-and-full-mesh-vnet-topologies-in-azure-arm.aspx

  Introduction One of the key benefit of Azure VPN is the flexibility in arranging almost any kind of connected Virtual Network (VNET) topology in Azure: you can essentially connect together VNETs in any region, of any type (ARM vs. ASM), Azure only or with on-premise hybrid environment, in different subscriptions, etc. This was also possible with the legacy Azure management API called Azure Service Management (ASM), but the drawback was a not-easy not-nice manageability of the topology configuration: since no complete scripting support was available, you had to export an XML configuration file, add your connected VNETs address spaces editing it, then import back into Azure. Additionally, no PowerShell cmdlet existed to script creation of subnets nor getting programmatically the assigned Virtual IP (VIP) assigned to the VPN gateway. If you have never done this before, I suggest you to give a quick look to the article below to have an idea, you will then better appreciate the amazing support in ARM and how easy is to fully automate even complex layouts.

Multihop Routing in Azure

https://blogs.msdn.com/b/rslaten/archive/2014/12/06/multihop-routing-in-azure.aspx

Then, what is effectively changed in Azure Resource Manager? In ARM, Azure Networking team took a totally different approach and promoted many network concepts to first-level objects with full and independent life cycle support. Now you have NET objects, subnet objects, gateway objects you can manipulate using REST PI, SDKs or PowerShell as explained below:
  • You can create subnet objects with New-AzureRmVirtualNetworkSubnetConfig;
  • You can create virtual network objects with New-AzureRmVirtualNetwork;
  • You can create VIP objects with New-AzureRmPublicIpAddress;
  • You can create You can create VPN gateway objects with New-AzureRmVirtualNetworkGateway;
Before going into details on the topologies I want to show you, let me first recap shortly the main limits of Azure VPN that you need to consider when using this connectivity feature:

About VPN gateways

https://azure.microsoft.com/en-us/documentation/articles/vpn-gateway-about-vpngateways

As you can see in the table above, at maximum you can connect up to 30 VNETs using the same VPN gateway using “High-Performance” SKU, be careful also with the bandwidth limit of 200 Mbit/sec. Also remember that you can have only one VPN gateway per VNET. A nice feature recently introduced is also the possibility to have coexistence of VPN gateway and ExpressRoute for the same VNET. It is also worth mentioning that Azure VPN provide high-availability SLA of 99,90%. In the remaining part of this blog post I will show you how to create different connected VNET topologies, the base article you can start with to acquire necessary concepts for PowerShell is reported below:

Configure a VNet-to-VNet connection using Azure Resource Manager and PowerShell

https://azure.microsoft.com/en-us/documentation/articles/vpn-gateway-vnet-vnet-rm-ps

I will only use three VNETs to do not over-complicate the diagrams and PowerShell scripts, but the concepts are essentially the same for any number of connected networks. Attached to this blog post you will find a sample PowerShell script (provided as-is with no warrant nor support!) I used to test all the scenarios with complete PowerShell syntax.   Configuration I prepared three VNETs, in two different regions (West Europe and North Europe), with simple three subnets composition, just to make the examples more comprehensive.

STEP(1): Creating subnets

 

NOTE: Please note that the “GatewaySubnet” subnet definition is mandatory.

STEP(2): Create VNET

STEP(3): Create the VNET Gateway

I repeated these steps for VNET(2) and VNET(3) using different IP address spaces. Hub&Spoke In this type of layout, there is a central VNET (Hub) that must be connected to each other VNET (Spokes), with no direct connection between tenants, all network traffic flows back and forth from/to VNET1 in the graphical example below. Bi-directional connectivity is not mandatory gut generally required.

All the logic to implement network traffic flow is in the definition of “Local Networks” objects, in PowerShell this is accomplished using “New-AzureRmLocalNetworkGateway” cmdlet. Here is what must be used for Hub&Spoke topology:

As you can see, the logic is pretty simple: for every VNET, I defined a “Local Network” logical object that I will later use in the connection definitions. The key information must be supplied with the “-AddressPrefix” parameter: you need to tell Azure what is on the other side of the connection and how to arrive there (“-GatewayIpAddress”). Once defined the connected address spaces with their respective gateway IPs, you need to create the connections:

Two important details to highlight:
  • A connection is unidirectional: if you want it bidirectional between VNET(1) and VNET(2), for example, you need to separately define a connection between VNET(1) and VNET(2), then another one between VNET(2) and VNET(1).
  • A connection is not transitive: if you define a connection between VNET(1) and VNET(2), and a second one between VNET(2) and VNET(3), it does not mean that traffic will flow between VNET(1) and VNET(3). If you want this, you need to look at the “Daisy-Chain” scenario later in this blog post.
Daisy-Chain Differently from the previous scenario, here you want to define a “transitive” network traffic flow: you want your packets to travel (back and forth?) from VNET(1) to VNET(3) through VNET(2).

What is different from the previous approach? Look at the local network definitions you have to use as reported below. In short, from VNET(1) perspective you need to tell Azure that on the other side of the connection, that you will create later, there are two address spaces, that is VNET(2) and VNET(3). If you want to have bi-directionality, you have to create a second object representing the VNET(3) perspective.

And here are the connections that you will need to define:

It is worth noting that the first two connections represent the “transitive” paths between VNET(1) and VNET(3) and vice-versa, but this is not sufficient: you still need to define explicitly the “direct” point-to-point connections between VNET(2) and VNET(1) and VNET(2) and VNET(3) otherwise transitive routing will not happen. Finally, keep in mind that failure on intermediate VPNs will cause a break in the chain and will make some VNETs unreachable.   Full-Mesh This last scenario is similar to the first one, that is “Hub&Spoke”: what is different is that you need also to add connections between each tenant VNETs, including all possible combinations. Based on my example, here you simply need to add connections between VNET(2) and VNET(3). Remember that these are two separate connections since they are mono-directional.

Here is the list of the connections you need, as you can notice the number will increase very quickly since you will need to explicitly define [N*(N-1)] objects.

  Final Notes
  • VNET and VPN flexibility will allow you to change dynamically the layout of your connections, then it’s pretty easy to eventually add more VNETs and/or more connections. What is important to note is that (today) VPN does not support route advertising as ExpressRoute does with BGP: this is the reason why you need to define explicitly local network objects to tell Azure how to route traffic from one side of a connection to the other side and beyond.
  • In this blog post, I used “imperative” ARM model in PowerShell, but it is absolutely possible to “templatize” your connectivity schema using very powerful ARM template feature. My suggestion is to start coding your network layout using “imperative” mode, then when you will be satisfied with the results, switch to “declarative” mode and build your template. See "Template deployment" section at the link below:

Azure Resource Manager overview

https://azure.microsoft.com/en-us/documentation/articles/resource-group-overview

  • You can list all the existing connections in your resource group using the PowerShell cmdlet below:

Get-AzureRmVirtualNetworkGatewayConnection -ResourceGroupName $rgname | select name, location

 

  • If you want to check connectivity status for all your connections and gateways, there is a nice PowerShell cmdlet that will report specific connection status along with some nice details on traffic transferred in/out from/to the connection (please note the “-Debug” switch):
    Get-AzureRmVirtualNetworkGatewayConnection -ResourceGroupName $rgname -Name conn1to2and3 -Debug

  • Be aware that, at least today, you will not be able to see VPN Gateway with multiple connections in the Azure Portal.
That’s all folks, hope you enjoyed this tour of Azure Virtual Network VPN connectivity, let me know your feedbacks. You can also follow me on Twitter (@igorpag). Regards.     SampleScript(2).ps1.txt]]>