Azure Stack Id Manager

Notes

AAD

ADFS

Pre-requisites

• Deploy Azure Stack Development Kit (ASDK) and use ADFS type. https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-run-powershell-script#deploy-the-development-kit
• Prepare a Separate AD forest (BLUE.CLOUD in my setup).
• Deploy a WAP environment. In my setup, I put all the WAP roles on single VM (BLUE-WAP.BLUE.CLOUD). I hosted VMM and SPF are on another VM (BLUE-VMM.BLUE.CLOUD)

Prepare Certificates for *.BLUE.CLOUD

Install-WindowsFeature -Name ADCS-Cert-Authority -IncludeAllSubFeature -IncludeManagementTools

[Version] 
Signature="$Windows NT$"

[NewRequest] 
Subject = "CN=*.blue.cloud"   ; For a wildcard use "CN=*.CONTOSO.COM" for example 
; For an empty subject use the following line instead or remove the Subject line entierely 
; Subject = 
Exportable = TRUE                  ; Private key is not exportable 
KeyLength = 2048                    ; Common key sizes: 512, 1024, 2048, 4096, 8192, 16384 
KeySpec = 1                         ; AT_KEYEXCHANGE 
KeyUsage = 0xA0                     ; Digital Signature, Key Encipherment 
MachineKeySet = True                ; The key belongs to the cph01 computer account 
ProviderName = "Microsoft RSA SChannel Cryptographic Provider" 
ProviderType = 12 
SMIME = FALSE 
RequestType = CMC

; At least certreq.exe shipping with Windows Vista/Server 2008 is required to interpret the [Strings] and [Extensions] sections below

[Strings] 
szOID_SUBJECT_ALT_NAME2 = "2.5.29.17" 
szOID_ENHANCED_KEY_USAGE = "2.5.29.37" 
szOID_PKIX_KP_SERVER_AUTH = "1.3.6.1.5.5.7.3.1" 
szOID_PKIX_KP_CLIENT_AUTH = "1.3.6.1.5.5.7.3.2"

[Extensions] 
%szOID_SUBJECT_ALT_NAME2% = "{text}"
_continue_ = "dns=*.blue.cloud&"
_continue_ = "dns=enterpriseregistration.blue.cloud&"
_continue_ = "dns=fs.blue.cloud&"
_continue_ = "dns=certauth.fs.blue.cloud&"
%szOID_ENHANCED_KEY_USAGE% = "{text}%szOID_PKIX_KP_SERVER_AUTH%,%szOID_PKIX_KP_CLIENT_AUTH%"

[RequestAttributes]

cmd.exe /c "certreq -new Bluecloud-Cert.inf Bluecloud-Cert.req"
cmd.exe /c "certreq -submit Bluecloud-Cert.req"

cmd.exe /c "certreq -accept Bluecloud-Cert.cer"

$cert = get-Childitem cert:\localmachine\my | where-object {$_.Subject -eq "CN=*.blue.cloud"}
$PfxPass = ConvertTo-SecureString "123" -AsPlainText -Force
Export-pfxCertificate -Cert $cert -FilePath c:\Bluecloud-Cert.pfx -Password $PfxPass
$CAcert = Get-Childitem cert:\localmachine\root | where-object {$_.Subject -eq "CN=blue-BLUE-WAP-CA"}
Export-Certificate -Cert $CAcert -FilePath c:\Bluecloud-CACert.cer

Import-Module WebAdministration
$sslBindings = Get-childItem IIS:\SslBindings\
$sslBindings | foreach-Object{$path = "IIS:\SslBindings\" + $_.IPAddress.IPAddressToString + "!" + $_.port; Remove-Item $path; $cert | new-item $path}

Install ADFS in Domain "BLUE.CLOUD"

Install-WindowsFeature -Name ADFS-Federation -IncludeAllSubFeature -IncludeManagementTools

# Set these values:
$domainName = 'blue.cloud'
$adfsPrefix = 'fs'
$username = 'administrator' 
$password = 'User@123'
$dnsName = ($adfsPrefix + "." + $domainName)
$PfxPass = ConvertTo-SecureString "123" -AsPlainText -Force
$securePassword = ConvertTo-SecureString -String $password -Force -AsPlainText
$adfsServiceCredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList ($domainname + '\' + $username), $securePassword

Import-Certificate -FilePath C:\Bluecloud-CACert.cer -CertStoreLocation cert:\localmachine\root 
$cert = Import-pfxCertificate -FilePath C:\Bluecloud-Cert.pfx -CertStoreLocation cert:\localmachine\my -Exportable -Password $PfxPass

# Configure AD FS
Install-AdfsFarm `
    -CertificateThumbprint $cert.Thumbprint `
    -FederationServiceName $dnsName `
    -ServiceAccountCredential $adfsServiceCredential `
    -OverwriteConfiguration

Set-ADFSProperties -IgnoreTokenBinding $True
Set-ADFSWebConfig –HRDCookieEnabled $false
cmd.exe /c "setspn -U -S http/fs.blue.cloud administrator"
cmd.exe /c "setspn -U -S http/blue-dc.blue.cloud administrator"

Configure the Name Resolution between Azure Stack and WAP

1. On ASDK Host Machine, open DNS management console and connect to AzS-DC01.AZURESTACK.LOCAL.
2. Add a conditional forwarding and forward the query for domain "BLUE.LCOUD" to IP address of the DNS server "BLUE-DC.BLUE.CLOUD".
3. On BLUE-DC.BLUE.CLOUD, open DNS management console and add a new DNS zone called "LOCAL.AZURESTACK.EXTERNAL".
4. Add an HOST record (A record) under that new created DNS zone "ADFS.LOCAL.AZURESTACK.EXTERNAL" and point to the external IP of AzS-BGPNAT01.

Configure the Trust between WAP Tenant Portal, Azure Stack's ADFS and BLUE.CLOUD's ADFS

@RuleTemplate = "LdapClaims"
@RuleName = "LDAP UPN"
c:[Type == "https://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname", Issuer == "AD AUTHORITY"]
 => issue(store = "Active Directory", types = ("https://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn"), query = ";userPrincipalName;{0}", param = c.Value);

@RuleTemplate = "LdapClaims"
@RuleName = "LDAP Groups"
c:[Type == "https://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname", Issuer == "AD AUTHORITY"]
 => issue(store = "Active Directory", types = ("https://schemas.xmlsoap.org/claims/Group"), query = ";tokenGroups(domainQualifiedName);{0}", param = c.Value);

@RuleTemplate = "PassThroughClaims"
@RuleName = "Passthru UPN"
c:[Type == "https://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn"]
 => issue(claim = c);

@RuleTemplate = "PassThroughClaims"
@RuleName = "Passthru Groups"
c:[Type == "https://schemas.xmlsoap.org/claims/Group"]
 => issue(claim = c);

#Save Azure Stack's ADFS Metadata
invoke-webrequest https://adfs.local.azurestack.external/FederationMetadata/2007-06/FederationMetadata.xml -OutFile "C:\FederationMetadata-AzureStack.xml"
#Save Azure Stack's CA Certificate
Copy-Item \\su1fileserver\SU1_Infrastructure_1\AzureStackCertStore\Internal\Current\RootCert\AzureStackCertificationAuthority.cer C:\

#Publish ADFS's HTTPS port
Enter-PSSession AzS-BGPNAT01.azurestack.local
$externalNic = Get-NetIPConfiguration | ? { [string]$_.IPv4Address -ne "192.168.200.1" }
$externalNicName = $externalNic.InterfaceAlias
$externalIP = $externalNic.IPv4Address.IPAddress + "/" + $externalNic.IPv4Address.PrefixLength
Add-NetNatExternalAddress -NatName BGPNAT -IPAddress $externalNic.IPv4Address.IPAddress -PortStart 443 -PortEnd 443
Add-NetNatStaticMapping -ExternalIPAddress $externalIP -ExternalPort 443 -InternalIPAddress "192.168.102.5" -InternalPort 443 -NatName BGPNAT -Protocol TCP

#Copy Rules.txt to AzS-ADFS01
Copy-Item C:\Rules.txt \\AzS-ADFS01.azurestack.local\C$
#Create trust between BLUE.CLOUD ADFS and Azure Stack ADFS
Enter-PSSession AzS-ADFS01.azurestack.local
Set-ADFSWebConfig –HRDCookieEnabled $false
Set-ADFSProperties -IgnoreTokenBinding $True
$policy = Get-AdfsAccessControlPolicy -Name "Permit everyone"
Add-ADFSRelyingPartyTrust -Name "Blue Cloud" `
    -MetadataUrl https://fs.blue.cloud/federationmetadata/2007-06/federationmetadata.xml `
    -AutoUpdateEnabled:$true  `
    -MonitoringEnabled:$true `
    -IssuanceTransformRulesFile "C:\rules.txt" `
    -AccessControlPolicyName $policy.name `
    -ClaimsProviderName @("Active Directory") `
    -EnableJWT $true

$policy = Get-AdfsAccessControlPolicy -Name "Permit everyone"
Add-AdfsRelyingPartyTrust -Name "WAP Tenant Portal" `
    -MetadataUrl "https://blue-wap.blue.cloud:30081/FederationMetadata/2007-06/FederationMetadata.xml" `
    -EnableJWT:$true `
    -AutoUpdateEnabled:$true `
    -IssuanceTransformRulesFile "C:\rules.txt" `
    -ClaimsProviderName @() `
    -AccessControlPolicyName $policy.name
Add-AdfsClaimsProviderTrust -Name "Azure Stack" `
    -MetadataFile "C:\FederationMetadata-AzureStack.xml" `
    -AutoUpdateEnabled:$false `
    -AcceptanceTransformRulesFile "C:\rules.txt"
Set-AdfsRelyingPartyTrust -TargetName "WAP Tenant Portal" -ClaimsProviderName @()

Import-Certificate -FilePath C:\AzureStackCertificationAuthority.cer -CertStoreLocation cert:\localmachine\root

$fqdn = 'fs.blue.cloud'
$dbServer = 'blue-wap.blue.cloud'
$dbpassword= 'User@123' 
$portalConfigStoreConnectionString = [string]::Format('Data Source={0};Initial Catalog=Microsoft.MgmtSvc.PortalConfigStore;User ID=sa;Password={1}', $dbServer, $dbPassword)

# Configure Tenant to use ADFS
Set-MgmtSvcRelyingPartySettings -Target Tenant `
    -MetadataEndpoint https://$fqdn/FederationMetadata/2007-06/FederationMetadata.xml `
    -DisableCertificateValidation `
    -ConnectionString $portalConfigStoreConnectionString

Verify the user AzureStackAdmin@AzureStack.local can login both Azure Stack and WAP

1. How to keep Azure Stack's IP range and DNS server unchanged when it's roaming among different networks?
2. How to connect my mini Azure Stack to a wireless network? (You don't expect have cable connection in coffee shop, do you?)

1. Connect the laptop to the mini Azure Stack with a network cable.
2. Install Hyper-V role on it.
3. Create an internal virtual switch and an external virtual switch. Bind the external virtual switch to the physical NIC which used to connect to mini Azure Stack.
4. Created a VM with two vNICs (one connect to the internal virtual switch, the other connect to the external virtual switch) on the laptop.
5. Assign the IP addresses to the above two vNICs, In my case, I use 192.168.2.1/24 and default gateway 192.168.2.254 on the vNIC which connect to Internal vSwitch and 192.168.20.1/24 on the vNIC which connect to external vSwitch.
6. Install DNS role on the VM and configure DNS forwarding on the DNS server. For example, forward all the DNS queries to external DNS server (say, 8.8.8.8).
7. Configure the physical NIC on mini Azure Stack to use IP address: 192.168.20.253/24; default gateway: 192.168.20.1; DNS server: 192.168.2.1. Later on I can deploy Azure Stack with static IP and fixed DNS server IP (192.168.2.1). (I use an intermediate DNS server here because in some networks, you might not be able to access the previous designated DNS server (say, 8.8.8.8). When that happened, you only need to update the DNS forwarding here.)

1. Open a PowerShell windows with administrative privilege.
2. Run cmdlet "Get-NetAdapter" and  identify the index number of the above internal virtual switch you created. In my case, it's 10.
3. Run the cmdlets below to configure that virtual switch to use NAT.

New-NetIPAddress -IPAddress 192.168.2.254 -PrefixLength 24 -InterfaceIndex 10
New-NetNat -Name NATnetwork -InternalIPInterfaceAddressPrefix 192.168.2.0/24

1. Install RRAS role on the above VM.
2. Configure NAT on the above VM, use vNIC which connect to internal virtual switch as the external network and use vNIC which connect to the external virtual switch as the internal network. (I know it's a little bit confusing here. Actually from DNS/NAT VM's point of view, the vNIC which connect to the external virtual switch is used to connect to mini Azure Stack and the vNIC which connect to the NAT enabled internal virtual switch is used to access Internet.)
3. Now you may validate and see if mini Azure Stack is able to access Internet (192.168.20.253 -> 192.168.20.1 -> 192.168.2.1 -> 192.168.2.254 -> <ip of laptop wireless network gateway> ->...)

$adminpass = ConvertTo-SecureString "xxx" -AsPlainText -Force
$aadpass = ConvertTo-SecureString "yyy" -AsPlainText -Force
$aadcred = New-Object System.Management.Automation.PSCredential ("serviceadmin@azurestack.onmicrosoft.com", $aadpass) 
.\InstallAzureStackPOC.ps1 -AdminPassword $adminpass -AADAdminCredential $aadcred -NATIPv4Subnet 192.168.20.0/24 -NATIPv4Address 192.168.20.254 -NATIPv4DefaultGateway 192.168.20.1 -EnvironmentDNS 192.168.2.1

Hardware

• Width 10" (254mm)
• Height 1.7" (43mm)
• Depth 8.9" (226mm)

Hardware

• ECGCAT-R4-16 (DELL 720, 2 E5-2650, 192GB Memory, 4 600 15K SAS HDD RAID10, 2 EMC P320h 700GB PCIe SSD, 1 Chelsio T580-CR NIC + 1 Mellanox ConnectX-3 NIC
• ECGCAT-R4-18 (DELL 720, 2 E5-2650, 192GB Memory, 4 600 15K SAS HDD RAID10, 2 EMC P320h 700GB PCIe SSD, 1 Chelsio T580-CR NIC + 1 Mellanox ConnectX-3 NIC
• ECGCAT-R4-20 (DELL 720, 2 E5-2650, 192GB Memory, 4 600 15K SAS HDD RAID10, 2 EMC P320h 700GB PCIe SSD, 1 Chelsio T580-CR NIC + 1 Mellanox ConnectX-3 NIC

Software

Test Case

Raw Performance of Mirrored Space

• 4KB Random IO, 100% Read
• 4KB Random IO, 90% Read, 10% Write
• 4KB Random IO, 70% Read, 30% Write

VM Fleet

• 4KB Random IO, 100% Read
• 4KB Random IO, 90% Read, 10% Write
• 4KB Random IO, 70% Read, 30% Write

Create Hyper-Converged Storage Space Direct Cluster

Assumption

Create Cluster

$nodes = ("ECGCAT-R4-16", "ECGCAT-R4-18", "ECGCAT-R4-20")
icm $nodes {
Install-WindowsFeature -Name Hyper-V, File-Services, Failover-Clustering -IncludeManagementTools -Restart
}
New-Cluster -Name ECGCAT-R4-C1 -Node $nodes -NoStorage -StaticAddress 192.168.20.50

Enable S2D and Create Space

Enable-ClusterS2D -AutoConfig:0
New-StoragePool -StorageSubSystemFriendlyName *Cluster* -FriendlyName S2D -ProvisioningTypeDefault Fixed -PhysicalDisk (Get-PhysicalDisk | ? CanPool -eq $true)
Get-StoragePool S2D* | Get-PhysicalDisk | Set-PhysicalDisk -Usage AutoSelect
Get-ClusterNode |% {New-Volume -StoragePoolFriendlyName S2D* -FriendlyName $_ -FileSystem CSVFS_ReFS -Size 600GB -PhysicalDiskRedundancy 1}
New-Volume -StoragePoolFriendlyName S2D* -FriendlyName collect -FileSystem CSVFS_ReFS -Size 138GB -PhysicalDiskRedundancy 1

Get-SmbClientNetworkInterface

Enable/Disable RDMA

$nodes = ("ECGCAT-R4-16", "ECGCAT-R4-18", "ECGCAT-R4-20")
icm $nodes {
Set-Netoffloadglobalsetting -Networkdirect Disabled; `
Update-Smbmultichannelconnection
}

icm $nodes {
Set-Netoffloadglobalsetting -Networkdirect Enabled; `
Update-Smbmultichannelconnection
}

Performance of Mirrored Space

4KB Random IO, 100% Read

diskspd.exe -c100G -d300 -w0 -t32 -o16 -b4K -r -h -L -D C:\ClusterStorage\VolumeX\IO.dat

4KB Random IO, 90% Read, 10% Write

diskspd.exe -c100G -d300 -w10 -t32 -o16 -b4K -r -h -L -D C:\ClusterStorage\VolumeX\IO.dat

4KB Random IO, 70% Read, 30% Write

diskspd.exe -c100G -d300 -w30 -t32 -o16 -b4K -r -h -L -D C:\ClusterStorage\VolumeX\IO.dat

VM Fleet Performance Test

Prepare VM Fleet

$nodes = ("ECGCAT-R4-16", "ECGCAT-R4-18", "ECGCAT-R4-20")
.\create-vmfleet.ps1 -basevhd "C:\ClusterStorage\collect\VMFleet-ENGWS2016COREG2VM.vhdx" -vms 8 -adminpass User@123 -connectpass User@123 -connectuser "infra\administrator" -nodes $nodes
.\set-vmfleet.ps1 -ProcessorCount 8 -MemoryStartupBytes 16GB -DynamicMemory $false
.\start-vmfleet.ps1

4KB Random IO, 100% Read

.\start-sweep.ps1 -b 4 -t 1 -o 120 -w 0 -d 300

4KB Random IO, 90% Read

.\start-sweep.ps1 -b 4 -t 1 -o 120 -w 10 -d 300

4KB Random IO, 70% Read

.\start-sweep.ps1 -b 4 -t 1 -o 120 -w 30 -d 300

Conclusion

Kudo to Chelsio

Prerequisites

• Windows Server 2016
• VMM 2016
• SPF 2016 (You may find it from Orchestrator 2016's media.)
• WAP – UR10

Configure VMM

1. As as the deployment of WAP+SPF2012 R2+WMM 2012 R2, you need to create a cloud in VMM. In my setup its name is "Infra Cloud".
2. On the resource tab, select the corresponding host group.
3. In the logical Networks tab, please select HNV logical network
4. On the capability Profiles tab, clear all the checkboxes.
5. As for the steps didn't mentioned above, you may either use the default option/value or select based on your environment.
6. After you create the cloud, you may want to create a VM in that cloud from VMM Console and make sure the provision could complete successfully.

Install SPF

Import-Module ServerManager
Add-WindowsFeature FileAndStorage-Services,Storage-Services,Web-Server,Web-WebServer,Web-Common-Http,Web-Default-Doc,Web-Dir-Browsing,Web-Http-Errors,Web-Static-Content,Web-Health,Web-Http-Logging,Web-Request-Monitor,Web-Http-Tracing,Web-Performance,Web-Stat-Compression,Web-Dyn-Compression,Web-Security,Web-Filtering,Web-Basic-Auth,Web-Url-Auth,Web-Windows-Auth,Web-App-Dev,Web-Net-Ext45,Web-Asp-Net45,Web-ISAPI-Ext,Web-ISAPI-Filter,Web-Ftp-Server,Web-Mgmt-Tools,Web-Mgmt-Console,Web-Mgmt-Compat,Web-Scripting-Tools,NET-Framework-Features,NET-Framework-Core,NET-Framework-45-Features,NET-Framework-45-Core,NET-Framework-45-ASPNET,NET-WCF-Services45,NET-WCF-HTTP-Activation45,NET-WCF-TCP-PortSharing45,ManagementOdata,FS-SMB1,PowerShellRoot,PowerShell,PowerShell-V2,PowerShell-ISE,WAS,WAS-Process-Model,WAS-Config-APIs,WoW64-Support

Configure WAP Admin Portal

1. On t he WAP Admin portal, clikc VM CLoud and click "Register System Center Service Provider Foundation" on the first page.
2. Input the service URL of your SPF server. In my case, it's https://infra-vmm.infra.lab:8090. Also input the username and password which has the permission to access SPF.
3. Then add the existing VMM server. In my case, it's infra-vmm.infra.lab.
4. Then you can add/configure a Plan to add plan service "Virtual Machine Clouds". (In my setup, the plan is "IaaS".)
5. You may configure the plan service and use the existing VMM server and Virtual Machine Cloud configured in VMM console. The rest of  the configuration is same as the previous WAP VM Cloud configuration.

Validate from Tenant Portal

NC based VM network creation and management

1. Create a Virtual Network > Custom Create.
2. Then give it a name and click Next.
3. Configure its DNS server (in my setup, the DNS server is 192.168.20.3). And click Next.
4. Accept the default address space, which is 10.0.0.0/24. Click OK. Now you will see the Virtual Network Created successfully.
5. Then you may create a new VM to use above new create virtual network.

Outbound NAT

1. Click the new created Virtual Network and select the "CONFIGURE" page. Select the checkbox "Enable direct internet access using NAT".
2. You may login your new created VM in the above virtual network and check if it could access external network

Inbound NAT

1. Click the new created Virtual Network and select the "RULES" page. Add a new rule as below and click OK.

S2S IPSec Gateway connectivity

1. Click the new created Virtual Network and select "SITE-TO-SITE VPN" page and click "CREATE VPN".
2. Give it a name and input the remote VPN device's IP address.
3. Specify the address space of remote site.
4. Set the key same as the one you used to configure the remote VPN device.
5. Accept the default value on other pages including using the default "Unlimited" VPN incoming and outgoing bandwidth setting.
6. Click OK to finish the wizard.
7. On the dashboard of the virtual network, you may find the Gateway IP address.
8. You may follow the steps in the following post to configure the remote VPN device. https://blogs.technet.microsoft.com/larryexchange/2016/06/01/step-by-step-for-deploying-a-sdnv2-using-vmm-part-4/
9. You may connect to VM in the above virtual network and verify the S2S IPSec connection.

Other SDN functionalities

Network Security Group (NSG a.k.a., Distributed Firewall)

Software Load Balancer (SLB)

• CPU: E5-2650
• Memory: 192GB
• Boot Drive: H710 with 4 15K 600GB SAS HDDs (RAID10)
• Data Drive: Micron P420m 1400GB
• NIC: Chelsio T580 40Gb (RDMA capable)

Get-ClusterNode |% {New-Volume -StoragePoolFriendlyName S2D* -FriendlyName $_ -FileSystem CSVFS_ReFS -Size 600GB -PhysicalDiskRedundancy 1}

New-Volume -StoragePoolFriendlyName S2D* -FriendlyName collect -FileSystem CSVFS_ReFS -Size 138GB -PhysicalDiskRedundancy 1

.\install-vmfleet.ps1 -source D:\VMFleet

1. I used Convert-WindowsImage.ps1 (https://gallery.technet.microsoft.com/scriptcenter/Convert-WindowsImageps1-0fe23a8f) to convert a Windows Server 2016 WIM image to a Server Core VHD.
2. Use the above VHD to start a VM and set the initial password. (In my case, I use the password User@123.)
3. Shutdown the VM (DON'T syspreped the  VM.)
4. Copy the VHD to C:\ClusterStorage\Collect\. (In my case, the VHD is ENGWS2016COREG2VM.vhdx.)

.\create-vmfleet.ps1 -basevhd "C:\ClusterStorage\Collect\ENGWS2016COREG2VM.vhdx" -vms 10 -adminpass User@123 -connectpass User@123 -connectuser "infra\administrator" -nodes "ECGCAT-R4-22","ECGCAT-R4-24","ECGCAT-R4-26"

.\set-vmfleet.ps1 -ProcessorCount 8 -MemoryStartupBytes 16GB -DynamicMemory $false

.\start-vmfleet.ps1

.\Watch-Cluster.ps1

.\start-sweep.ps1 -b 4 -t 2 -o 40 -w 0 -d 300

.\start-sweep.ps1 -b 4 -t 2 -o 20 -w 30 -d 300

• Some of the people don't have 10Gb physical network in the lab but internal and private virtual switch can provide 10Gb link (although due to hardware limitation, they might not always be able to achieve 10Gb/s throughput).
• If you want to demo S2D in an event or a customer meeting, you probably don't want to waste your time troubleshooting  the network connectivity issue.

Understand IPsec and GRE Gateway in Windows Server 2016

• GRE Tunneling in Windows Server Technical Preview
• RAS Gateway Deployment Architecture
• RAS Gateway High Availability
• Leverage GRE Tunneling to enable the multi-tenant direct routing on Windows Server Gateway

IPSec

• The Top-of-the-Rack switch (or BGP Router in Provider Address space / Private Cloud fabric) is connected to SLB
• ToR actually has a BGP Peering setup with SLB MUXes to learn all the SLB VIPs that need to be advertised to public network (internet)
• Tenant’s Enterprise GW connects to this Public VIP and SLB MUX will internally translate it to DIP Address which will belong to one of the Gateway VMs (GW VM’s Transit IP Address)

GRE

• The only difference here is that Gateway VMs will directly peer with ToR and advertise their own GRE VIPs
• ToR again needs to be configured to accept connections from all possible Gateway VM Transit IP Addresses.

L3

BGP Routing at Enterprise GW

Create GRE VIP logical network required for gateway deployment

Create the GRE VIP logical network

1. From the VMM console, start the Create Logical Network Wizard.
2. Type a name and optional description for this network and click Next.
3. On the Settings page, ensure you select One Connected Network. Optionally, you can also check Create a VM network with the same name box to allow virtual machines to access this logical network directly and the Managed by the Network Controller box then click Next.
4. On the Network Site panel, add the network site information for your VIP subnet. This should include the Host Group and subnet information for your VIP network.
5. Review the Summary information and complete the Logical Network wizard

Create an IP pool for GRE VIP addresses

1. Right-click the GRE VIP logical network in VMM and select Create IP Pool from the drop down menu.
2. Type a name and optional description for the IP Pool and ensure that the VIP network is selected for the logical network. Click Next.
3. Accept the default network site and click Next.
4. Choose a starting and ending IP address for your range.

   Important
   Start your range on the fourth addresses of your available subnet. For example, if your available subnet is from .1 to .254, start your range at .4.

5. In the IP addresses reserved for load balancer VIPs box, type the IP addresses range in the subnet. This should match the range you used for starting and ending IP addresses.
6. You do not need to provide gateway, DNS or WINS information as this pool is used to allocate IP addresses for VIPs only via the network controller, so click Next to skip these screens.
7. Review the summary information and complete the wizard.

Deploy Gateway Service

1.  In VMM, navigate to the Library.
2. In the top of the left pane, in the Templates section, select Service Templates.
3. In the ribbon at the top, click Import Template.
4. Browse to your service template folder, select EdgeServiceTemplate Generation2.xml and click Next.
5. Update the parameters to reflect the your environment configuration .
6. Click Next.
7. On the Summary page, click Import.
8. Same as the pervious blog, if your syspreped image is based on volume license image, you may jump to the step 12.
9. (Optional) If your syspreped image is not based on volume license image, we need to modify the service template and input the product key. You may right click the template and select "Open Designer".
10. Then right click "Windows Server Gateway" and select "Properties".
11. On "OS Configuration" page, input the product key.
12. Select the EdgeServiceTemplate Generation2.xml service template and click Configure Deployment to begin. Type a name and choose a destination for the service instance. The destination must map to a Host Group that contains the hosts configured previously for gateway deployment purposes.
13. Give the service a name.
14. Click OK.
15. On the left side of the Configure Deployment window, there are a number of settings that you must configure.
16. After you configure these settings, you can click Deploy Service to begin the service deployment job. Deployment times will vary depending on your hardware but are typically between 30 and 60 minutes.
17. You may check the result of the deployment and make sure it's successful.

Configure the Gateway Manager Role

1. Open the Fabric workspace.
2. Click Network Service to display the list of network services installed.
3. Right-click your network controller service and select Properties.
4. Click the Services tab and select the Gateway Manager Role in the services panel.
5. Find the Associated Service field under Service information and click Browse.
6. Select the gateway service instance you created earlier and click OK.
7. Public IPv4 address, provide an IP address from the previous pool, and ensure you don’t select the initial 3 IP addressed from the range. In my case, I used 192.168.133.99. (192.168.133.100 had already been used as VIP of SLB in my previous test.)In Public IPv4 pool, select the Public IP Pool you configured during the SLB deployment. You will use the IP as the destination IP on the Enterprise gateway.
8. In GRE VIP subnet, select the VIP subnet that you created previously.
9. Select the Run As account that will be used by network controller to access the gateway virtual machines.
10. On each node's configuration page, select the IPv4 Frontend Subnet.
11. Specify the local ASN.
12. Add the peering device info for the BGP peer.
13. Click OK. Then check the job "Update network service device" complete successfully.

Configure BGP Router

add-bgppeer -Name GW01 -LocalIPAddress 192.168.30.1 -PeerIPAddress 192.168.30.56 -LocalASN 65000 -PeerASN 65011 -OperationMode Mixed -PeeringMode Automatic
add-bgppeer -Name GW02 -LocalIPAddress 192.168.30.1 -PeerIPAddress 192.168.30.55 -LocalASN 65000 -PeerASN 65012 -OperationMode Mixed -PeeringMode Automatic
add-bgppeer -Name GW03 -LocalIPAddress 192.168.30.1 -PeerIPAddress 192.168.30.54 -LocalASN 65000 -PeerASN 65013 -OperationMode Mixed -PeeringMode Automatic
get-bgppeer

Get-BgprouteInformation -type all

Validation

Test Case 4: S2S IPSec connection

1. Open the properties dialog window of an existing VM Network called "VNET-IPSEC".
2. On connectivity page and select the checkbox "Connect to another network through a VPN tunnel". (Optional) You may also enable NAT for that VM network if you want the VM in that VM network to be able to access external network.
3. Select the VPN Connections tab.
4. Type a subnet as shown in the following diagram. This subnet is used to route packets out of the VM Network. This subnet need not be pre-configured in your datacenter.Click Add and select the IPSec type connection.
5. Give the connection a name (e.g., IPSEC-VMM-RTM).
6. Type the IP address of the Remote endpoint. In my setup, it's 192.168.30.2, which is the remote IPSec gateway's external IP.
7. Select the Authentication tab.
8. Choose your preferred authentication method for the connection. If you choose the authentication using ‘Run As Account’, then you need to create a user account with the username of your choice and the IPSec key as the password for the account. Browse and select this account as your Run As Account. In my case, I use the account "Run As". The account's password will be used as IPsec key.
9. Select the Routes tab and Type all the remote subnets that you need to connect to.
10. You may accept the default setting on Advanced tab. Then click "OK".
11. Make sure the VPN connection for VM Network gateway had already been added successfully.
12. Different from Windows Server 2012 R2, on Windows Server 2016, all the IPSec VPN will share the same external IP, which is the public IP (VIP) you configured on the Gateway Service page of Network Controller's properties dialog window. You cannot see it from individual VM Network's properties any more.

1. Open Routing and Remote Access console.
2. Right click "Network Interface" and select "New Demand-dial Interface".
3. Type a name and click "Next".
4. Select "Connect using virtual private network (VPN)". Then click "Next".
5. Select "IKEv2", click "Next".
6. Type the VIP of the gateway service. In my case, it's 10.10.10.99.
7. Select the checkbox "Route IP packets on this interface". Then click "Next".
8. Add the subnets of the VM network you want to connect to. In my case it's "12.0.0.0/24". Then click "Next".
9. Accept the default setting on the "Dial-Out Credentials" page and click "Next".
10. Click "Finish".
11. Open the properties dialog window of the new created VPN interface. On the Option tab select Connection Type "Persistent connection". Optional you may also modify the dialing policy.
12. On the Security tab, select "Use Pre-Shared key for authentication and key in the password. Then click "OK".
13. Right click the new created VPN interface and select "Connect".
14. Now you will see the interface connected.
15. You may login the TESTVMIPSEC (12.0.0.4) and check if you could ping enterprise gateway (12.1.1.254) and the VM in the remote network (12.1.1.4).

Test Case 5: To validate S2S GRE connections

1. Select the VM Network where you want to configure a S2S GRE connection and click Connectivity.
2. Select Connect to another network through a VPN tunnel. You may also enable the outbound NAT for the VM Network if you want the VMs in that VM Network to be able to access external network. And select your Network Controller Service for the Gateway Device.
3. Select the VPN Connections tab. Click Add and then select GRE type connection. Type a subnet as shown in the following diagram. This subnet is used to route packets out of the VM Network. This subnet does not need to be preconfigured in your datacenter.
4. Give the connection a name. The name used in the example screenshot is GRE.
5. Type the IP address of the Remote endpoint. In my setup, it's 192.168.30.3, which is the external IP of the remote GRE gateway.
6. Type the GRE key.
7. Select the Routes tab.
8. Type all the remote subnets that you need to connect to.

1. Same as the IPsec Gateway, you could not find the external IP from VMM Console in VMM TP5.
2. You have to access https://<NC Management IP>/Networking/v1/VirtualGateways and download JSON file. Then you may find the sourceIPAddress from the JSON file. That's the external IP of the gateway. In my case, it's 30.30.30.51.
3. Open PowerShell window and run the cmdlet below.

   Add-VpnS2SInterface -Name GREtoTenant -Destination 30.30.30.51 -IPv4Subnet "11.0.0.0/24:100" -GreTunnel -GreKey "1234" -SourceIpAddress: 192.168.30.3

4. Run the cmdlet below to verify GRE interface connected and there is traffic on it.

   Get-VpnS2SInterface -name GREtoTenant

   Get-VpnS2SInterface -Name GREtoTenant | Get-VpnS2SInterfaceStatistics

   
5. To validate that your S2S GRE connectivity is configured properly, you may login the TESTVMGRE (11.0.0.4) and try to ping the remote GRE gateway's internal IP (11.1.1.254) and a VM in the remote network (11.1.1.1).

Test Case 6: To validate L3 routing between HNV VM network and VLAN based VM Network

• Multiple HNV VM networks connect to the same VLAN based VM Network
• HNV VM networks connect to the logical network's VM Network directly

1.  Right click "Logical Network" and select "Create Logical Network".
2. Give it a name. In my case, I use the name "TENANT101".
3. Select the check box "Create a VM network with the same name to allow virtual machines to access this logical network directly" and "Managed by Microsoft Network Controller".  Please be informed it's different from the traditional VLAN based VM Network.
4. On the "Network Site" tab, select "All Hosts" and input the corresponding VLAN ID and subnet. In my case, VLAN ID is 101 and Subnet is "192.168.11/24". Accept other default values.
5. On the Summary tab click "Finish".
6. Create a IP Pool under the above new created logical network. Give it a name and accept the default value on the "Network Site" tab.
7. Configure the Starting IP address and Ending IP address of the IP Pool. In my case, it's 192.168.11.51 and 192.168.11.100.
8. Insert the default gateway
9. Insert the DNS server.
10. You may skip the WINS configuration and click Finish on Summary tab.
11. Then you may create VM (in my setup , it's TENANT-L3-VM101) in the above new created VM Network.

param(
[Parameter(Mandatory=$true)]
# Name of the L3 VPN connection
$L3VPNConnectionName,
[Parameter(Mandatory=$true)]
# Name of the VM network to create gateway
$VmNetworkName,
[Parameter(Mandatory=$true)]
# Name of the Next Hop one connected VM network
# used for forwarding
$NextHopVmNetworkName,
[Parameter(Mandatory=$true)]
# IPAddresses on the local side that will be used
# for forwarding
# Format should be @("10.10.10.100/24")
$LocalIPAddresses,
[Parameter(Mandatory=$true)]
# IPAddresses on the remote side that will be used
# for forwarding
# Format should be @("10.10.10.200")
$PeerIPAddresses,
[Parameter(Mandatory=$false)]
# Subnet for the L3 gateway
# default value 10.254.254.0/29
$GatewaySubnet = "10.254.254.0/29",
[Parameter(Mandatory=$false)]
# List of subnets for remote tenants to add routes for static routing
# Format should be @("14.1.20.0/24","14.1.20.0/24");
$RoutingSubnets = @()
)

Import-Module virtualmachinemanager

$vmNetwork = Get-SCVMNetwork -Name $VmNetworkName;

$nextHopVmNetwork = Get-SCVMNetwork -Name $NextHopVmNetworkName;

$gatewayDevice = Get-SCNetworkGateway | Where {$_.Model -Match "Microsoft Network Controller"};

$vmNetworkGatewayName = $VmNetwork.Name + "_Gateway";
$VmNetworkGateway = Add-SCVMNetworkGateway -Name $vmNetworkGatewayName -EnableBGP $false -NetworkGateway $gatewayDevice
    -VMNetwork $vmNetwork -RoutingIPSubnet $GatewaySubnet;

$vpnConnection = Add-SCVPNConnection  -NextHopNetwork $nexthopvmNetwork  -Name $L3VPNConnectionName
    -IPAddresses $LocalIPAddresses -PeerIPAddresses $PeerIPAddresses -VMNetworkGateway $VmNetworkGateway -protocol L3;
Write-Output "Created VPN Connection " $vpnConnection;

foreach($route in $RoutingSubnets)
{
    Add-SCNetworkRoute -IPSubnet $route -RunAsynchronously -VPNConnection $vpnConnection
        -VMNetworkGateway $VmNetworkGateway
}

.\Config-L3.ps1 -L3VPNConnectionName L3 -VmNetworkName VNET-L3 -NextHopVmNetworkName TENANT101 -LocalIPAddresses @("192.168.11.253/24") -PeerIPAddresses @("192.168.11.254") -GatewaySubnet 10.254.254.0/29 -RoutingSubnets @("192.168.11.0/24")

route ADD 13.0.0.0 MASK 255.255.255.0 192.168.11.253 METRIC 3 -p

Kudo to SDN & VMM team

• Manish Jha
• Ramandeep Singh Dhillon
• Subba Raju Thikkireddy
• Pradeep Reddy
• Sanjay Pavan Chaluvadi

Create Logical Network "TRANSIT", "Private VIP" and "Public VIP"

1. From the VMM console, start the Create logical network Wizard.
2. Type a name and optional description for this network and click Next.
3. On the Settings page, ensure you select One Connected Network. Check Create a VM network with the same name box to allow virtual machines to access this logical network directly and the Managed by the network controller box then click Next.
4. On the Network Site panel, add the network site information for your subnet.
5. Review the Summary information and complete the logical network wizard
6. Create IP Address pool for the new created TRANSIT logical network.
7. Use the existing network site.
8. Set starting IP address 192.168.30.51 and ending IP address 192.168.30.100.
9. The default gateway is 192.168.30.1.
10. On Summary page, click Finish.
11. Start the Create logical network Wizard.
12. Type a name and optional description for this network. Click Next.
13. On the Settings page, ensure you select One Connected Network. check Create a VM network with the same name box to allow virtual machines to access this logical network directly and the Managed by the network controller box then click Next.
14. On the Network Site panel, add the network site information for your Private VIP logical network. VLAN ID is "0" and in my case the IP subnet is "20.20.20.0/24".
15. Review the Summary information and complete the wizard.
16. Start the Create logical network Wizard.
17. Type a name and optional description for this network and click Next.
18. On the Settings page, ensure you select One Connected Network. Check the Create a VM network with the same name box to allow virtual machines to access this logical network directly and the Managed by the network controller box. Also select Public IP Address Network and then click Next.
19. On the Network Site panel, add the network site information for your Public VIP network. This should include the Host Group and subnet information for your Public VIP network.
20. Review the Summary information and complete the logical network wizard
21. Right click "Uplink-MGMT" and select Properties.
22. Associate this logical network with the Management switch uplink port profile you created during the network controller deployment.

Create VIP Pool for Private VIP Logical Network

1.  Right-click the Private VIP logical network in VMM and select Create IP Pool from the drop down menu.
2. Provide a name and optional description for the IP Pool and ensure that the Private VIP logical network is selected for the logical network. Click Next.
3. Accept the default network site and click Next.
4. Choose a starting and ending IP address for your range.

   Important
   Start your range on the fourth addresses of your available subnet. For example, if your available subnet is from .1 to .254, start your range at .4.

5. In the IP addresses reserved for load balancer VIPs box, type the IP addresses range in the subnet. This should match the range you used for starting and ending IP addresses.
6. You do not need to provide gateway, DNS or WINS information as this pool is used to allocate IP addresses for VIPs only via the network controller, so click Next to skip these screens.
7. Review the summary information and complete the wizard.

Create VIP Pool for Public VIP Logical Network

1. Right-click the Public VIP logical network in VMM and select Create IP Pool from the drop down menu.
2. Provide a name and optional description for the IP Pool and ensure that the VIP network is selected for the logical network. Click Next.
3. Accept the default network site and click Next.
4. Choose a starting and ending IP address for your range.

   Important
   Start your range on the fourth addresses of your available subnet. For example, if your available subnet is from .1 to .254, start your range at .4.

5. In the IP addresses reserved for load balancer VIPs box, type the IP addresses range in the subnet. This should match the range you used for starting and ending IP addresses.
6. You do not need to provide gateway, DNS or WINS information as this pool is used to allocate IP addresses for VIPs only via the network controller, so click Next to skip these screens.
7. Review the summary information and complete the wizard
8. Now you had already configured all the logical network and IP pool.

Create SLB service

1. In the VMM console, navigate to Library.
2. In the top of the left pane, in the Templates section, select Service Templates.
3. In the ribbon at the top, click Import Template.
4. Browse to your service template directory, select the SLB Production Generation 2 VM.xml file and follow the prompts to import it.
5. This service template uses the following virtual machine configuration parameters. Update the parameters to reflect your environment configuration.
6. On Summary, click "Import".
7. If your syspreped image is based on volume license image, you may jump to step 11.
8. (Optional) If your syspreped image is not based on volume license image, you need to modify the service template and input the product key. Right click "SlbMuxServiceTemplate" and select "Open Designer".
9. Right click "Software Load Balancer" and select "Properties".
10. Type the product key.
11. Right click "Software Load Balancer" and select "Configure Deployment".
12. Type the name "SLB-VMM-RTM". Select the corresponding Transit network and Management network.
13. Configure the settings in the page below. Then click "Refresh". Then click "Deploy Service".
14. Make sure the deployment complete successfully.
15. Open the Fabric workspace.
16. Click Network Service to display the list of network services installed.
17. Right-click the network controller service and select Properties.
18. In the Wizard, Select the Services tab, and then click Load Balancer Role
19. Find the Associated Service field under Service information and click Browse. Select the SLB/MUX service instance you created earlier and click OK.
20. Choose the appropriate Run as Account.
21.  For the Management IP address, use the last IP address from the Private VIP pool you created earlier. In my case, it's 20.20.20.100. In addition, please select Public IP pool and private IP pool you just created. VMM will advertise those VIP address pools to the SLB Manager.
22. Click the SLB/MUX instance listed under Load Balancer Role in the wizard.
23. Type the local ASN for your datacenter and details for the devices or BGP peers the SLB/MUX can peer with. You should use the Transit network for BGP peering. (In TP5, we use MGMT network for BGP Peering.) 
24. Make sure associate service instance with fabric role.
25. The SLB Service instance is now associated with the SLBM service, and you should see the SLB/MUX virtual machine instance with all the settings listed under the Load Balancer role.

Configure BGP Router

Add-BgpRouter -BgpIdentifier 192.168.30.1 -LocalASN 65000

add-bgppeer -Name MUX001 -LocalIPAddress 192.168.30.1 -PeerIPAddress 192.168.30.53 -LocalASN 65000 -PeerASN 65001 -OperationMode Mixed -PeeringMode Automatic
add-bgppeer -Name MUX002 -LocalIPAddress 192.168.30.1 -PeerIPAddress 192.168.30.52 -LocalASN 65000 -PeerASN 65002 -OperationMode Mixed -PeeringMode Automatic
add-bgppeer -Name MUX003 -LocalIPAddress 192.168.30.1 -PeerIPAddress 192.168.30.51 -LocalASN 65000 -PeerASN 65003 -OperationMode Mixed -PeeringMode Automatic

Create a VIP Template

1. Navigate to the Fabric Workspace in the VMM console.
2. Right click the VIP Templates node and select Create VIP Template, or alternately you can click the Create VIP Template in the ribbon toolbar.
3. Type a name in the Template Name field and an optional description in the Description field.
4. In the Virtual IP Port field, provide a value for the port you wish to test.
5. For the Backend Port, provide a value for the port from which you wish to map traffic on the back end.
6. Click Next.
7. On the Specify a Template Type screen, click Specific, and select Microsoft for the Manufacturer and for the Model, select Microsoft network controller. Click Next.
8. On the Specify Protocol Options screen, select the Custom option and type TCP in the Protocol Name field. Click Next.
9. You can optionally select enable persistence if you wish to have the load balancer make the connection from the client “sticky”. Click Next.
10. For the Load Balancing method, select Round Robin from the drop down list. Click Next.
11. On Health Monitors page, click Next to move past this screen.
12. Confirm your settings and then click Finish when you are ready to create the VIP Template.

Validation

Test Case 2: SLB

param(

[Parameter(Mandatory=$false)]
# Name of the Network Controller Network Service
# This value should be the name you gave the Network Controller service
# when you on-boarded the Network Controller to VMM
$LBServiceName = "NC",

[Parameter(Mandatory=$false)]
# Name of the VM instances to which you want to assign the VIP
$VipMemberVMNames =  @("TESTVMSLB1","TESTVMSLB2"),

[Parameter(Mandatory=$false)]
# VIP address you want to assign from the VIP pool.
# Pick any VIP that falls within your VIP IP Pool range.
$VipAddress = "10.10.10.100",

[Parameter(Mandatory=$false)]
# Name of the VIP VM Network
$VipNetworkName = "PublicVIP",

[Parameter(Mandatory=$false)]
# The name of the VIP template you created via the VMM Console.
$VipTemplateName = "WEB",

[Parameter(Mandatory=$false)]
# Arbitrary but good to match the VIP you're using.
$VipName = "scvmm_10_10_10_100_80"

)

Import-Module virtualmachinemanager

$lb = Get-scLoadBalancer | where { $_.Service.Name -like $LBServiceName};
$vipNetwork = get-scvmnetwork -Name $VipNetworkName;

$vipMemberNics = @();
foreach ($vmName in $VipMemberVMNames)
{
$vm = get-scvirtualmachine -Name $vmName;
#    if ($vm.VirtualNetworkAdapters[0].VMNetwork.ID -ne $vipNetwork.ID)
#    {
#        $vm.VirtualNetworkAdapters[0] | set-scvirtualnetworkadapter -VMNetwork $vipNetwork;
#    }

$vipMemberNics += $vm.VirtualNetworkAdapters[0];
}

$existingVip = get-scloadbalancervip -Name $VipName
if ($existingVip -ne $null)
{
#    foreach ($mem in $existingVip.VipMembers)
#    {
#        $mem | remove-scloadbalancervipmember;
#    }

$existingVip | remove-scloadbalancervip;
}

$vipt = get-scloadbalancerviptemplate -Name $VipTemplateName;

$vip = New-SCLoadBalancerVIP -Name $VipName -LoadBalancer $lb -IPAddress $VipAddress -LoadBalancerVIPTemplate $vipt -FrontEndVMNetwork $vipNetwork -BackEndVirtualNetworkAdapters $vipMemberNics;
Write-Output "Created VIP " $vip;

$vip = get-scloadbalancervip -Name $VipName;
Write-Output "VIP with members " $vip;

Test Case 3: SNAT

1. Open the VMM Administrator console and select the VMs and Services tab.
2. Select the VM Networks tab and then double click the VM network you want to configure with NAT rules.
3. Select the Connectivity tab on the property wizard.
4. Check Connect directly to an additional Network and select Network Address Translation (NAT).
5. In Gateway Device, select your network controller service name.
6. Select the Network Address Translation tab. Then select the public IP pool.
7. In the IP address pool field, choose your Public VIP pool.
8. Leave the IP address field empty. A VIP address will be automatically assigned to this rule.
9. Make sure add VM Network gateway successfully. And add the corresponding NAT rules successfully.
10. You may also check the NAT connection and NAT rules with PowerShell.
11. You may also verify both inbound NAT and outbound NAT work as expected.

To Be Continued...

Add Hyper-V Hosts to the VMM

1. Open VMM Console, click "Fabric" and expand the "Servers > All Hosts".
2. Right click the "All Hosts" and select "Add Hyper-V Hosts and Clusters"
3. Select "Windows Server computer in a trusted Active Directory domain".
4. In the next page, I created a Run As account, which is also a member domain admins group (That's not recommended for production environment.).
5. Key in the computer name of the each Hyper-V hosts. Then click "Next".
6. Click the button "Select All", then click "Next".
7. Select the checkbox "Re-associate this host with this VMM environment" and add the C:\ as the VM placement path.
8. On next page confirm the settings and click "Finish".
9. Now you could see VMM is adding those Hyper-V hosts to the VMM environment.
10. After the above 4 jobs complete, you would see 4 Hyper-V hosts in the "All Hosts" group.

Create Logical Network and Logical Switch for "MGMT"

1. Right click "Networking > Logical Network" and select "Create Logical Network".
2. Type the name "MGMT".
3. Select the checkbox "Create a VM network with the same name to allow virtual machines to access this logical network directly.". Then click "Next".
4. Select the host group "All Hosts" and add the VLAN (ID=0, IP Subnet = 192.168.2.0/24).
5. Confirm the settings and click "Finish".
6. Right click the logical network you just created. Select "Create IP Pool".
7. Give it a name "IPPool-MGMT". Click "Next".
8. Accept the default Network Site, click "Next".
9. Configure the IP pool start from 192.168.20.51 and end at 192.168.20.100. Then click "Next".
10. Insert the default gateway address 192.168.20.1. Then click "Next".
11. Insert the DNS server address 192.168.20.3. Then click "Next".
12. You don't need specify WINS. Just click "Next" and confirm the settings on the next page. Last click "Finish".
13. Right click "Network > Port Profiles", select "Create Hyper-V Port Profile".
14. Give it a name "Uplink-MGMT". Select the option "Uplink port profile".
15. Select the site "MGMT_0". DO NOT select the checkbox "Enable Hyper-V Networking Virtualization".
16. On the confirm the settings page, click "Finish".
17. Right click "Networking > Logical Switch" and select "Create Logical Switch".
18. Type the name "SDN-Switch" and Select Uplink mode "Embedded Team"
19. On the next page, accept the default setting and click "Next".
20. Clear the checkbox "Microsoft Window Filtering Platform". New SDN stack uses Virtual Filtering Platform (VFP) from Azure instead of the default Windows Filtering Platform.
21. In this test, I don't use any virtual port. So I skip the Virtual Port page.
22. On the Uplink page, click "Add > Existing Uplink Port Profile".
23. Select the uplink profile we just created. It's Uplink-MGMT in my case.
24. Click the button "New Virtual Network Adapter". Type the name "MGMT". Select the checkbox "This virtual network adapter will be used for host management" and "Inherit connection settings from host network adapter". Since I have DHCP server available in the management VLAN, I selected "DHCP" as my IP address configuration. If you don't have DHCP, you may select "Static" and select the corresponding IP pool (e.g., "IPPool-MGMT").
25. Right click one of the Host and select "Properties".
26. Click "Virtual Switch" and click the button "New Virtual Switch". Then select "New Logical Switch".
27. Bind the logical switch to the corresponding NICs. In my case, I have one NIC on the host. If you have multiple NICs, you may add them to the "Physical Adapters" list.
28. Use the same procedure you may bind logical switch "SDN-Switch" to the rest hosts.
29. Remove the VLAN ID from all the physical NICs which bind to the above logical switch.

Create NC Network Service

1. Copy a syspreped Windows Server 2016 image to the share folder of VMM VM \\localhost\MSSCVMMLibrary\VHDs. Volume License (VL) image is recommended. Otherwise you would have to modify the service template later and input the product key.
2. Download the SDN VMM template. Unzip the compressed file and copy the folder GW, NC and SLB to the share folder \\localhost\MSSCVMMLibrary\
3. Run the script below create a self-signed certificate and export two version of the certificate (cer and pfx) to C:\. Then copy the exported certificate files to the corresponding folders.

   $cert = New-SelfSignedCertificate -KeyUsageProperty All -Provider "Microsoft Strong Cryptographic Provider" -FriendlyName "MultiNodeNC" -DnsName @("192.168.2.34")
   Export-Certificate -Cert $cert -FilePath c:\MultiNodeNC.cer
   $PlainPassword = "123"
   $SecurePassword = $PlainPassword | ConvertTo-SecureString -AsPlainText -Force
   Export-PfxCertificate -Cert $cert -FilePath c:\MultiNodeNC.pfx -Password $securepassword -ChainOption BuildChain
   copy C:\MultiNodeNC.pfx \\localhost\MSSCVMMLibrary\ServerCertificate.cr\
   copy C:\MultiNodeNC.cer \\localhost\MSSCVMMLibrary\TrustedRootCertificate.cr\
   copy C:\MultiNodeNC.cer \\localhost\MSSCVMMLibrary\NCCertificate.cr\

4. Click "Library" in VMM console and browse to "Template > Service Template". Click the button "Import Template"
5. Browse to \\localhost\MSSCVMMLibrary\NC and select "Network Controller Production Generation 2 VM.xml". If you don't have 3+ compute nodes, you may select "Network Controller Standalone Generation 2 VM.xml" or "Network Controller Standalone Generation 1 VM.xml", which depends on your syspreped image is Gen 1 VM or Gen 2 VM.
6. On the Configure Reference page, map the WinServer.vhdx to the Syspreped image you copied to \\localhost\MSSCVMMLibrary\VHDs\. In addition, you may also need to map other resource to the corresponding folder under \\localhost\MSSCVMMLibrary\ if they were not recognize automatically.
7. Then click the "Import".
8. If your syspreped image is based on Volume License image, you may jump to step 12.
9. (Optional) If your syspreped image is not based on Volume License image, please right click the "Network Controller Production" and select "Open Designer".
10. Right click the "Windows Server Network Controller" and select "Properties".
11. In Windows Server Network Controller Properties window, click "OS Configuration" and key in a valid product key or 6XBNX-4JQGW-QX6QG-74P76-72V67. If you are not using a Volume Licensed VHD\VHDX or if the VHD\VHDX is not supplied the Product Key using an Answer file, then the deployment will stop at the Product Key page during network controller virtual machine(s) provisioning.
12. Right click the "Network Controller Production" and select "Configure Deployment".
13. Type the name of the service. In this case, the name is "NC-VMM-RTM". Select the logical network "MGMT". Then click OK.
14. Configure the settings for the service deployment. For local Admin, you may need to create a new Run As account if you don't have one for local admin. The username is ".\administrator".
15. In my case, I didn't create security group for NC Client and NC Admin. I just used Domain Users group and Domain Admins instead. As  for the diagnostic logging settings, they're optional. The important is the format of RestIPaddress is 192.168.20.34/24 instead of just IP address.  The format of MgmtDomainAccountName is "<domainname>\<username>" instead of just "<username">.
16. Then click the button "Refresh". Now you could click the button "Deploy Service".
17. In the prompt windows, click "Deploy".
18. Later on you might find the deployment complete successfully.
19. Click Fabric in the VMM console. Navigate to "Networking > Network Service". Right click "Network Service" and select "Add Network Service".
20. Click "Next" on the "Getting Started" page.
21. Give the network service a name, "NC" for example.
22. Select Manufacturer: Microsoft and Model: Microsoft Network Controller.
23. Still use "Run As" as the Run As account.
24. Connection String is serverurl=https://192.168.20.34;SouthBoundIPAddress=192.168.20.34;servicename=NC-VMM-RTM 
25.  On the next page, select the checkbox "These certificate have been reviewed and can be imported to the trusted certificate store." Then click "Next"
26. Click Scan Provider. Then click Next.
27. Select host group "All Hosts". Click "Next".
28. Click "Finish".
29. Please make sure the job "Add Network Service device" complete successfully.

Create Logical Network for "HNV"

1. Right click "Logical Network" and select "Create Logical Network".
2. Select the checkbox "All new VM networks created on this logical network to use network virtualization" and "Managed by Microsoft Network Controller".
3. On the next page, associate the VLAN (ID=201, IP Subnet=192.168.21.0/24) to the logical network site.
4. Click Finish.
5. Right click new created HNV logical network and select "Create IP Pool".
6. Name is "IPPool-HNV". Click "Next".
7. Accept the default Network Site, click "Next".
8. Configure starting IP address as 192.168.21.51 and ending IP address is 192.168.21.100.
9. Default gateway is 192.168.21.1.
10. DNS server is 192.168.20.3.
11. You don't need to configure WINS server. On the "Confirm the settings" page, click "Finish".
12. Navigate to "Networking > Port Profiles". Right click the new created uplink profile and select Properties.
13. Select the checkbox "HNV_0" to bind the network site "HNV_0" to the uplink profile, which means Hyper-V hosts will be able to access the logical network thru logical switch "MGMT-Switch".

Validation

Test Case 1: East-West Communication

1. Click "VM and Services" in the VMM Console. Then right click "VM Networks" and select "Create VM Network".
2. Give it a name "VNET-SLB".
3. Accept the default isolation setting and click "Next".
4. Add a VM Subnet  "SUB1" and set its subnet as "10.0.0.0/24". Then click "Next".
5. Don't change the setting on "Connectivity" page at this moment. Click "Next".
6. Click "Finish".
7. Right click "VNET-SLB" and select "Create IP Pool".
8. You may give it a name here (e.g., CATPool-VNET-SLB)
9. Accept the default IP address space, click "Next".
10. Set default gateway to 10.0.0.1.
11. Set DNS server to 192.168.20.3.
12. You don't need to configure WINS server. Click "Next". On the "Confirm the settings" page, click "Finish".
13. Click "Library" in the VMM Console and navigate to "Library Servers > INFRA-VMM > MSSCVMMLibrary > VHDs". Right click the Syspreped Image you copied before and select "Create VM Template".
14. Give the new VM template a name (E.g., "Windows Server 2016"). In my case, I used Gen 2 template. So I also select the generate 2. Click "Next".
15. On the Configure Hardware page, select Hyper-V as Cloud Capability.
16. Change the memory to 2048GB.
17. Connect the above new create VM network "VNET-SLB" and subnet "SUB1".
18. Configure the default password and product key.
19. Don't change any setting in the next two pages (Application Configuration and SQL Server Configuration). On the Summary page, click "Create".
20. Right click the new created VM template and select "Create Virtual Machine".
21. Give the VM a name "TESTVMSLB1".
22. Accept the default settings in all the rest pages in the wizard. On Summary page select checkbox "Start the virtual machine after deploying it". Then click "Create".
23. Follow the same procedure to create another VM "TESTVMSLB2" in a different Hyper-V host.
24. After the above two VM provisioned successfully, you may connect them and disable the Windows Firewall (If your Syspreped image didn't disable it before). Then try to ping each other and make sure east-west communication is ok.
25. You may run the cmdlet "Get-PACAMapping" on the host and make sure the PA address had already been assigned.

To Be Continued...