Practical Development

Creating a Key Encrypting Key (KEK)

Chris Clayton (AzureCAT) — Tue, 03 Jan 2017 02:33:39 GMT

Previously I have taken the steps to add secrets and keys to the Key Vault as well as create my self signed certificates using PowerShell. With the opportunities to use drive encryption in Azure on IaaS machines it makes sense to go through the steps of using a Key Encryption Key (KEK) to increase security but also to allow the encrypted machines to be backed up and restored. After doing some preparation for building encrypted IaaS virtual machines I found there was not a lot of “clear” guidance on building and using the KEK so I thought I would describe the steps I took here. For the purposes of my development efforts I used a self signed certificate although in production I would prefer to use a certificate issued from a trusted authority. I used the self signed certificate creation script outlined in my previous post with the legacy provider (Microsoft Enhanced Cryptographic Provider v1.0) for 2048 bit RSA.

 .\CreateSelfSignedCertificate.ps1 -subjectName 'My KEK Certificate' -fileName 'c:\certs\mykek.pfx' -useProviderV1 -password (ConvertTo-SecureString 'P@ssw0rd' -AsPlainText -Force)

For simplicity, I upload the certificate using the Azure Portal. After the key has been uploaded I use the portal to get the version number for use later. For creating and uploading the wrapped secret I will use the REST technique discussed in my previous post.

function Get-OAuth2Uri
(
  [string]$vaultName
)
{
  $response = try { Invoke-RestMethod -Method GET -Uri "https://$vaultName.vault.azure.net/keys" -Headers @{} } catch { $_.Exception.Response }
  $authHeader = $response.Headers['www-authenticate']
  $endpoint = [regex]::match($authHeader, 'authorization="(.*?)"').Groups[1].Value

  return "$endpoint/oauth2/token"
}

function Get-AccessToken
(
  [string]$vaultName,
  [string]$aadClientId,
  [string]$aadClientSecret
)
{
  $oath2Uri = Get-OAuth2Uri -vaultName $vaultName

  $body = 'grant_type=client_credentials'
  $body += '&client_id=' + $aadClientId
  $body += '&client_secret=' + [Uri]::EscapeDataString($aadClientSecret)
  $body += '&resource=' + [Uri]::EscapeDataString("https://vault.azure.net")

  $response = Invoke-RestMethod -Method POST -Uri $oath2Uri -Headers @{} -Body $body

  return $response.access_token
}

These methods are used to leverage the OAUTH2 endpoint with the applications client id and client secret. Rather than trying to build the URL for the KEK I retrieve the key metadata using the following function.

function Get-Key
(
  [string]$accessToken,
  [string]$vaultName,
  [string]$keyName,
  [string]$keyVersion
)
{
  $headers = @{ 'Authorization' = "Bearer $accessToken" }
  $queryUrl = "https://$vaultName.vault.azure.net/keys/$keyName/$keyVersion" + '?api-version=2016-10-01'
  $keyResponse = Invoke-RestMethod -Method GET -Uri $queryUrl -Headers $headers

  return $keyResponse.key
}

The drive will be encrypted using an AES 256 key created randomly using the following function.

function Create-AesKey
(
  [int]$keySize
)
{
  $aesProvider = New-Object System.Security.Cryptography.AesCryptoServiceProvider
  $aesProvider.KeySize = $keySize
  $aesProvider.GenerateKey()

  return $aesProvider.Key
}

The final step will be wrapping the AES 256 key with the KEK in the Key Vault. The KEK is never extracted, instead for security reasons I let the Key Vault do the work for me. To wrap the key the following method is used.

function Wrap-SymmetricKey
(
  [string]$accessToken,
  [string]$vaultName,
  [string]$keyName,
  [string]$keyVersion,
  [byte[]]$plainArray
)
{
  $base64Array = [Convert]::ToBase64String($plainArray)

  $queryUrl = "https://$vaultName.vault.azure.net/keys/$keyName/$keyVersion" + '/wrapkey?api-version=2016-10-01'   
  $headers = @{ 'Authorization' = "Bearer $accessToken"; "Content-Type" = "application/json" }

  $bodyObject = @{ "alg" = "RSA-OAEP"; "value" = $base64Array }
  $bodyJson = ConvertTo-Json -InputObject $bodyObject

  $response = Invoke-RestMethod -Method POST -Ur $queryUrl -Headers $headers -Body $bodyJson

  return $response.value
}

It is as simple as piecing these methods together as follows.

$accessToken = Get-AccessToken -vaultName $vaultName -aadClientId $aadClientID -aadClientSecret $aadClientSecret

$aesKey = Create-AesKey -keySize 256
$wrappedAesKey = Wrap-SymmetricKey -accessToken $accessToken -vaultName $vaultName -keyName $kekName -keyVersion $kekVersion -plainArray $aesKey

$keyEncryptingKey = Get-Key -accessToken $accessToken -vaultName $vaultName -keyName $kekName -keyVersion $kekVersion
$kekUri = $keyEncryptingKey.kid

$queryUrl = "https://$vaultName.vault.azure.net/secrets/$secretName" + '?api-version=2016-10-01'  
$secretAttributes = @{ 'enabled' = $true }
$secretTags = @{ 'DiskEncryptionKeyEncryptionAlgorithm' = 'RSA-OAEP'; 'DiskEncryptionKeyFileName' = "$secretName.BEK"; 'DiskEncryptionKeyEncryptionKeyURL' = $kekUri }
$headers = @{ 'Authorization' = "Bearer $accessToken"; "Content-Type" = "application/json" }

$bodyObject = @{ 'value' = $wrappedAesKey; 'attributes' = $secretAttributes; 'tags' = $secretTags; 'contentType' = 'Wrapped BEK' }
$body = ConvertTo-Json -InputObject $bodyObject

$secretResponse = Invoke-RestMethod -Method PUT -Uri $queryUrl -Headers $headers -Body $body

Write-Host 'secret URL is ' + $secretResponse.id

The above code generates the AES key, wraps it using the KEK and inserts the secret with the required tags. After running this script I was able to successfully encrypt my Azure Virtual Machines.

Using Key Vault Secrets in PowerShell

Chris Clayton (AzureCAT) — Mon, 02 Jan 2017 16:13:03 GMT

Interacting with Key Vault through the standard cmdlets is very simple and straight forward, but what happens when I want to use the Key Vault functions that are not exposed in this way such as encrypting or signing a value using the key stored in the key vault? I was experimenting with some ideas to try and keep separation of knowledge when it comes to operation vs. development and found the excessively heavy dependence on .NET knowledge I see in most explanations seems a bit overkill for non-developers. Trying to find a better way I dissected some code samples for Linux disk encryption and came up with the following technique. First was figuring out the most appropriate way to authenticate. In most of the examples I see on the internet people are using the ClientCredential class from .NET, but this comes with a bit of a cost. The cost I am talking about is the importing of the types which can be done, or a requirement to at least run the Azure login powershell cmdlet. Obviously, this does not align with my goal of reducing the .NET knowledge. The technique I decided on was to first perform a failed access to my Key Vault which will reply with an endpoint to authenticate against.

function Get-OAuth2Uri
(
  [string]$vaultName
)
{
  $response = try { Invoke-RestMethod -Method GET -Uri "https://$vaultName.vault.azure.net/keys" -Headers @{} } catch { $_.Exception.Response }
  $authHeader = $response.Headers['www-authenticate']
  $endpoint = [regex]::match($authHeader, 'authorization="(.*?)"').Groups[1].Value

  return "$endpoint/oauth2/token"
}

This finds the endpoint for my key vault and appends the oauth2 specific portion of the URL providing me with a location to authenticate against which can change and my scripts should continue to function. The one requirement that this technique has for the next step is to already have registered an application identity with Key Vault as I have done in my setting up Key Vault description previously. Assuming I have the Azure Active Directory Client Id and Client Secret I can continue without issue.

function Get-AccessToken
(
  [string]$vaultName,
  [string]$aadClientId,
  [string]$aadClientSecret
)
{
  $oath2Uri = Get-OAuth2Uri -vaultName $vaultName

  $body = 'grant_type=client_credentials'
  $body += '&client_id=' + $aadClientId
  $body += '&client_secret=' + [Uri]::EscapeDataString($aadClientSecret)
  $body += '&resource=' + [Uri]::EscapeDataString("https://vault.azure.net")

  $response = Invoke-RestMethod -Method POST -Uri $oath2Uri -Headers @{} -Body $body

  return $response.access_token
}

With the above function, I get the access token for the Key Vault based on my Client Id’s permissions. Now the challenging portions are done. Yes, that is the hardest part. To make use of this for the simple REST APIs exposed via the GET verb I can use the following technique.

function Get-Keys
(
  [string]$accessToken,
  [string]$vaultName
)
{
  $headers = @{ 'Authorization' = "Bearer $accessToken" }
  $queryUrl = "https://$vaultName.vault.azure.net/keys" + '?api-version=2016-10-01'

  $keyResponse = Invoke-RestMethod -Method GET -Uri $queryUrl -Headers $headers

  return $keyResponse.value
}

In this function, I retrieve a list of all the keys in my Key Vault. The important parts to pay attention to here are

The Authorization header is set in the hash table as a bearer token and the access token retrieved from my Get-AccessToken function
The Invoke-RestMethod cmdlet does all of the heavy listing

Now to handle the important POST methods for encrypting and decrypting data using a key that is in my Key Vault letting it do what it is meant to do, keep my keys safe!

function Encrypt-ByteArray
(
  [string]$accessToken,
  [string]$vaultName,
  [string]$keyName,
  [string]$keyVersion,
  [byte[]]$plainArray
)
{
  $base64Array = [Convert]::ToBase64String($plainArray)
  $queryUrl = "https://$vaultName.vault.azure.net/keys/$keyName/$keyVersion" + '/encrypt?api-version=2016-10-01'   
  $headers = @{ 'Authorization' = "Bearer $accessToken"; "Content-Type" = "application/json" }

  $bodyObject = @{ "alg" = "RSA-OAEP"; "value" = $base64Array }
  $bodyJson = ConvertTo-Json -InputObject $bodyObject

  $response = Invoke-RestMethod -Method POST -Ur $queryUrl -Headers $headers -Body $bodyJson

  return $response.value
}

When performing the post methods there is a common theme of converting to Base64 strings which just makes sense so we do not introduce any invalid characters etc. and creating a JSON body as you can see here. For the most part the Post Methods will have the same body structure but there are a few that vary slightly such as Verifying data signatures. Now all I must do is decrypt the data as you can see here.

function Decrypt-ByteArray
(
  [string]$accessToken,
  [string]$vaultName,
  [string]$keyName,
  [string]$keyVersion,
  [string]$cipher
)
{
  $queryUrl = "https://$vaultName.vault.azure.net/keys/$keyName/$keyVersion" + '/decrypt?api-version=2016-10-01'       
  $headers = @{ 'Authorization' = "Bearer $accessToken"; "Content-Type" = "application/json" }

  $bodyObject = @{ "alg" = "RSA-OAEP"; "value" = $cipher }
  $bodyJson = ConvertTo-Json -InputObject $bodyObject

  $response = Invoke-RestMethod -Method POST -Ur $queryUrl -Headers $headers -Body $bodyJson
  $base64Array = $response.value

  # This next section fixes missing characters on the base64
  $missingCharacters = $base64Array.Length % 4

  if($missingCharacters -gt 0)
  {
    $missingString = New-Object System.String -ArgumentList @( '=', $missingCharacters )
    $base64Array = $base64Array + $missingString       
  }

  return [Convert]::FromBase64String($base64Array)
}

You can see that this is almost the same as the Encrypt-ByteArray with a change the query string and one piece of code identified as missing characters. I will say it bothers me that this is needed but as much as I looked I saw tons of people hitting this in general encrypting and decrypting of base 64 strings without any great answers. It turns out that for some reason the process of encrypting and decrypting removes the ‘=’ characters that represent padding on a base 64 string to the length that is divisible by 4 (all base 64 encoded strings must have a length that is divisible by 4 to be valid). So as a workaround I figure out the number of missing characters and pad it myself with the ‘=’ character. After several tests this technique seems to hold up. I can now use secrets from my Key Vault in my PowerShell scripts etc. for operations without a great knowledge of .NET. To really simplify things, I will wrap all the REST API in this. Obviously, the downside to this is that I need to register an application for use by my infrastructure team.

Self Signed Certificate Creation

Chris Clayton (AzureCAT) — Fri, 30 Dec 2016 22:11:00 GMT

The number of times in the past that I have had to create self signed certificates is far too many to count! There have been various tools to help with it include IIS Server Management and the old standby MAKECERT utility. When I needed to create a couple new certificates for my Azure Drive Encryption I decided to see if there was a new and better way to do this. Low and behold PowerShell has a great cmdlet for this, but after reading the provider type issues some people were having I was gun-shy. After some investigation, I seem to have overcome the issue others were complaining about and have decided to post my PowerShell script I will use to generate self signed certificates moving forward. The first thing I needed to do was to setup my standard parameters that I will want to offer. There are a couple things that are worth calling out here.

Param
(
  [Parameter(Mandatory=$true, Position=1, HelpMessage="Certificates subject (e.g. CN=mysubject)")]
  [string]$subjectName,
  [Parameter(Mandatory=$true, Position=2, HelpMessage="PFX output file name (e.g. c:\certs\mycert.pfx)")]
  [string]$fileName,
  [Parameter(Mandatory=$true, Position=3, HelpMessage="Password for the PFX File")]
  [Security.SecureString]$password,
  [Parameter(Mandatory=$true, Position=4, HelpMessage="Expiry date of the certificate")]
  [DateTime]$expiryDate,
  [Parameter(Mandatory=$false, Position=5, HelpMessage="Friendly name of the certificate")]
  [string]$friendlyName,
  [Parameter(Mandatory=$false, Position=6, HelpMessage="Certificates description")]
  [string]$description,
  [Parameter(Mandatory=$true, Position=7, HelpMessage="Cert store location (e.g. Cert:\CurrentUser\My)")]
  [string]$certificateLocation,
  [Parameter(Mandatory=$false, HelpMessage="Indicates the V1 cryptographic provider should be used")]
  [switch]$useProviderV1
)

The standard certificate information should be more than self explanatory, but I have added a provider switch to get around some of the issues I have heard about (and seen). It seems that the issue with the provider type seems to be tied to the use of the CNG libraries so I will provide the ability to revert to “Microsoft Enhanced Cryptographic Provider v1.0” when desired through the inclusion of the -useProviderV1 switch. The next thing I do is to make sure that I throw an error if any of the required parameters are null or empty strings.

if($password -eq $null -or $password.Length -lt 1)
{
  Write-Error 'The password provided was null or empty'
  throw [System.ArgumentException] 'The password must not be null or empty'
}
else
{
  Write-Verbose 'The password valid'
}

if($subjectName -eq $null -or $subjectName.Length -lt 1)
{
  Write-Error 'The subject name provided was null or empty'
  throw [System.ArgumentException] 'The subject name must not be null or empty'
}
else
{
  Write-Verbose 'The subject name provided was valid'
}

if($subjectName -eq $null -or $subjectName.Length -lt 1)
{
  Write-Error 'The subject name provided was null or empty'
  throw [System.ArgumentException] 'The subject name must not be null or empty'
}
else
{
  Write-Verbose 'The subject name provided was valid'
}

Following this I will validate the output directory where the PFX file is to be placed is valid as well as ensuring the filename ends with the PFX extension.

if($fileName -eq $null -or $fileName.Length -lt 1)
{
  $fileName = $null
  Write-Error 'The file name provided was null or empty'
  throw [System.ArgumentException] 'The file name must not be null or empty'
}
else
{
  if ($fileName.EndsWith('.pfx') -eq $false)
  { 
    $fileName = "$($fileName).pfx"
  }

  $directoryName = [System.IO.Path]::GetDirectoryName($fileName) 
 
  if(($directoryName -eq $null -or $directoryName.Length -lt 1) -or ((Test-Path -Path $directoryName) -eq $true))
  {
    Write-Verbose 'The file name provided was valid'
  }
  else
  {
    $fileName = $null
    Write-Error 'The directory does not exist'
    throw [System.ArgumentException] 'The directory does not exist'
  } 
}

The final preparation work to do is to make sure the expiry date is in the future.

if($expiryDate -le (Get-Date))
{
  Write-Error 'Expiry date provided was not in the future.'
  throw [System.ArgumentException] 'Expiry date provided was not in the future.'
}
else
{
  Write-Verbose 'The date provided was valid.'
}

I can now create the certificates and add them to the current user’s certificate store. If the expiration date is not provided I will not set the expiration date on the certificate. If the legacy provider is required I also set that.

if($useProviderV1 -eq $true)
{
  if($expiryDate -ne $null)
  {
    $cert = New-SelfSignedCertificate -Type SSLServerAuthentication -Subject "CN=$($subjectName)" -KeyAlgorithm RSA -KeyLength 2048 -NotAfter $expiryDate -CertStoreLocation Cert:\CurrentUser\My -FriendlyName $friendlyName -KeyDescription $description -KeyUsageProperty All -Provider 'Microsoft Enhanced Cryptographic Provider v1.0'
  }
  else
  {
    $cert = New-SelfSignedCertificate -Type SSLServerAuthentication -Subject "CN=$($subjectName)" -KeyAlgorithm RSA -KeyLength 2048 -CertStoreLocation Cert:\CurrentUser\My -FriendlyName $friendlyName -KeyDescription $description -KeyUsageProperty All -Provider 'Microsoft Enhanced Cryptographic Provider v1.0' 
  }
}
else
{
  if($expiryDate -ne $null)
  {
    $cert = New-SelfSignedCertificate -Type SSLServerAuthentication -Subject "CN=$($subjectName)" -KeyAlgorithm RSA -KeyLength 2048 -NotAfter $expiryDate -CertStoreLocation Cert:\CurrentUser\My -FriendlyName $friendlyName -KeyDescription $description -KeyUsageProperty All 
  }
  else
  {
    $cert = New-SelfSignedCertificate -Type SSLServerAuthentication -Subject "CN=$($subjectName)" -KeyAlgorithm RSA -KeyLength 2048 -CertStoreLocation Cert:\CurrentUser\My -FriendlyName $friendlyName -KeyDescription $description -KeyUsageProperty All 
  }
}

If I stop here the certificate will be added to my certificate store, but I must manually export to use it for my purposes later. Instead of sticking with that I will export the certificate and delete the old one.

Export-PfxCertificate -FilePath $fileName -Cert $cert -Password $password
Remove-Item -path "$($certificateLocation)\$($cert.Thumbprint)"

After this process I no longer have to resort to the older techniques of certificate creation!

Ronin – Key Vault

Chris Clayton (AzureCAT) — Sat, 10 Dec 2016 13:37:36 GMT

The first thing I am going to do is setup a standard way to work with and store the solutions primary secrets and cryptographic materials. I find that as a developer I am called on to decide or at least make recommendations on how to handle implementations of protocols, tools and technologies that need access to secrets and or cryptographic materials. I obviously see merit in this if the proper security review is in place to protect everyone from myself! Once the system goes into production should the operations team really trust me with these sensitive materials? Enter Azure Key Vault (Key Vault). Key Vault is an Azure based PaaS service that can be used to encrypt keys and secrets (e.g. connection strings, X509 certificates etc.) through the use of a Hardware Security Module (HSM). HSMs are used quite frequently in data centers so I would not expect any less in our solution. The Azure Key Vault Service can store the keys in a FIPS 140-2 Level 2 validated HSM, adding the level of security I am looking for at the same time as enabling a separation concerns for the developer and operations. The separation of concerns comes from providing the ability to upload an manage the secrets and keys to Operations while granting my application access in to use them. The solution makes requests of the Key Vault to retrieve secrets or decrypt values using named keys based on the permissions setup by the Operations team. The named keys cannot be retrieved by the solution making it a great defense technique, but as with all public cloud services it has consumption throttles in place. These consumption throttles are based on 10 second windows making this something I want to use in a well thought out fashion (e.g. use it when required and appropriate vs. trying to use it for every sing transaction in the system). I am going to use my organizations Azure Active Directory to support my solution's interactions with the Key Vault so I will not have to setup one of my own. The Key Vault will sit outside of my Virtual Network and have its own life cycle. In fact, once I setup the Key Vault I would probably leverage the same one for other solutions in the same region, thus allowing my operations team to reduce the number of interactions they need to perform in the case of general key rollovers etc. To setup the Key Vault I use a simple ARM template again. The first part of the template is to define the parameters of interest, the goal being to make as few parameters as possible while making the template reusable.

"parameters":
{
  "keyVaultName":
  {
    "type": "string",
    "defaultValue": "[concat(uniqueString(subscription().id), 'vault')]",
    "metadata": { "description": "Name of the vault" }
  },
  "adUserObjectId":
  {
    "type": "string",
    "metadata": { "description": "Object Id of the AD user. Get using Get-AzureRmADUser or Get-AzureRmADServicePrincipal cmdlets" }
  },
  "skuName":
  {
    "type": "string",
    "defaultValue": "Premium",
    "allowedValues": [ "Standard", "Premium" ],
    "metadata": { "description": "SKU for the vault" }
  }
}

In the template above you can see that I will support 3 parameters

keyVaultName: the name of the Key Vault to be created.
adUserObjectId: this is the id of the user that will be the administrator of the Key Vault. This id can be obtained by running the GetAzureRmADUser or Get-AzureRmADServicePrincipal powershell cmdlets.
skuName: this is the service offering within Key Vault that I want to use. My immediate reaction is usually to take the smaller SKU (standard) and just change to the appropriate one based on my needs depending on what environment I am in. For Key Vault, I prefer looking at the Premium SKU seeing it offers true HSM, not just software based

The variables section will be used to hold values that must be consistent, mitigating against human error, or adding values that I may want to surface at a later time to a parameter for customizations.

"variables":
{
  "tenantId": "[subscription().tenantId]",
  "enableForDeployment": true,
  "enableForDiskEncryption": true,
  "enableForTemplateDeployment": true,
  "secretsPermissions": "all",
  "keysPermissions": "all"
},

It is worth noting that in the variables above the tenant Id uses the subscriptions tenant id for deploying the Key Vault. I will deploy the Key Vault without any secrets or keys but will associate the user identified by the adUserObjectId to it in the role of administrator.

"resources":
[
  {
    "type": "Microsoft.KeyVault/vaults",
    "name": "[parameters('keyVaultName')]",
    "apiVersion": "2015-06-01",
    "location": "[resourceGroup().location]",
    "properties":
    {
      "enabledForDeployment": "[variables('enableForDeployment')]",
      "enabledForDiskEncryption": "[variables('enableForDiskEncryption')]",
      "enabledForTemplateDeployment": "[variables('enableForTemplateDeployment')]",
      "tenantId": "[variables('tenantId')]",
      "accessPolicies":
      [
        {
          "tenantId": "[variables('tenantId')]",
          "objectId": "[parameters('adUserObjectId')]",
          "permissions":
          {
            "keys": ["[variables('keysPermissions')]"],
            "secrets": ["[variables('secretsPermissions')]"]
          }
        }
      ],
      "sku":
      {
        "name": "[parameters('skuName')]",
        "family": "A"
      }
    }
  }
]

I decided not to deploy any secrets or keys so I can force the separation of concerns. In this scenario although the standard operations team can deploy the Key Vault for use in the solution their needs to be a security officer or other trusted authority with the security materials to setup the keys and secrets. The final piece is to output the newly created Key Vault object, allowing me to use this from other scripts or get the important details in my PowerShell session without much work.

"outputs":
{
  "vault":
  {
    "type": "object",
    "value": "[reference(parameters('keyVaultName'))]"
  }
}

Building a parameter file is easy as you can see here.

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters":
  {
    "keyVaultName": { "value": "RoninVault" },
    "adUserObjectId": { "value": "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" },
    "skuName": { "value": "Premium" }
  }
}

To deploy the Key Vault using the ARM template and parameter file I run a PowerShell script as seen here.

<#
Select the proper account and subscription
#>
Login-AzureRmAccount
Set-AzureRmContext -SubscriptionId ‘xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx’

<#
Checks to see if the resource group exists
#>
function Test-ResourceGroupExists
{
  Param
  (
    [string]$name
  )

  Get-AzureRmResourceGroup -Name $name -ErrorAction SilentlyContinue

  return $?
}

<#
Creates the resource group if it does not exist
#>
function Create-ResourceGroupIfNotExists
{
  Param
  (
    [string]$name,
    [string]$location
  )

  $success = $false

  if(!(Test-ResourceGroupExists -name $name))
  {
    Write-Host 'Resource group does not exist, attempting to create it.'
    New-AzureRmResourceGroup -Name $name -Location $location -ErrorAction SilentlyContinue

    $success = $?
  }
  else
  {
    $success = $true
  }

  return $success
}

<#
Define common variables
#>
$resourceGroupName = 'RoninSecrets'
$location = 'East US'
$templateFile = './azuredeploy.keyvault.json'
$templateParameterFile = './azuredeploy.keyvault.parameters.json'

if(Create-ResourceGroupIfNotExists -name $resourceGroupName -location $location)
{
  New-AzureRmResourceGroupDeployment -ResourceGroupName $resourceGroupName -TemplateFile $templateFile -TemplateParameterFile $templateParameterFile

  if($?)
  {
    Write-Host 'script deployment successful'
  }
  else
  {
    Write-Host 'script deployment failed'
  }
}
else
{
  Write-Host 'Resource Group does not exist and could not be created... aborting script'
}

Once I run the script I see that the Key Vault has deployed successfully. With the Key Vault created I could continue to add additional principals with varying rights, but for the time being I will keep it with myself possessing full permissions. For production reasons, I would do a security review and determine the appropriate levels of access by user to keep with a least privilege mindset. To add a secret, I am going to use a simple script that prompts for the secret name and value. Based on the input it will submit the secret to the store that can later be retrieved by name.

$subscriptionId = ‘xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx’

<#
Select the proper account and subscription
#>
Login-AzureRmAccount
Set-AzureRmContext -SubscriptionId $subscriptionId

$vaultName = 'RoninVault'
$secretName = Read-Host -Prompt "Enter Secret Name"
$secretValue = Read-Host -Prompt "Enter password" -AsSecureString

Set-AzureKeyVaultSecret -VaultName $vaultName -Name $secretName -SecretValue $secretValue

Once this runs I will get the following output outlining the details of the secret entry. This works great for strings and generally simple data, but to store a certificate as a secret I need to handle it a bit different.

$subscriptionId = ‘xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx’

<#
Select the proper account and subscription
#>
Login-AzureRmAccount
Set-AzureRmContext -SubscriptionId $subscriptionId

function Get-PfxLocation
{
  [string]$resultingFilePath = $null
  [Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms") | Out-Null
  [System.Windows.Forms.Application]::EnableVisualStyles()
  
  $browse = New-Object System.Windows.Forms.OpenFileDialog
  $browse.filter = "Certificate (*.pfx)| *.pfx"

  if ($browse.ShowDialog() -eq "OK")
  {
    $resultingFilePath = $browse.filename
  }

  $browse.Dispose()

  return $resultingFilePath
}

$vaultName = 'RoninVault'
$secretName = Read-Host -Prompt "Enter Secret Name"
$pfxFile = Get-PfxLocation
$securePassword = Read-Host -Prompt "Enter PFX Password" -AsSecureString
$password = [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($securePassword))

$collection = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2Collection
$collection.Import($pfxFile, $password, [System.Security.Cryptography.X509Certificates.X509KeyStorageFlags]::Exportable)
$clearBytes = $collection.Export([System.Security.Cryptography.X509Certificates.X509ContentType]::Pkcs12)
$fileContentEncoded = [System.Convert]::ToBase64String($clearBytes)
$secretValue = ConvertTo-SecureString -String $fileContentEncoded -AsPlainText –Force

Set-AzureKeyVaultSecret -VaultName $vaultName -Name $secretName -SecretValue $secretValue -ContentType 'application/x-pkcs12'

This technique will use the .NET file browser to retrieve a PFX file and store the Base64String representation of it. The following output shows the metadata associated with the certificate stored as a secret in my Key Vault. Finally, to add a key to the hardware HSM I can use the following script.

$subscriptionId = ‘xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx’

<#
Select the proper account and subscription
#>
Login-AzureRmAccount
Set-AzureRmContext -SubscriptionId $subscriptionId

function Get-PfxLocation
{
  [string]$resultingFilePath = $null
  [Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms") | Out-Null
  [System.Windows.Forms.Application]::EnableVisualStyles()

  $browse = New-Object System.Windows.Forms.OpenFileDialog
  $browse.filter = "Certificate (*.pfx)| *.pfx"

  if ($browse.ShowDialog() -eq "OK")
  {
    $resultingFilePath = $browse.filename
  }

  $browse.Dispose()

  return $resultingFilePath
}

$vaultName = 'RoninVault'
$keyName = Read-Host -Prompt "Enter Key Name"
$pfxFile = Get-PfxLocation
$securePassword = Read-Host -Prompt "Enter PFX Password" -AsSecureString
$password = [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($securePassword))

Add-AzureKeyVaultKey -VaultName $vaultName -Name $keyName -KeyFilePath $pfxFile -KeyFilePassword $securePassword -Destination HSM

I see a bit different output because I am working with a key instead of secret. Now that I have the Key Vault deployed and have added both keys and secrets. The only think left is to setup the application to access the key vault and begin using it in my solution. The following script will create an Active Directory application and an associated Service Principal. The Service Principal is used to both authenticate the application and associated permissions in Key Vault. It is important in an production system to have a security review performed to determine the appropriate permissions to grant.

$subscriptionId = ‘xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx’

<#
Select the proper account and subscription
#>
Login-AzureRmAccount
Set-AzureRmContext -SubscriptionId $subscriptionId

$startDate = [System.DateTime]::Now
$endDate = $startDate.AddYears(2)
$applicationPassword = [Guid]::NewGuid()

$vaultName = Read-Host -Prompt "Enter Key Vault Name:"
$displayName = Read-Host -Prompt "Enter Application Name:"
$identifierUri = [string]::Format("https://localhost:8080/{0}{1}", $displayName,[Guid]::NewGuid().ToString("N"))
$homePage = "https://localhost/$($displayName)"

$activeDirectoryApplication = New-AzureRmADApplication -DisplayName $vaultName -HomePage $homePage -IdentifierUris $identifierUri  -StartDate $startDate -EndDate $endDate -Password $applicationPassword
$servicePrincipal = New-AzureRmADServicePrincipal -ApplicationId $activeDirectoryApplication.ApplicationId
Set-AzureRmKeyVaultAccessPolicy -VaultName $vaultName -ObjectId $servicePrincipal.Id -PermissionsToKeys all -PermissionsToSecrets all -PermissionsToCertificates all

Write-Host "Client Id: $($servicePrincipal.ApplicationId)"
Write-Host "Client Secret: $($applicationPassword)"

Now that the environment is configured I can validate my access to the Key Vault from a .NET console application. I create a windows console application and add the Microsoft.Azure.KeyVault NUGET package to the project, providing an easy to use interop layer for Key Vault. Using the Client Id and Client Key output when I registered the application in Azure Active Directory I can authenticate as follows.

private const string ClientId = @"XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX";
private const string ApplicationKey = @"XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX";

public static async Task GetToken(string authority, string resource, string scope)
{
  var credential = new ClientCredential(ClientId, ApplicationKey);
  var authenticationContext = new AuthenticationContext(authority, null);
  
  return (await authenticationContext.AcquireTokenAsync(resource, credential)).AccessToken;
}

This retrieves the access token when the application first attempts to perform an operation on the Key Vault using the client. To create the client and receive the secret I inserted called StorageConnectionString I use the URL that was output when it was inserted or I can obtain it through the portal.

private const string SecretId = "https://roninvault.vault.azure.net/secrets/StorageConnectionString/0394b3ee1e0e46f6934e71b52898417f";

var vaultClient = new KeyVaultClient(GetToken);
var secret1 = vaultClient.GetSecretAsync(SecretId).Result;
var value = secret1.Value;

This returns the desired value from the secret store, validating it's functionality and completing this piece of the solution.

It is important to remember that if you are accessing the Key Vault from a locked down virtual network it is done using HTTPS port 443 for most commands but additionally HTTP port 80 is used to retrieve the certificate revocation list at times.

<< Previous

Ronin Building Blocks - Network Template

Chris Clayton (AzureCAT) — Mon, 28 Nov 2016 13:37:33 GMT

I am a strong believer in building security into the solution from the ground up in general but when it comes to IoT it is pretty much mandatory. There seems to be a consistent stream of exploits and attacks on connected devices so to properly defend my solution against this I will start any deployment with a more traditional networking take. This is only a single layer of the security that will be introduced to the solution throughout the project. A very key point to remember here is that no security solution is impenetrable and the security I apply here is my opinion of the appropriate technique. If you are to follow my lead on your project in taking this approach you will want to run security reviews and penetration testing with a security specialist and or firm. Seeing this solution is not being built for production I will not be taking this cost on myself. When designing my network, I have several different zones that each have their own functions and trust levels making them ideal candidates for subnets.

DMZ: as in any traditional solution a DMZ is a great demarcation point between the front end “low trust” zone. I will host the front-end services that are accessible to the internet or untrusted clients. If required by the solution the DMZ will have the ability to connect with the Application Services zone over controlled network ports and protocols.
Application Service: This will host most of the logic associated with the application and is where most the “real work” happens. This zone is a higher trust with access to data stores and if required having the ability to reach out to the DMZ.
Data: This will host any data stores that can take part in a network, including IaaS hosted databases. This zone should have as little business logic in it as possible. The initial plans it to lock this zone down so it cannot reach out to the application services or other zones requiring them always to establish the connection.
Monitoring: is a zone used to centralize the monitoring logic for the overall solution. Much of the monitoring and instrumentation will occur on the specific instances but at regular intervals this zone will receive the most current information. The reason for this zone is to reduce the chance that a compromised Data, Application Service or DMZ zone does not allow the bad actor to destroy the logs that will provide evidence of the intrusion.
Support: this zone is used to house support tools and the jump boxes, ideally only being accessible by VPN connections from the corporate network of the solution operator.

I am going to build the ARM (Azure Resource Manager) template to get the core elements of this structure in place and adjust them later as some of the decisions are finalized. If you have not used an ARM template before it is a mandatory skill for Azure Development. The entry point template for my deployment is called azuredeploy.json (“the template”) with a parameters file of azuredeploy.parameters.json (“the parameter file”). The first thing I do is create the template with parameters. The only parameters I will ask for are the network name and CIDR addresses that correspond to the network and each subnet contained in it.

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters":
  {
    "networkName":
    {
      "type": "string",
      "defaultValue": "IoTNetwork",
      "minLength": 3,
      "maxLength": 63,
      "metadata":
      {
        "description": "The name of the virtual network"
      }
    },
    "networkCIDR":
    {
      "type": "string",
      "defaultValue": "192.168.0.0/16",
      "minLength": 9,
      "maxLength": 19,
      "metadata":
      {
        "description": "The CIDR address of the network e.g. 192.168.0.0/16"
      }
    },
    "dmzCIDR":
    {
      "type": "string",
      "defaultValue": "192.168.1.0/24",
      "minLength": 9,
      "maxLength": 19,
      "metadata":
      {
        "description": "The CIDR address of the DMZ subnet e.g. 192.168.1.0/24"
      }
    },
    "appServicesCIDR":
    {
      "type": "string",
      "defaultValue": "192.168.2.0/24",
      "minLength": 9,
      "maxLength": 19,
      "metadata":
      {
        "description": "The CIDR address of the application services subnet e.g. 192.168.2.0/24"
      }
    },
    "dataCIDR":
    {
      "type": "string",
      "defaultValue": "192.168.3.0/24",
      "minLength": 9,
      "maxLength": 19,
      "metadata":
      {
        "description": "The CIDR address of the DMZ subnet e.g. 192.168.3.0/24"
      }
    },
    "monitoringCIDR":
    {
      "type": "string",
      "defaultValue": "192.168.4.0/24",
      "minLength": 9, 
      "maxLength": 19,
      "metadata":
      {
        "description": "The CIDR address of the monitoring subnet e.g. 192.168.4.0/24"
      }
    },
    "supportCIDR":
    {
      "type": "string",
      "defaultValue": "192.168.5.0/24",
      "minLength": 9,
      "maxLength": 19,
      "metadata":
      {
        "description": "The CIDR address of the support subnet e.g. 192.168.4.0/24"
      }
    }
  },
  "variables": { },
  "resources": [ ],
  "outputs": {}
}

Notice in the template above I am trying to keep the required parameters to a minimum while defining default values and descriptions for each parameter. I did this to follow the guidance defined as best practices for creating ARM templates. The next piece I will do is add each of these parameter values into the parameter file.

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters":
  {
    "networkName": { "value": "RoninNetTemplate" },
    "networkCIDR": { "value": "192.168.0.0/16" },
    "dmzCIDR": { "value": "192.168.1.0/24" },
    "appServicesCIDR": { "value": "192.168.2.0/24" },
    "dataCIDR": { "value": "192.168.3.0/24" },
    "monitoringCIDR": { "value": "192.168.4.0/24" },
    "supportCIDR": { "value": "192.168.5.0/24" }
  }
}

Now I will run the script to test it out and make sure there are no errors in it. I will use the following PowerShell command each time I want to deploy the template.

<#
Select the proper account and subscription
#>
Login-AzureRmAccount
Set-AzureRmContext -SubscriptionId {Place your own subscription Id Here}
<#
Checks to see if the resource group exists
#>
function Test-ResourceGroupExists
{
  Param
  (
    [string]$name
  )
  Get-AzureRmResourceGroup -Name $name -ErrorAction SilentlyContinue

  return $?
}

<#
Creates the resource group if it does not exist
#>
function Create-ResourceGroupIfNotExists
{
  Param
  (
    [string]$name,
    [string]$location
  )
  $success = $false
  if(!(Test-ResourceGroupExists -name $name))
  {
    Write-Host 'Resource group does not exist, attempting to create it.'
    New-AzureRmResourceGroup -Name $name -Location $location -ErrorAction SilentlyContinue
    $success = $?
  }
  else
  {
    $success = $true
  }

  return $success
}

<#
Define common variables
#>
$resourceGroupName = 'RoninNetRG'
$location = 'East US'
$templateFile = 'C:\Templates\azuredeploy.json'
$templateParameterFile = 'C:\Templates\azuredeploy.parameters.json'

if(Create-ResourceGroupIfNotExists -name $resourceGroupName -location $location)
{
  New-AzureRmResourceGroupDeployment -ResourceGroupName $resourceGroupName -TemplateFile $templateFile -TemplateParameterFile $templateParameterFile

  if($?)
  {
    Write-Host 'script deployment successful'
  }
  else
  {
    Write-Host 'script deployment failed'
  }
}
else
{
  Write-Host 'Resource Group does not exist and could not be created... aborting script'
}

This PS1 file can now be used to deploy my VNET each time once I finish the template file. I now populate the template resources subnet structure to reflect the network I want by adding the following information to the variables and resources section of the template file.

"variables":
{
  "DmzSubnetName": "Dmz",
  "DmzSubnetResourceId": "[concat(variables('DmzSubnetName'), '_sg')]",
  "AppServicesSubnetName": "AppServices",
  "AppServicesSubnetResourceId": "[concat(variables('AppServicesSubnetName'), '_sg')]",
  "DataSubnetName": "Data",
  "DataSubnetResourceId": "[concat(variables('DataSubnetName'), '_sg')]",
  "MonitoringSubnetName": "Monitoring",
  "MonitoringSubnetResourceId": "[concat(variables('MonitoringSubnetName'), '_sg')]",
  "SupportSubnetName": "Support",
  "SupportSubnetResourceId": "[concat(variables('SupportSubnetName'), '_sg')]"
},
"resources":
[
  {
    "apiVersion": "2016-03-30",
    "type": "Microsoft.Network/virtualNetworks",
    "name": "[parameters('networkName')]",
    "location": "[resourceGroup().location]",
    "comments": "Virtual network based on the Ronin template.",
    "properties": {
      "addressSpace": { "addressPrefixes": [ "[parameters('networkCIDR')]" ] },
        "subnets":
        [
          { "name": "[variables('DmzSubnetName')]", "properties": { "addressPrefix": "[parameters('dmzCIDR')]" } },
          { "name": "[variables('AppServicesSubnetName')]", "properties": { "addressPrefix": "[parameters('appServicesCIDR')]" } },
          { "name": "[variables('DataSubnetName')]", "properties": { "addressPrefix": "[parameters('dataCIDR')]" } },
          { "name": "[variables('MonitoringSubnetName')]", "properties": { "addressPrefix": "[parameters('monitoringCIDR')]" } },
          { "name": "[variables('SupportSubnetName')]", "properties": { "addressPrefix": "[parameters('supportCIDR')]" } }
       ]
     }
  }
]

Running this script now I can see the following network created in the Azure Portal. This looks exactly how I want it but there currently are no network security groups defined to control the access to and from the various network. In this next section I will put those controls in place. The first thing I do is define the security groups that will be used to control access to the various subnets. In the section below the basic rules are defined. As I move forward with this template in the specific implementations I will just need to add or remove additional rules. This section is in the resource directly above the network definition from the previous step.

  {
    "apiVersion": "2016-03-30",
    "type": "Microsoft.Network/networkSecurityGroups",
    "name": "[variables('DmzSubnetResourceId')]",
    "location": "[resourceGroup().location]",
    "comments": "Security group to control access to the DMZ subnet",
    "properties": {
      "securityRules": [
        {
          "name": "Allow_Mqtts",
          "properties": {
            "description": "Allow Device Connection",
            "protocol": "Tcp",
            "sourcePortRange": "*",
            "sourceAddressPrefix": "Internet",
            "destinationAddressPrefix": "*",
            "destinationPortRange": "8883",
            "access": "Allow",
            "direction": "Inbound",
            "priority": 100
          }
        }
      ]
    }
  },
  {
    "apiVersion": "2016-03-30",
    "type": "Microsoft.Network/networkSecurityGroups",
    "name": "[variables('AppServicesSubnetResourceId')]",
    "location": "[resourceGroup().location]",
    "comments": "Security group to control access to the application services subnet",
    "properties": {
      "securityRules": [
        {
          "name": "Allow_Rest",
          "properties": {
            "description": "Allow REST calls from the network",
            "protocol": "Tcp",
            "sourcePortRange": "*",
            "sourceAddressPrefix": "[parameters('networkCIDR')]",
            "destinationAddressPrefix": "*",
            "destinationPortRange": "10080",
            "access": "Allow",
            "direction": "Inbound",
            "priority": 100
          }
        },
        {
          "name": "Allow_SupportSubnet",
          "properties": {
            "description": "Allow all access from the support network",
            "protocol": "*",
            "sourcePortRange": "*",
            "sourceAddressPrefix": "[parameters('supportCIDR')]",
            "destinationAddressPrefix": "*",
            "destinationPortRange": "*",
            "access": "Allow",
            "direction": "Inbound",
            "priority": 101
          }
        },
        {
          "name": "Allow_KpiMonitoring",
          "properties": {
            "description": "Allow the monitoring subnet to query logs and KPI",
            "protocol": "Tcp",
            "sourcePortRange": "*",
            "sourceAddressPrefix": "[parameters('monitoringCIDR')]",
            "destinationAddressPrefix": "*",
            "destinationPortRange": "10100",
            "access": "Allow",
            "direction": "Inbound",
            "priority": 102
          }
        }
      ]
    }
  },
  {
    "apiVersion": "2016-03-30",
    "type": "Microsoft.Network/networkSecurityGroups",
    "name": "[variables('DataSubnetResourceId')]",
    "location": "[resourceGroup().location]",
    "comments": "Security group to control access to the data subnet",
    "properties": {
      "securityRules": [
        {
          "name": "Allow_SupportSubnet",
          "properties": {
            "description": "Allow all access from the support network",
            "protocol": "*",
            "sourcePortRange": "*",
            "sourceAddressPrefix": "[parameters('supportCIDR')]",
            "destinationAddressPrefix": "*",
            "destinationPortRange": "*",
            "access": "Allow",
            "direction": "Inbound",
            "priority": 101
          }
        },
        {
          "name": "Allow_KpiMonitoring",
          "properties": {
            "description": "Allow the monitoring subnet to query logs and KPI",
            "protocol": "Tcp",
            "sourcePortRange": "*",
            "sourceAddressPrefix": "[parameters('monitoringCIDR')]",
            "destinationAddressPrefix": "*",
            "destinationPortRange": "10100",
            "access": "Allow",
            "direction": "Inbound",
            "priority": 102
          }
        }
      ]
    }
  },
  {
    "apiVersion": "2016-03-30",
    "type": "Microsoft.Network/networkSecurityGroups",
    "name": "[variables('MonitoringSubnetResourceId')]",
    "location": "[resourceGroup().location]",
    "comments": "Security group to control access to the monitoring subnet",
    "properties": {
      "securityRules": [
        {
          "name": "Allow_SupportSubnet",
          "properties": {
            "description": "Allow all access from the support network",
            "protocol": "*",
            "sourcePortRange": "*",
            "sourceAddressPrefix": "[parameters('supportCIDR')]",
            "destinationAddressPrefix": "*",
            "destinationPortRange": "*",
            "access": "Allow",
            "direction": "Inbound",
            "priority": 101
          }
        }
      ]
    }
  },
  {
    "apiVersion": "2016-03-30",
    "type": "Microsoft.Network/networkSecurityGroups",
    "name": "[variables('SupportSubnetResourceId')]",
    "location": "[resourceGroup().location]",
    "comments": "Security group to control access to the support subnet",
    "properties": {
      "securityRules": [
        {
          "name": "Allow_RDP",
          "properties": {
            "description": "Allow RDP access to the support network",
            "protocol": "Tcp",
            "sourcePortRange": "*",
            "sourceAddressPrefix": "Internet",
            "destinationAddressPrefix": "*",
            "destinationPortRange": "3389",
            "access": "Allow",
            "direction": "Inbound",
            "priority": 101
          }
        }
      ]
    }
  }

Notice in the support network that I have allowed RDP from the internet. This is clearly not what I will want to do in production etc. Restricting this rule at a later point to use a VPN or a specific IP address of a known client is the approach I would take. To make the appropriate ties to the network the first thing I need to do is add some depends on statements letting the resource manager know the proper order to create resources or they may not be ready when I need them. I add the following statement under the comments property in the network definition.

"dependsOn": [
  "[concat('Microsoft.Network/networkSecurityGroups/', variables('DmzSubnetResourceId'))]",
  "[concat('Microsoft.Network/networkSecurityGroups/', variables('AppServicesSubnetResourceId'))]",
  "[concat('Microsoft.Network/networkSecurityGroups/', variables('DataSubnetResourceId'))]",
  "[concat('Microsoft.Network/networkSecurityGroups/', variables('MonitoringSubnetResourceId'))]",
  "[concat('Microsoft.Network/networkSecurityGroups/', variables('SupportSubnetResourceId'))]"
]

I then assign the security groups to their appropriate subnets by changing the properties in each subnet definition to include it.

"subnets": [
{
  "name": "[variables('DmzSubnetName')]",
  "properties": 
  {
    "addressPrefix": "[parameters('dmzCIDR')]",
    "networkSecurityGroup": { "id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('DmzSubnetResourceId'))]" }
  }
},
{
  "name": "[variables('AppServicesSubnetName')]",
  "properties": 
  {
    "addressPrefix": "[parameters('appServicesCIDR')]",
    "networkSecurityGroup": { "id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('AppServicesSubnetResourceId'))]" }
  }
},
{
  "name": "[variables('DataSubnetName')]",
  "properties": 
  {
    "addressPrefix": "[parameters('dataCIDR')]",
    "networkSecurityGroup": { "id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('DataSubnetResourceId'))]" }
  }
},
{
  "name": "[variables('MonitoringSubnetName')]",
  "properties": 
  {
    "addressPrefix": "[parameters('monitoringCIDR')]",
    "networkSecurityGroup": { "id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('MonitoringSubnetResourceId'))]" }
  }
},
{
  "name": "[variables('SupportSubnetName')]",
  "properties": 
  {
    "addressPrefix": "[parameters('supportCIDR')]",
    "networkSecurityGroup": { "id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('SupportSubnetResourceId'))]" }
  }
}]

Running the script now will deploy the entire network with security groups in place giving me my template that will be the starting point for the solution. You can see the details in the image below Now that the standard network template for my project has been created I can use it to enforce my trust zones within the solution adding another layer to my security. << Previous Next >>

Ronin Building Blocks – Configuration

Chris Clayton (AzureCAT) — Fri, 25 Nov 2016 13:27:10 GMT

Database, XML, CSV, JSON, INI and event to my horror hard coding are all techniques that I have seen for handling configuration information. With .NET, it has typically been the accepted practice of using the host mechanisms configuration management style, whether that is web.config, app.config or settings.xml and in general that has worked out fine but what about code portability and leveraging knowledge or skills across the hosts as we move portable libraries etc. As you have probably guessed by now, there is a new framework released to help with this allowing us to use the same common techniques, not caring about the host, well for the consumers at least. This framework is the Microsoft.Extensions.Configuration framework and it leverages a provider system to abstract this information. This framework takes the traditional configuration key value pairs into an optional hierarchical key as we will see later. In short this is a framework that I will use for the IoT for it’s portability, surfacing of common paradigms and hierarchical structure. I will start with a basic sample in case you have not seen it before and build on that as I did with the logging framework. I started by creating a console application and adding the configuration framework NUGET package (Microsoft.Extensions.Configuration). To make use of the configuration framework we always start by creating an instance of the configuration builder. This is the heart of the configuration framework allowing me to associate configuration providers potentially pulling their information from a series of configuration stores. You will notice the use of the in-memory collection for the sample. This provider is a simple provider that takes a dictionary as its data source, but my code will not know the difference seeing the providers are abstracted. Using the configuration is as simple as providing a key.

  using System;
  using System.Collections.Generic;
  using Microsoft.Extensions.Configuration;

  class Program
  {
    static void Main(string[] args)
    {
      var builder = new ConfigurationBuilder();
      builder.AddInMemoryCollection(GetConfigurationData());
      var configuration = builder.Build();

      Console.WriteLine(configuration["Greeting"]);
      Console.WriteLine($"Your name is { configuration["User:FirstName"] } { configuration["User:LastName"] }");
    }

    private static Dictionary GetConfigurationData()
    {
      return new Dictionary
      {
        {"Greeting", "Hello World!"},
        { "User:FirstName", "Chris"},
        { "User:LastName", "Clayton"},
        { "AzureStorage:ConnectionString", "UseDeveleopmentStorage=true;"}
      };
    }
  }

Here is the output when I run the application Now that I have the basic framework setup I want to do more than just access the data as a key value pair so I will use sections instead. By using the framework’s section functionality, I can get a nice user container with all the information I care about like a dictionary. I do this by replacing this line of code.

  Console.WriteLine($"Your name is { configuration["User:FirstName"] } { configuration["User:LastName"] }");

The new code will retrieve the section of interest and use that like a dictionary.

  var userSection = configuration.GetSection("User");
  Console.WriteLine($"Your name is { userSection["FirstName"] } { userSection["LastName"] }");

As you can see with the output below it has not changed the content etc. but has made it so I can have multiple first names (e.g. user first name verses administrator first name) without collisions. Now that I can use this in a safe collision free way it not be great to get this into a simple POCO, coincidently they have already thought of this, through the use of extensions methods found in the Microsoft.Extensions.Configuration.Binder NUGET package. I have added a class named UserInformation that will contain the configuration data. I retrieve the section that the POCO will be created from and execute the get method on it.

  var userInfo = configuration.GetSection("User").Get();
  Console.WriteLine($"Your name is { userInfo.FirstName } { userInfo.LastName }");

The data has not changed but the usage of the framework and contained configuration is a lot more intuitive. I would say one of the most important things that are handled by a good configuration framework but I have not looked at yet encrypted values for things like my AzureStorage:ConnectionString setting. Although in this case I am just using the emulator the reality is that if I was to use this in production a plain text connection string is bound to trigger a ton of security questions. At first glance the framework provides this mechanism out of the box for us, but that is only for production. The final thing that I will do is look over a couple of the providers that exist out of the box for the framework that may be of interest in future development.

Microsoft.Extensions.Configuration.Json uses a JSON files as a source
Microsoft.Extensions.Configuration.CommandLine uses command line arguments as a source
Microsoft.Extensions.Configuration.EnvironmentVariables uses environment variables as a source
Microsoft.Extensions.Configuration.KeyVault uses Azure Key Vault as a source

There are many available configuration providers and if required I can customize it by creating my own. The framework is powerful and a welcome abstraction. << Previous Next >>

Ronin Building Blocks – Logging

Chris Clayton (AzureCAT) — Mon, 21 Nov 2016 16:36:27 GMT

Working with large scale distributed systems can be a challenge for the operations team whether it is handling capacity demands, scaling for cost, applying updates, rolling out new versions or keeping it safe and secure. Over the last two decades I have seen an ever-increasing demand for availability and uptime with an cyber attacks and other related threats on the rise. How do you support this? The answer is instrumentation, visibility and forethought! Significant planning and collaboration is required to instrument and surface the data for your operations team. A representative from the operations team should be part of the design discussions, or at least brought in at key points to discuss what they believe they need to support it. I find it amazing that this is something I see again and again, like the operations and development teams are part of a dysfunctional family. Sometimes the operations team has not even reviewed the solution until training day they are thrown into supporting it and it is too late for change. Could you imagine buying a car and when you pick it up you notice you have a tachometer but no speedometer. Now you are driving down the street having to gauge your speed by the RPM, engine noise and feel? Too much instrumentation can reduce performance and cause poor user experiences or drive the cost of goods too high to be viable; too little instrumentation results in the inability to debug and diagnose issues when they arise. I have heard this called psychic debugging due to the lack of numbers and facts to base it on. Trust me, this is not what I want to be doing! The first piece of instrumentation I always add is logging. Logging is your best friend in a distributed system, but obviously is not everything as there will be many other pieces of data you will need to debug a solution. I have found that logging frameworks feel like they are changed in a solution every year for reasons I hear like it is “better performance”, “more extensible” or “easiest to use”. I have seen this time and time again so it brings one word to mind as an architect… ABSTRACTION! Microsoft has recently released a logging framework that exposes a common set of interfaces while allowing other frameworks to be plugged in below the waterline. This means to change the logging framework, instead of having to modify my entire code base I can just change my code in a single location. For years now I have leveraged the concept of the ILogger interface to handle this having to then implement my own wrappers over the framework of the day. Seeing Microsoft’s framework already has the simple factories for the most common frameworks (e.g. NLog, Serilog, Loggr, Log4Net, etc.) I get that for free. The Microsoft framework is a NUGET package that works on .NET Core called Microsoft.Extensions.Logging. I am going to walk through the most interesting usage aspects for the solution I am working on, building up from the most basic in case you have not used this framework before. The typical way to start learning a framework is to do the “Hello World” sample, so why not just go with that. For this we will build out the plumbing with an implementation that will not have any logging sinks added, resulting in no messages being written. This is ok for the sample seeing we can wire those up easily after. The first thing I have to do is add the Microsoft.Extensions.Logging NUGET package to the solution. Using the framework always starts with the instantiation of the logger factory. This factory is the heart of the framework being used to manage the various logging providers. By default there are no logging providers added to it but it will function without exceptions being thrown as you see here.

  using Microsoft.Extensions.Logging;

  class Program
  {
    static void Main(string[] args)
    {
      using (ILoggerFactory factory = new LoggerFactory())
      {
        ILogger logger = factory.CreateLogger();

        logger.LogInformation("Hello world!");
      }
    }
  }

As you can see above I create a logger using the CreateLogger method which will use the name of the generic type provided as the category name. I find that more times than not this is the category name I want to use. I use the log level specific methods as a rule of thumb when writing out the messages verses the more generic Log method. The log levels can be used to turn up and down the verbosity for one or all providers as I will demonstrate later. Here is the log level enumeration that the framework provides.

  public enum LogLevel
  {
    Trace,
    Debug,
    Information,
    Warning,
    Error,
    Critical,
    None,
  }

Adding the debug stream as a logging sink is easy, I first add the NUGET package for the provider (Microsoft.Extensions.Logging.Debug) as seen here. Most of the providers you want to add will have NUGET packages that are prefixed with Microsoft.Extensions.Logging. Once this is installed I also log a message at warning level to demonstrate how you the provider shows each level slightly differently. Here is the updated code.

  using Microsoft.Extensions.Logging;

  class Program
  {
    static void Main(string[] args)
    {
      using (ILoggerFactory factory = new LoggerFactory())
      {
        factory.AddDebug();
        ILogger logger = factory.CreateLogger();

        logger.LogInformation("Hello world!");
        logger.LogWarning("Hello world, the aliens are invading!");
      }
    }
  }

And the output from the code that is displayed in the debug stream is as follows. RoninLogging.Sample.Program: Information: Hello world! RoninLogging.Sample.Program: Warning: Hello world, the aliens are invading! This is great, we now have a fully functioning logger that writes to the debug stream. I am also going to have it write to the console sink. To add the console sink I add it’s NUGET package. Using the add console extension method that the provider offers I attach it to the factory. This means that the logging framework will write each message to both the debug and the console provider. factory.AddConsole(); When I run the program now I see the same output as before being written to the debug stream and the following being output to the console. The important thing to note here is that not all the providers will have a nice and easy to use extension method like AddConsole or AddDebug, but it is common. If the provider does not expose this, you can create an instance of the provider and add it using the AddProvider method. You might even want to create your own provider to tie in with your operation team’s specialized logger, this can be done through the creation of your own customer provider. Looking at this now it is possible that the whole world does not want the informational message, just the warning so I would like to hide the informational from the console, but still have it show up in the debug channel. I can do this simply by using a filter when the console provider is added.

  factory.AddConsole((text, logLevel) => logLevel >= LogLevel.Warning);

When I run this, I can see that the debug stream has the same output as before but the console only has the warning. Using this framework in a demo application is simple enough, but what happens when I am using it in a larger system, what should I do? I like to share a factory with any instances that require the same providers and have each of those instances create their own instance of the logger to try and reduce mitigate between them. When I am working in larger systems there are quite a few times that I want to write log messages in a scope (call chain etc.). Think of the scenario where I am logging within a class but want to easily the way it arrived at a point in my code, such as a method to display an error message. I can accomplish this through the creation of logging scopes that will continue until the scope is disposed of as seen below. I made a couple changes to highlight the scope use that included removing the console filter I had added previously and telling the console provider to include the scope printout.

  using Microsoft.Extensions.Logging;

  class Program
  {
    static void Main(string[] args)
    {
      using (ILoggerFactory factory = new LoggerFactory())
      {
        factory.AddDebug();
        factory.AddConsole(true);

        ILogger logger = factory.CreateLogger();
        logger.LogInformation("Hello world!");

        using (logger.BeginScope("Planetary Surveillance System"))
        {
          logger.LogWarning("Hello world, the aliens are invading!");
        }

        logger.LogInformation("Goodbye world!");
      }
    }
  }

So now when we look at the output we see the scope included above the scoped message and absent on the others. Just for demonstration purposes let’s see what happens when we use the logger from something that is not in the scope (e.g. asynchronous tasks running that should not have their call stack modified) seeing that was one of my original concerns. I am modifying the code to have an asynchronous method called WriteAnotherMessageAsync that will be called before it enters the scope (index 1), but not write until in the scope and then call it within the scope again (index 2).

  using Microsoft.Extensions.Logging;

  class Program   
  {
    private static readonly SemaphoreSlim _control = new SemaphoreSlim(1);

    static void Main(string[] args)
    {
      using (ILoggerFactory factory = new LoggerFactory())
      {
        factory.AddDebug();
        factory.AddConsole(true);

        ILogger logger = factory.CreateLogger();
        _control.Wait();
        var writeTask = WriteAnotherMessageAsync(logger, 1);
        logger.LogInformation("Hello world!");

        using (logger.BeginScope("Planetary Surveillance System"))
        {
          logger.LogWarning("Hello world, the aliens are invading!");
          _control.Release();
          writeTask.Wait();
          WriteAnotherMessageAsync(logger, 2).Wait();
          logger.LogWarning("Hello world, the aliens are invading again!");
          _control.Wait();
        }

        logger.LogInformation("Goodbye world!");
        _control.Release();
      }
    }

    private static async Task WriteAnotherMessageAsync(ILogger logger, int index)
    {
      await _control.WaitAsync().ConfigureAwait(false);

      try
      {
        logger.LogInformation($"Another message { index }");
      }
      finally
      {
        _control.Release();
      }
    }
  }

You notice it honors the behavior we would expect as you see here! This means that scopes are very usable for us. Now that I have demonstrated all the basic semantics and boiler plate aspects of the logging framework you should be jumping out of your chair saying, “but this is just writing strings, we are to be doing better than that”. I will use the Serilog provider to write to Azure Table Storage with each of the properties passed into the logged message being broken out into individual properties as seen here.

  using Microsoft.Extensions.Logging;
  using Serilog;

  class Program
  {
    static void Main(string[] args)
    {
      var account = CloudStorageAccount.DevelopmentStorageAccount;

      Log.Logger = new LoggerConfiguration().WriteTo.AzureTableStorageWithProperties(account, storageTableName:"DemoTable").CreateLogger();

      using (ILoggerFactory factory = new LoggerFactory())
      {
        factory.AddSerilog();
        var logger = factory.CreateLogger();

        logger.LogInformation("Hello world {index}!", 1);
      }
    }
  }

Using this technique a table named DemoTable is created in my local storage emulator with an entity that has a broken out timestamp and index making it easier and typically faster to consume the logs when needed. Serilog offers a lot more sinks than Azure Table Storage so now that we know the capabilities I will choose the proper sink at a later point. This is the beauty of loose coupling after all! << Previous Next >>

Ronin Building Blocks - Dependency Injection

Chris Clayton (AzureCAT) — Fri, 18 Nov 2016 20:00:31 GMT

As much as I enjoy coding I must constantly decide whether there is more value in writing a custom piece of code or taking one off the shelf that is supported and has had more eyes on it. A great example of this can be seen in the next few building blocks starting with this one on Dependency Injection. I will say that I have always been a little hesitant on using the mainstream dependency injection frameworks as they often have too much reflection associated costs with them for the value I found they brought in a high scale system. I recently started to reflect more on this and began to change my stance due to:

The ease of substituting stubbed in classes during testing
The logical extensibility points it represents
Centralization of the instances lifetime control

Up until now I had not used the new Microsoft.Extensions.DependencyInjection framework but from my research I like its approach to allowing substitution other DI frameworks and what appears to be a lighter touch. Let's start by discussing the basics of what it is for anyone that has not used it in the past. Dependency Injection is a technique used to resolve references to an instance of a type using factories, typically to enable the Inversion of Control (IoC) pattern. A great time to use DI is with logging frameworks. In my experience this seems to be a common point of change within a solution due to developer, performance or operations requirements. If you did not use a DI style approach, custom or through a common framework, you would have to go back through most your code to make these changes. Through DI you can make the change easily in one or at most a handful of changes, saving cost and risk. Let’s use look at a simple example of a message generator that will be created using the DI framework. The first thing we need to do once we create our application is to add the NUGET package. The Microsoft.Extensions.DependencyInjection package installs the entire DI framework while the Microsoft.Extensions.DependencyInjection.Abstractions package provides the minimal interfaces as seen here. Next I will define the IMessageGenerator interface that defines the contract my instance will use regardless of the concrete implementation. To transform this console application so it uses simple DI is done with a few simple steps.

Add a reference to the framework
Create an instance of the service collection
Add the instances with the appropriate lifetime constructs (I will talk more about this shortly)
Build the dependency injection frameworks service provider
Request an instance of the object

As you can see here in the final transformation it appears to be a lot more work but it really only a few additional lines but the flexibility it provides is worth it.

  using System;
  using Microsoft.Extensions.DependencyInjection;

  class Program  
  {
    static void Main(string[] args)
    {
      var serviceCollection = new ServiceCollection();
      serviceCollection.AddSingleton(new SimpleMessageGenerator());
      var serviceProvider = serviceCollection.BuildServiceProvider();


      var generator = serviceProvider.GetService();
      Console.WriteLine(generator.GetMessage());
    }
  }

Running this program again we see the DI version of the program produces the same output. Now to show our dependency injection is working I will change the instance to a complex generator as follows and run it again.

  serviceCollection.AddSingleton(new ComplexMessageGenerator("test"));

That sample demonstrates a simple singleton version of the service that will continue to hand the same instance out to all clients that ask for an instance. That would be great if all the instance of classes I will every use did not contain state but there are quite a few that will. The following changes modify this class to hand out a different instance each time requested.

  using System;
  using Microsoft.Extensions.DependencyInjection;


  class Program
  {
    static void Main(string[] args)
    {
      var serviceCollection = new ServiceCollection();
      serviceCollection.AddTransient(ImplementationFactory);
      var serviceProvider = serviceCollection.BuildServiceProvider();

      var generator1 = serviceProvider.GetService();
      Console.WriteLine($"{ generator1.GetId():N}:{ generator1.GetMessage() }");

      var generator2 = serviceProvider.GetService();
      Console.WriteLine($"{ generator2.GetId():N}:{ generator2.GetMessage() }");
    }


    private static IMessageGenerator ImplementationFactory(IServiceProvider serviceProvider)
    {
      return new SimpleMessageGenerator();
    }
  }

As can be seen before one way to do this is using a factory method that will be invoked each time the instance of the service is requested. Obviously in this case I am using a simple instantiation of the instance in the method but I could replace it with a resource pool or other. Here is the console output proving by the instance id that it creates a new instance with each request. There are more options and techniques you can use with the DI framework but for our purposes that will most likely be the key areas of interest. Hopefully by seeing how easy and light this is you can understand why I now feel it is worth taking on. Building this solution using the framework allows me to introduce contracts and make the overall solution easily extensible. << Previous Next >>

Ronin Building Blocks - Resource Pools

Chris Clayton (AzureCAT) — Mon, 07 Nov 2016 15:24:11 GMT

In Azure, just as most other shared resource situations, there are policies put in place to govern fairness of use (most limits can be found here). These limits are the best friend of solutions that do not over consume, protecting them from abusers etc., but must be handled appropriately by solutions that push up against them. Take for example a solution that performs a lot of IO operations against an Azure Storage Account. If the solution is not architected to handle this scale will be limited to the maximum limits of the resource, which may not be acceptable. I am a firm believer that any requirement written on paper today is most likely going to change at some point in the future, it just seems to happen that way. What this means to me is even if my solution does not currently exceed the maximum limits of a resource I should make a conscious decision on how to handle it. One way I can overcome these solutions is to use more than one instance of the resource, but this introduces a couple of important decisions. 1. How do I determine which resource instance to use at what point in time? 2. Once chosen for a specific context do I need to be sticky to that instance? This is a great example of a time when a Resource Pool is the right fit. The best-known example of a Resource Pool is your local public library. Books cost money to buy and many people cannot afford the associated cost so the library buys a defined quantity of each book and allows people to check them out. When the person is done with the book they check it back into the library, freeing it up for others to use. A key piece that makes this possible is the fact that the resources are not permanently mutated by the consumer, so for the library example the person checking it out does not rip pages out at random. Every consumer gets the book in a known acceptable condition. In most use cases a solution that implements a Resource Pool either 1. Request a resource without providing any information leaving it to the Resource Pools logic to decide which resource to offer. 2. Request a resource providing a single piece of information (a key) and get a resource that is specifically assigned to process that value. To keep the pools as simple as possible I will focus on pools with immutable, pre-allocated resources. More advanced pooling can offer the ability to adjust pool levels but this comes with a series of challenges including locking in most cases. The basic functions of any of the Resource Pools I will use at this point will provide the ability set a maximum reference count per instance (e.g. an instance will not be handed out to anymore than 50 consumers at any point in time) and some general health metrics. The interface for the basic pool constructs look like

namespace Ronin.ResourcePools
{
  #region Using Clauses
  using System;
  #endregion

  /// 
  /// An interface that all pool objects implement
  /// 
  public interface IPool : IDisposable
  {
    #region Pool Health
    /// 
    /// The combined percentage of reference count to max reference count of all resources
    /// 
    double PercentFull { get; }

    /// 
    /// The number of resources that are full
    /// 
    int NumberOfFull { get; }

    /// 
    /// The number of resources in the pool
    /// 
    int Count { get; }

    /// 
    /// The number of resources in the pool that have no references allowed
    /// 
    int MaxZeroCount { get; }

    /// 
    /// The number of resources in the pool that have no references 
    /// 
    int ZeroCount { get; }

    /// 
    /// The number of references
    /// 
    int ReferenceCount { get; }

    /// 
    /// The percent of resources in the pool that have handed out all of their instances
    /// 
    double PercentFullResources { get; }
    #endregion
  }
}

As mentioned previously the pools will either be generic algorithms like Round Robin that hand the next available resource without worrying about affinity between calls. These simple pools will implement the following interface.

namespace Ronin.ResourcePools
{
  /// 
  /// The interface implemented by a pool that does not use a key to determine the resource to retrieve
  /// 
  /// The resource type
  public interface INonKeyedPool : IPool
  {
    /// 
    /// Takes an instance from the pool if available
    /// 
    /// The container holding the resource that was checked out or the default if not available
    ResourceContainer Take();
  }
}

For more advanced pools that handle things like affinity of resources we will offer the ability to provide a key string. This could easily be changed to any other type but string is a good general all around key.

namespace Ronin.ResourcePools
{
  /// 
  /// The interface implemented by a pool that uses a key to determine the resource to retrieve
  /// 
  /// The resource type
  public interface IKeyedPool : IPool
  {
    /// 
    /// Takes an instance from the pool if available
    /// 
    /// The key that is hashed to determine the resource to assign
    /// The container holding the resource that was checked out or the default if not available
    ResourceContainer Take(string key);
  }
}

As you can see in the previous two interfaces there is an option to retrieve an instance but there is no method to check it back in, this is by design. I decided to not use an explicit method to return the instance to the pool opting instead for the use of another object that when disposed of returns it to the queue. This technique seems to align with the technique used in the observable implementations. In the creation of this object I also made the conscious decision to not use a finalizer, forcing the consumer to be diligent about disposing of the container.

namespace Ronin.ResourcePools
{
  #region Using Clauses
  using Internal;
  using System;
  using System.Threading;
  #endregion

  /// 
  /// A container object used to return the resource to the pool
  /// 
  public class ResourceContainer : IDisposable
  {
    #region Variables
    /// 
    /// The tracker that the resource is associated with
    /// 
    private readonly ReferenceTracker _tracker;

    /// 
    /// The number of times the instance has been disposed of
    /// 
    private int _disposalCount;
    #endregion
    #region Constructors
    /// 
    /// A container to handle returning the resource to the pool
    /// 
    /// The tracker that the resource was retrieved from
    /// The resulting resource
    /// True if the resources was successfully retrieved
    /// The id of the reference tracker that provided the resource
    internal ResourceContainer(ReferenceTracker tracker, TResource resource, bool success, Guid id)
    {
      Resource = resource;
      Success = success;
      ReferenceTrackerId = id;
      _tracker = tracker;
    }
    #endregion
    #region Properties
    /// 
    /// The resulting resource
    /// 
    public TResource Resource { get; }

    /// 
    /// True if the resources was successfully retrieved
    /// 
    public bool Success { get; }

    /// 
    /// The id of the reference tracker that provided the resource
    /// 
    internal Guid ReferenceTrackerId { get; }
    #endregion
    #region Disposal
    /// 
    public void Dispose()
    {
      if (Interlocked.Increment(ref _disposalCount) == 1)
      {
        try
        {
          _tracker.Release();
        }
        catch (Exception)
        {
          // suppress exception in case the dispose is called after the pool is disposed of
        }
      }
    }
    #endregion
    #region Operators
    /// 
    /// An implicit conversion from the requested container to the resource requested
    /// 
    /// The container to be converted
    public static implicit operator TResource(ResourceContainer container)
    {
     return container.Resource;
    }
    #endregion
  }
}

Notice in the code above the use of an implicit conversion operator to the resource. This makes it easy for the consumer to receive the object from the pool and start using it as they would expect. The only risk with this approach I figured was the thought it may cause the consumer to forget to dispose of the container object because the resource in the event the resource did not support disposal. The pool will use an internal container for the resource to keep track of the number of references to the resource it contains. Letting the container own this logic keeps the overall pool logic simpler while making it easily reusable.

namespace Ronin.ResourcePools.Internal
{
  #region Using Clauses
  using System;
  using System.Threading;
  using Parameters;
  #endregion

  /// 
  /// Keeps track of the resources instance and the number consumers using it
  /// 
  internal class ReferenceTracker : IDisposable
  {
    #region Variables
    /// 
    /// The number of times the instance has been disposed of
    /// 
    private int _disposalCount;

    /// 
    /// The number of reference to the pooled instance
    /// 
    private int _referenceCount;

    /// 
    /// The maximum number of references that can be taken out on the instance
    /// 
    private readonly int _maxReferenceCount;

    /// 
    /// True if the resource should be disposed of when the reference tracker is disposed
    /// 
    private readonly bool _disposeOfResource;
    #endregion
    #region Constructors
    /// 
    /// The identifier that will be used to 
    /// 
    /// The instance of the resource being tracked
    /// The maximum references that can be taken out on this instance
    /// True if the resource should be disposed of when the reference tracker is disposed
    internal ReferenceTracker(TResource resource, int maxReferenceCount, bool disposeOfResource)
    {
      Guard.NotLessThan(nameof(maxReferenceCount), maxReferenceCount, 0);

      Resource = resource;
      Id = Guid.NewGuid();
      _disposeOfResource = disposeOfResource;
      _maxReferenceCount = maxReferenceCount;
    }
    #endregion
    #region Properties
    /// 
    /// The resource being tracked
    /// 
    internal TResource Resource { get; }

    /// 
    /// The unique identifier used to track the instance
    /// 
    internal Guid Id { get; }

    /// 
    /// The current reference count
    /// 
    internal int ReferenceCount => _referenceCount;

    /// 
    /// The maximum reference count allowed
    /// 
    internal int MaxReferenceCount => _maxReferenceCount;

    /// 
    /// True if the resource has all of the references it can handle
    /// 
    internal bool IsFull => _maxReferenceCount <= _referenceCount;
    #endregion
    #region Methods
    /// 
    /// Takes an instance from the pool if available
    /// 
    /// The resource that was checked out or the default if not available
    /// The id of the reference tracker
    /// True if the instance was retrieved successfully, false if not
    internal bool Take(out TResource resource, out Guid id)
    {
      var success = false;

      id = Id;

     // Check if the maximum count has been exceeded and if not return the instance
     if (Interlocked.Increment(ref _referenceCount) > _maxReferenceCount)
     {
       Interlocked.Decrement(ref _referenceCount);
       resource = default(TResource);
     }
     else
     {
       resource = Resource;
       success = true;
     }

     return success;
  }

  /// 
  /// Reduces the reference count by 1
  /// 
  internal void Release()
  {
    Interlocked.Decrement(ref _referenceCount);
  }
  #endregion
  #region Disposal
  /// 
  public void Dispose()
  {
    if (Interlocked.Increment(ref _disposalCount) == 1)
    {
      if (_disposeOfResource) (Resource as IDisposable)?.Dispose();
    }
  }
  #endregion
}

The basic pool implementation is going to store a list of immutable resources that it checks in and out while reporting on the standard Pool health properties. Here is my implementation.

namespace Ronin.ResourcePools.Internal
{
  #region Using Clauses
  using System;
  using System.Collections.Generic;
  using System.Collections.Immutable;
  using System.Linq;
  using System.Threading;
  using Parameters;
  #endregion

  /// 
  /// A base pool implementation
  /// 
  /// The resource stored in the pool
  public abstract class Pool : IPool
  {
    #region Variables
    /// 
    /// The pool of resource instances
    /// 
    private readonly ImmutableArray> _pool;

    /// 
    /// Tracks the number of times the instance has been disposed of
    /// 
    private int _disposalCount;

    /// 
    /// True to dispose when complete
    /// 
    private readonly bool _disposeOfResource;
    #endregion
    #region Constructors
    /// 
    /// Initializes a new instance of the type
    /// 
    /// The resources in the pool
    /// The maximum reference count allowed for each resource instance
    /// True to dispose when complete
    protected Pool(IEnumerable resources, int maxReferenceCount, bool disposeOfResource)
    {
      Guard.NotLessThan(nameof(maxReferenceCount), maxReferenceCount, 0);

      _disposeOfResource = disposeOfResource;

      _pool = resources.Select(resource => new ReferenceTracker(resource, maxReferenceCount, disposeOfResource)).ToImmutableArray();
    }
    #endregion
    #region Resource Check Out
    /// 
    /// Takes an instance from the pool if available
    /// 
    /// The index of the resource to retrieve
    /// The container holding the resource that was checked out or the default if not available
    protected ResourceContainer InternalTake(int index)
    {
      Guard.NotLessThan(nameof(index), index, 0);
      Guard.NotMoreThan(nameof(index), index, _pool.Length);

      TResource resource;
      Guid id;

      var tracker = _pool[index];
      var success = tracker.Take(out resource, out id);

      return new ResourceContainer(tracker, resource, success, id);
    }

    /// 
    /// Takes an instance from the pool if available and if not tries the next index until it either gets one or arrives at the original index again
    /// 
    /// The index of the resource to start trying to retrieve at and replaces the value with the first success index
    /// The container holding the resource that was checked out or the default if not available
    protected ResourceContainer InternalTakeNextAvailable(ref int index)
    {
      Guard.NotLessThan(nameof(index), index, 0);
      Guard.NotMoreThan(nameof(index), index, _pool.Length);

      bool success;
      TResource resource;
      ReferenceTracker tracker;
      Guid id;
      var attemptIndex = index;

      do
      {
        if (attemptIndex >= _pool.Length) attemptIndex = 0;
        tracker = _pool[index];
        success = tracker.Take(out resource, out id);
        if (success) index = attemptIndex;
      } while (++attemptIndex != index && !success);

      return new ResourceContainer(tracker, resource, success, id);
    }

    /// 
    /// Takes the first available instance from the pool
    /// 
    /// The container holding the resource that was checked out or the default if not available
    protected ResourceContainer InternalTakeFirstAvailable()
    {
      var index = 0;
      return InternalTakeNextAvailable(ref index);
    }

    /// 
    /// Takes the lowest reference count instance from the pool
    /// 
    /// The container holding the resource that was checked out or the default if not available
    protected ResourceContainer InternalTakeLowestReference()
    {
      var success = false;
      TResource resource = default(TResource);
      ReferenceTracker tracker;
      Guid id = Guid.Empty;
      var attemptIndex = 0;

      do
      {
        tracker = _pool.Where(item => item.ReferenceCount < item.MaxReferenceCount).OrderBy(item => item.ReferenceCount).Take(1).FirstOrDefault();

        if (tracker != null)
        {   
          success = tracker.Take(out resource, out id);
        }
      } while (++attemptIndex < 100 && !success);

      return new ResourceContainer(tracker, resource, success, id);
    }
    #endregion
    #region Pool Health
    /// 
    /// The combined percentage of reference count to max reference count of all resources
    /// 
    public double PercentFull
    {
      get
      {
        var items = _pool.Select(item => new Tuple(item.ReferenceCount, item.MaxReferenceCount)).ToArray();
        var referenceCount = items.Sum(item => item.Item1);
        var maxReferenceCount = items.Sum(item => item.Item2);

        return maxReferenceCount == 0 ? 100.0 : (double)referenceCount / maxReferenceCount;
      }
    }

    /// 
    /// The number of resources that are full
    /// 
    public int NumberOfFull
    {
      get { return _pool.Count(item => item.IsFull); }
    }   

    /// 
    /// The number of resources in the pool
    /// 
    public int Count => _pool.Length;

    /// 
    /// The number of resources in the pool that have no references allowed
    /// 
    public int MaxZeroCount => _pool.Count(item => item.MaxReferenceCount == 0);

    /// 
    /// The number of resources in the pool that have no references 
    /// 
    public int ZeroCount => _pool.Count(item => item.ReferenceCount == 0);

    /// 
    /// The number of references
    /// 
    public int ReferenceCount => _pool.Sum(item => item.ReferenceCount);

    /// 
    /// The percent of resources in the pool that have handed out all of their instances
    /// 
    public double PercentFullResources
    {
      get
      {
        var resourceCount = _pool.Length;

        return resourceCount < 1 ? 100.0 : (double)NumberOfFull / resourceCount;
      }
    }
    #endregion
    #region Disposal
    /// 
    public void Dispose()
    {
      if (Interlocked.Increment(ref _disposalCount) == 1)
      {
        if (_disposeOfResource)
        {
          foreach (var item in _pool)
          {
            item.Dispose();
          }  
        }
      }
    }
    #endregion
  }
}

Starting with the simple implementations I decided to implement a simple round robin pool. This pool will just iterate through the resources handing back one then the next. This will provide no affinity of resources for the consumer but will be simple and fast.

namespace Ronin.ResourcePools.Simple
{
  #region Using Clauses
  using System;
  using System.Collections.Generic;
  using System.Threading;
  using System.Threading.Tasks;
  using Internal;
  #endregion

  /// 
  /// A pool that keeps a collection of resources, iterating through them as a new one is requested
  /// 
  public class RoundRobinPool : Pool, INonKeyedPool
  {
    #region Variables
    /// 
    /// The number of items in the pool
    /// 
    private readonly int _poolCount;

    /// 
    /// Keeps track of the index of the last pool item returned
    /// 
    private long _lastIndex = -1;
    #endregion
    #region Constructors
    /// 
    /// Initializes a new instance of the type
    /// 
    /// The resources in the pool
    /// True to dispose when complete
    public RoundRobinPool(IEnumerable resources, bool disposeOfResource) : base(resources, int.MaxValue, disposeOfResource)
    {
      _poolCount = Count;

      if (_poolCount < 1) throw new ArgumentException("Must have at least one element", nameof(resources));
    }
    #endregion
    #region Methods
    /// 
    /// Takes an instance from the pool if available
    /// 
    /// The container holding the resource that was checked out or the default if not available
    public ResourceContainer Take()
    {
      ResourceContainer resource = null;
      var attemptNumber = 0;

      do
      {
        var index = Interlocked.Increment(ref _lastIndex);

        resource = InternalTake((int)(index % _poolCount));
      } while (resource == null && attemptNumber++ < 100);

      return resource;
    }
    #endregion
  }
}

For the string based (key) version the easiest way to provide affinity on a pool of resources that will never contract or expand is to use a hash algorithm. There are a lot of write-ups on different hash algorithms and their benefits in dispersal but for simplicity and speed I will use the following hash algorithm. I have decided to put the hash algorithm in its own class for easy central changes for any pool that implements this functionality.

namespace Ronin.ResourcePools.Internal
{
  #region Using Clauses
  using System.Text;
  #endregion

  /// 
  /// Common utilities for hashing used by the library
  /// 
  internal static class Hashes
  {
    /// 
    /// Hashes the string to a ulong value
    /// 
    /// The value to hash
    /// The hashed value
    internal static ulong HashString(string value)
    {
      var hash = 3074457345618258791ul;
      var array = Encoding.ASCII.GetBytes(value);

      // ReSharper disable once ForCanBeConvertedToForeach
      for (var index = 0; index < array.Length; index++)
      {
        unchecked
        {
          hash ^= array[index];
          hash *= 3074457345618258799ul;
        }
      }

      return hash;
    }
  }
}

This first keyed resource pool will be the primary one I use in simple "sharding scenarios". This will allow for resources affinity by key while remaining simple and performant.

namespace Ronin.ResourcePools.Simple
{
  #region Using Clauses
  using System;
  using System.Collections.Generic;
  using System.Threading;
  using System.Threading.Tasks;
  using Internal;
  #endregion

  /// 
  /// A pool that keeps a collection of resources using as simple hash algrothm to add references
  /// 
  public class HashPool : Pool, IKeyedPool
  {
    #region Variables
    /// 
    /// The number of items in the pool
    /// 
    private readonly ulong _poolCount;
    #endregion
    #region Constructors
    /// 
    /// Initializes a new instance of the type
    /// 
    /// The resources in the pool
    /// True to dispose when complete
    public HashPool(IEnumerable resources, bool disposeOfResource) : base(resources, int.MaxValue, disposeOfResource)
    {
      _poolCount = (ulong)Count;

      if (_poolCount < 1) throw new ArgumentException("Must have at least one element", nameof(resources));
    }
    #endregion
    #region Methods
    /// 
    /// Takes an instance from the pool if available
    /// 
    /// The key that is hashed to determine the resource to assign
    /// The container holding the resource that was checked out or the default if not available
    public ResourceContainer Take(string key)
    {
      return InternalTake((int)(Hashes.HashString(key) % _poolCount));
    }
    #endregion
  }
}

The final Resource Pool implementation that I will demonstrate here is a balanced pool. This pool will return the resource with the lowest reference count each time. The problem with this is the performance of the query to get each instance but at times it may be worth the overhead to get balance.

namespace Ronin.ResourcePools.Advanced
{
  #region Using Clauses
  using System;
  using System.Collections.Generic;
  using System.Threading;
  using System.Threading.Tasks;
  using Internal;
  #endregion

  /// 
  /// A pool that keeps a collection of resources using as simple hash algrothm to add references
  /// 
  public class BalancedPool : Pool, INonKeyedPool
  {
    #region Constructors
    /// 
    /// Initializes a new instance of the type
    /// 
    /// The resources in the pool
    /// The maximum reference count allowed for each resource instance
    /// True to dispose when complete
    public BalancedPool(IEnumerable resources, int maxReferenceCount, bool disposeOfResource) : base(resources, maxReferenceCount, disposeOfResource)
    {
      if ((ulong)Count < 1) throw new ArgumentException("Must have at least one element", nameof(resources));
    }
    #endregion
    #region Methods
    /// 
    /// Takes an instance from the pool if available
    /// 
    /// The container holding the resource that was checked out or the default if not available
    public ResourceContainer Take()
    {
      return InternalTakeLowestReference();
    }
    #endregion
  }
}

The use of resource pools are key to overcoming some of the limitations of shared resources and maximizing throughput and performance. These are just three implementations of Resource Pools and there are many more. In general these three should cover most of the scenarios we will run into. << Previous Next >>

Ronin Building Blocks - Precision Time

Chris Clayton (AzureCAT) — Thu, 03 Nov 2016 13:40:19 GMT

Have you ever been working on debugging or diagnosing an issue and you need to get the time it takes to execute an external service call? I personally don't want to use a stop watch, although it is a viable solution, when I already have the start time (DateTime.UtcNow) that I wrote to my logs, I just want to subtract the start time from DateTime.UtcNow when I am done. Trying this approach presents some really strange and unreliable timings because the approximate resolution of the DateTime.UtcNow property 10 milliseconds. This was so frustrating to see a bunch of calls that had no I/O take zero millseconds so I decide to try a test. I created an array of 1 million DateTime elements and looped through populating the elements with the current date time then checked to see how many operations occurred before it advanced to the next time (approximately 10 milliseconds later). This resulted in around 87000 entries being written before the time advanced. Yes so if you were to use the DateTime.UtcNow method instead of the stop watch your analysis would show it took no notable time to perform those 87000 operations. To add to this looking at logs from my system I couldn't determine the order of operations if there was not more than 10 milliseconds between the entries. All of this begs for something better. Please remember that in distributed systems solutions the time makes no sense across machines, even if the machine is time synced. The computer times will always have some variance between them and a wall clock solution should only be used for single machine diagnostics or to act as a approximate alignment to events that occur. The solution I am going to outline here is to create a precision time class to run on Windows 2012 R2 and beyond. For this reason instead of using .NET Core I will build a classic C# library. The API that is the center of the library is the GetSystemTimePreciseAsFileTime providing an approximate resolution that is less than 1 microsecond. Performing the same test that I did on the DateTime.UtcNow this was able to reduce my entries written before the time changes from 87000 to 4. This is a great improvement assuming that even writing a log will take a minimum of 4 calls. When doing a PInvoke into unmanaged code a runtime security check is performed resulting in the call stack being walked. To improve the performance of this call I have decided to supress this check but that is something that should be done at your own discretion. Validate the security implications and decide if they are acceptable, in my scenario they are. I wrap the native method PInvoke signatures in an internal class to control access to them.

namespace Ronin.Clocks.Precision.Internal
{
    #region Using Clauses
    using System.Security;
    using System.Runtime.InteropServices;
    #endregion

    /// 
    /// Native methods that are not exposed externally
    /// 
    [SuppressUnmanagedCodeSecurity]
    internal static class NativeMethods
    {
        #region Methods
        /// 
        /// Definition for the Windows API to return the precise time.
        /// 
        /// The current system time as a file time
        [DllImport("kernel32.dll", ExactSpelling = true, EntryPoint = "GetSystemTimePreciseAsFileTime", SetLastError = true)]
        internal static extern void GetSystemTime(ref long fileTime);
        #endregion
    }
}

Now to expose it I want to offer some simple properties and methods that should be self explanatory to anyone consuming the library. I decided to expose a UtcNow property that mirrors the DateTime.UtcNow but with better resolution and a UtcNowOffset that returns a DateTimeOffset instead.

namespace Ronin.Clocks.Precision
{
    #region Using Clauses
    using System;
    using Internal;
    #endregion
    /// 
    /// A set of methods that expose the Windows System Time functions to retrieve a more precise time
    /// 
    public static class PrecisionTime
    {
        #region Properties
        /// 
        /// Returns the current system time as a file time construct
        /// 
        public static long CurrentSystemTime
        {
            get
            {
                long time = 0;

                NativeMethods.GetSystemTime(ref time);

                return time;
            }
        }

        /// 
        /// Returns a precise DateTime representing the current system time
        /// 
        public static DateTime UtcNow => DateTime.FromFileTimeUtc(CurrentSystemTime);

        /// 
        /// Returns a precise DateTimeOffset representing the current system time
        /// 
        public static DateTimeOffset UtcNowOffset => DateTimeOffset.FromFileTime(CurrentSystemTime);
        #endregion
        #region Methods
        /// 
        /// Retrieves a precise DateTime representing the current system time
        /// 
        /// The current system time
        public static DateTime GetDateTime()
        {
            return DateTime.FromFileTimeUtc(CurrentSystemTime);
        }

        /// 
        /// Retrieves a precise DateTimeOffset representing the current system time
        /// 
        /// The current system time
        public static DateTimeOffset GetDateTimeOffset()
        {
            return DateTimeOffset.FromFileTime(CurrentSystemTime);
        }
        #endregion
    }
}

You can choose to hide the methods but I like giving as many options as possible. So now let's see how to use it.

            var startTime = PrecisionTime.UtcNow;
            // Do my work
            var duration = PrecisionTime.UtcNow - startTime;

That wraps up the precision time block!

<< Previous Next >>