All the files relating to post can be found here, along with the script for rack awareness.

So what would be involved in pushing out a set of configuration changes to all nodes in the cluster?

If you consider each node in the cluster you would need:

• A script that modifies a property name/value pair for a given configuration file on the cluster node itself
• A script, along with corresponding configuration specifications, for making all the required configuration changes that are required on the cluster node

So lets tackle these first.

As a recap, within each configuration file (source) the property names and values have the following format:

<property>
  <name>topology.script.file.name</name>
  <value>hadoop-rack-configuration.cmd</value>
</property>

Thus, the starting point for managing configuration changes would be a set of configuration changes that one would like to make to the cluster nodes. In this implementation the configuration file would look like the following:

configure-cluster-properties.txt

# YARN Configuration
yarn-site.xml, yarn.nodemanager.resource.memory-mb, 8192
yarn-site.xml, yarn.scheduler.minimum-allocation-mb, 2048
 
# MR Configurations
mapred-site.xml, mapreduce.map.memory.mb, 2048
mapred-site.xml, mapreduce.reduce.memory.mb, 4096
mapred-site.xml, mapreduce.map.java.opts, -Xmx1536m
mapred-site.xml, mapreduce.reduce.java.opts, -Xmx3072m

This is basically a list of the configuration source, property name, and property values, comma separated, for all the required changes to be made to each node.

Based on these three values as inputs to make a single configuration change the PowerShell script would be as follows:

configure-cluster-property.ps1

param([string] $config_file_name, [string] $config_property_name, [string] $config_property_value)
 
if(Test-Path -Path $config_file_name -PathType Leaf) {
    $conf_doc = [System.Xml.XmlDocument](Get-Content $config_file_name)
 
    $property_node = ($conf_doc.DocumentElement.property | Where-Object {$_.name -eq $config_property_name})
    If ($property_node) {
        # Element found so ensure the property is correctly set
        write-host "$config_property_name Element Found, so updating value to $config_property_value..."
        $property_node.Value = $config_property_value
    } else {
        # No Element found so add a new one to the document
        write-host "$config_property_name Element Not Present, adding new element..."
        $property_element = $conf_doc.CreateElement("property")
 
        $property_element_name = $conf_doc.CreateElement("name")
        $property_element_name.AppendChild($conf_doc.CreateTextNode($config_property_name))
        $property_element.AppendChild($property_element_name)
 
        $property_element_value = $conf_doc.CreateElement("value")
        $property_element_value.AppendChild($conf_doc.CreateTextNode($config_property_value))
        $property_element.AppendChild($property_element_value)
 
        $conf_doc.DocumentElement.AppendChild($property_element)
    }
 
    $conf_doc.Save($config_file_name)
} else {
    Write-Error "Configuration File $config_file_name cannot be found"
}

This script opens the configuration document as an XmlDocument and then locates the property Element, where the name Element matches the one requiring modification. If the corresponding property Element is found then its value is changed. If the Element is not found then the property Element is added to the document.

To use this script the input file of configuration changes just needs to be parsed and for each configuration line a call is made to the property change modification script:

configure-cluster-properties.ps1

param([string] $working_path, [string] $configuration_file)
 
$hadoop_config_dir = $env:HADOOP_CONF_DIR
$configurationFile = "$working_path\$configuration_file";
 
function Get-PropertiesTable {
 
    $properties = @(); 
 
    if (Test-Path -Path $configurationFile -PathType Leaf) {    
        $propertylines = Get-Content $configurationFile
        foreach ($propertyline in $propertylines)
        {
            $propertyline = $propertyline.Trim()
            if (($propertyline) -and (-not $propertyline.StartsWith("#")))
            {
                $propertyline_values = ,@($propertyline -Split ",|\t") | % {$_.Trim()}
                if ($propertyline_values.length -eq 3) {
                    $hdp_file = $propertyline_values[0]
                    $properties += New-Object PSObject –Property @{ConfigFile="$hadoop_config_dir\$hdp_file"; PropertyName=$propertyline_values[1]; PropertyValue=$propertyline_values[2]}
                }
            }
        }            
    } else {
        Write-Error "Configuration File $configurationFile cannot be found..."
    }
 
    return $properties;
}
 
Get-PropertiesTable | % {. $working_path\configure-cluster-property.ps1 -config_file_name $_.ConfigFile -config_property_name $_.PropertyName -config_property_value $_.PropertyValue }

Calling this script on a local node is then easily achieved using a command line file:

set working_path=%~dp0
if %working_path:~-1%==\ set working_path=%working_path:~0,-1%
 
set script_path=%working_path%\configure-cluster-properties.ps1
set configuration_file=configure-cluster-properties.txt
 
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& '%script_path%' -working_path '%working_path%' -configuration_file '%configuration_file%'"
 
pause

However having to do this on possibly hundreds of remote machines is not feasible; especially if your cluster consists of hundreds of nodes.

Once again PowerShell provides an easy solution. A script can be created that can be used to push these configuration files to all the cluster nodes and then to execute the configuration change script remotely on all the nodes:

push_configure_cluster_properties.ps1

param( [string] $source_path, [string] $target_path, [string] $files_list, [string] $nodes_list, [int] $batch_size, [string] $cmd_word ) # followed by $cmd_args
 
$cmd_args = $args  # capture this right away so can use it in functions
 
function ListSplitUnique($values)
{
    return ,@((($values -Split ",") | % {$_.Trim()} | where-object {$_ -ne ""} ) | Sort-Object -Unique)
}
 
function GetRequiredNodes($nodelistpath)
{
    $required_nodes = @()
    write-host "$nodelistpath"
    if ($nodelistpath -notlike "skip") {
        if (Test-Path -Path $nodelistpath -PathType Leaf){
            # // File exists
            (get-content $nodelistpath) | foreach-object {$required_nodes += ListSplitUnique($_)}
        } else {
            # // File does not exist
            $required_nodes = ListSplitUnique($nodelistpath)
        }
    }
 
    return $required_nodes;
}
 
function GetIPAddress($hostname)
{
    try
    {
        [System.Net.Dns]::GetHostAddresses($hostname) | ForEach-Object { if ($_.AddressFamily -eq "InterNetwork") { $_.IPAddressToString } }
    }
    catch
    {
        throw "Error resolving IPAddress for host '$hostname'"
    }
}
 
function IsSameHost(
    [string] [parameter( Position=0, Mandatory=$true )] $host1,
    [array] [parameter( Position=1, Mandatory=$false )] $host2ips = ((GetIPAddress $env:COMPUTERNAME) -as [array]))
{
    $host1ips = ((GetIPAddress $host1) -as [array])
    $heq = Compare-Object $host1ips $host2ips -ExcludeDifferent -IncludeEqual
    return ($heq -ne $null)
}
 
#convert $files_list to an array
$all_files = @()
if ($files_list -ne $null) {
    $all_files = ListSplitUnique($files_list)
}
 
#convert $nodes_list to an array
$required_nodes = GetRequiredNodes($nodes_list);
 
# define some global values
$all_nodes = @()
$nl = [Environment]::NewLine
$current_host = gc env:computername
$ips = ((GetIPAddress $env:COMPUTERNAME) -as [array])
 
 
function Check-Files 
{    
    # check for missing arguments
    if (($all_files.Count -lt 2) -or ($cmd_word = $null)) {
        write-error "Usage: push_configure_hdp_dependencies.ps1 <source_path> <target_path> <files_list> <cmd_line...>
Files_list is a string containing a comma-delimited list of simple
file names, to be found in source_path and copied to target_path on
each node.  Cmd_line is a sequence of command line tokens constituting 
a valid PowerShell execution."
        Exit -1;
    }
    
    # validate that $target_path is an absolute path including volume name
    $path_pieces = $target_path.Split(":")
    if ($path_pieces.Length -lt 2) {
        write-error "$target_path path does not include volume name with colon.  Exiting."
        Exit -1;
    }
    if ($path_pieces.Length -gt 2) {
        write-error "$target_path path has multiple colons.  Exiting."
        Exit -1;
    }
    
    # validate existence and accessibility of files in the source_path
    if (! (Test-Path $source_path)) {
        write-error "$source_path is not accessible from install master server.  Exiting."
        Exit -1;
    }
    cd "$source_path"
    write-output "$($nl)Push install files: "
    ls $all_files
    if (! $?) {
        write-error "Some requested files are missing from $source_path.  Exiting."
        Exit -1
    }
    write-output "$nl" 
}
 
function Summarize-Results($results_set)
{
    $remoteErrors = @()
    $remoteSuccessCount = 0
    $remoteFailureCount = 0
    $remoteOutputResult = 0
    
    foreach ( $result in $results_set ) {
        if ( $result -is [System.Management.Automation.ErrorRecord] ) {
            $remoteFailureCount++
        }
        ElseIf ($result -eq "Done.") {
            $remoteSuccessCount++ 
        } Else {
            $remoteOutputResult++
        }
    }
    
    return New-Object psobject -Property @{
        remoteErrors = $results_set;
        remoteSuccessCount = $remoteSuccessCount;
        remoteFailureCount = $remoteFailureCount;
        remoteOutputResult = $remoteOutputResult
    }
}
 
function Report-Results($summary_results) {
    # Report summarized results to user
    # $summary_results must be the output of Summarize-Results()    
    write-output "Summary:"
    write-output ("" + $results.remoteSuccessCount + " nodes successfully completed")
    
    if ($results.remoteFailureCount -gt 0) {
        write-output ("" + $results.remoteFailureCount + " failure messages.")
    }
    
    if ($results.remoteOutputResult -gt 0) {
        write-output ("" + $results.remoteOutputResult + " Output Lines.")
    }
}
 
function CopyInstall-Files($node)
{
    if (-not ((($node -ieq $current_host) -or (IsSameHost $node $ips)) -and ($source_path -ieq $target_path)))  {
        # convert $target_path into the corresponding admin share path, so we can push the files to the node
        $tgtDir = '\\' + ($node) + '\' +  $target_path.ToString().Replace(':', '$')
 
        # attempt to create the target install directory on the remote node if it doesn't already exist
        if(! (Test-Path "$tgtDir")) { $r = mkdir "$tgtDir" }
 
        # validate that the $tgtDir admin share exists and is accessible on the remote node
        if (! (Test-Path "$tgtDir") ) {
            write-error "$target_path on $node is not accessible by admin share.  Skipping."
            return $false
        }
 
        # push the files to each node.  Skip node if any errors.
        cd "$source_path"
        cp $all_files "$tgtDir" -Force -ErrorAction Stop
        if (! $?) {
            write-error "Some files could not be pushed to $node.  Skipping."
            return $false
        }
        
        return $true
    }
    else {
        return $true
    }
}
 
function PushInstall-Files($nodes)
{
    $copied_nodes = @()
    
    foreach ($node in $nodes) {
        # tag output with node name:
        write-output ("Copying files for node $node $nl")    
        
        # Copy the install files and build list of those needed an install
        if (CopyInstall-Files($node)) {
            if (($node -ieq $current_host) -or (IsSameHost $node $ips)){
                write-output "Skipping $node because it is current node $nl"
            }        
            else {
                write-output ("Copied files to node $node $nl")    
                $copied_nodes += $node
            }
        }
    }
    
    if ($copied_nodes.length -gt 0) {
        # invoke the install, and wait for it to complete
        write-output ("Pushing to nodes: " + ($copied_nodes -join " ") + "$nl")
        write-output "With command : $cmd_word $cmd_args"
 
        $arg_list = @("-file", "$cmd_word") + $cmd_args
        Invoke-Command -ComputerName $copied_nodes -ScriptBlock {
            # everything in this scriptblock runs local to the node    
            $node = $env:COMPUTERNAME
            
            # launch the ps1 file
            $proc_record = Start-Process powershell.exe -ArgumentList $using:arg_list -PassThru -Wait -Verb "RunAs"
            if (! $?) {
                write-error "Start-Process call failed for $node.  Error $($error[0]).  Skipping."
                return
            }
            if ($proc_record.ExitCode -ne 0) {
                        write-error "$using:cmd_word failed for $node.  Error code $($proc_record.ExitCode).  Skipping."
                        write-error "For error code meanings, see http://msdn.microsoft.com/en-us/library/windows/desktop/aa390890(v=vs.85).aspx"
                        return
                    }
            
            write-output "Installed files to node $node $nl"
            write-output "Done."  #getting to this line indicates success
 
            } 2>&1
    }    
    write-output "$nl"
}
 
function Install-HDPDependencies( )
{
    # validate the input files
    Check-Files
    
    # Do the install.
    $all_results = @()
    write-output ("Nodes to install (parallel): " + ($required_nodes -join " ") + "$nl")
    
    # bucket the nodes for parallel installation
    $required_buckets = @()
    if ($batch_size -le 1) { $batch_size = 1 }
    if ($required_nodes.length -le $batch_size) {
        $required_buckets= ,@($required_nodes)
    } else {
        $required_buckets= for($i=0; $i -lt $required_nodes.length; $i+=$batch_size) { , $required_nodes[$i..($i+$batch_size-1)]}
    }
    
    foreach ($required_bucket in $required_buckets) {
        write-output ("Pushing to nodes: " + ($required_bucket -join " ") + "...")
        $push_results = PushInstall-Files($required_bucket) 2>&1
        if ($push_results) {
            write-output "output: $push_results"
            $all_results += $push_results
            $push_results
        }
    }                
 
    write-output "$nl$nl"
    
    # parse $all_results for failure messages and alert user
    $results = Summarize-Results $all_results
    Report-Results $results
}    
 
# Invoke the functionality of the script
Install-HDPDependencies

I will leave it up to the reader to look over what this script is doing. In short this script takes a series of files and deploys them to all nodes in the cluster. It then remotely executes a specified PowerShell script file on each node. This script forms the basis of not only deploying the configuration changes but can also be used for deploying dependency software to cluster nodes, and deploying rack awareness.

Calling this script is a bit more involved but can be managed through the following command file:

set working_path=%~dp0
if %working_path:~-1%==\ set working_path=%working_path:~0,-1%
 
set script_path=%working_path%\push_configure_cluster_properties.ps1
set source_path=%working_path%
 
set target_root=E:\hdp-installassets
set target_path=%target_root%\configuration
 
set files_list=configure-cluster-property.ps1,configure-cluster-properties.ps1,configure-cluster-properties.cmd,configure-cluster-properties.txt
set nodes_list=%working_path%\clusternodes.txt
 
set configuration_file=configure-cluster-properties.txt
 
set cmd_word=%target_path%\configure-cluster-properties.ps1
set cmd_args=-working_path '%target_path%' -configuration_file '%configuration_file%'
 
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& '%script_path%' -source_path '%source_path%' -target_path '%target_path%' -files_list '%files_list%' -nodes_list '%nodes_list%' -batch_size 4 -cmd_word '%cmd_word%' %cmd_args%"
 
pause

The items one will need to specify are the remote location where the configuration files can be deployed to, and the list of nodes forming the cluster. The list of cluster nodes can be a file of the names, such as the one below, or just a simple comma separated list:

HDP-NN1.ACME.LAB,HDP-NN2.ACME.LAB,HDP-MGMT1.ACME.LAB
HDP-DATA01.ACME.LAB,HDP-DATA02.ACME.LAB,HDP-DATA03.ACME.LAB,HDP-DATA04.ACME.LAB,HDP-DATA05.ACME.LAB

One last thing worth mentioning about the deployment script is that it does group the cluster nodes and make the changes to the node in parallel for each group. This parallel grouping is especially important when using the script to manage dependency software installations.

Deploying this solution involves two processes. Firstly copying the necessary files to the correct Hadoop folder on each node, and secondly making the necessary modifications to the core-site.xml configuration file.

If the necessary deployment files are copied to any folder on a node in the cluster (easily managed from a central location) an additional PowerShell file can be used to manage the actual deployment on the cluster node:

param([string] $working_path, [string] $script_filename, [string] $files_list)
 
function ListSplitUnique($values)
{
    return ,@((($values -Split ",") | % {$_.Trim()} | where-object {$_ -ne ""} ) | Sort-Object -Unique)
}
 
# Define the Hadoop files and paths
$hadoop_home = resolve-path $env:HADOOP_HOME
$hadoop_bin = "$hadoop_home\bin"
$hadoop_conf = resolve-path $env:HADOOP_CONF_DIR
$hadoop_conf_file = "$hadoop_conf\core-site.xml"
 
$topology_property_name = "topology.script.file.name"
$topology_property_value = $script_filename
 
# First copy the files to the Hadoop Bin directory
# Convert $files_list to an array
$all_files = @()
if ($files_list -ne $null) {
    $all_files = ListSplitUnique($files_list + ", $script_filename")
}
 
$all_files | % { Copy-Item -Path $working_path\$_ -Destination $hadoop_bin -Force }
 
# Next Update the core-site-topology.xml file
# With topology.script.file.name Property
$conf_doc = [System.Xml.XmlDocument](Get-Content $hadoop_conf_file)
 
$topo_node = ($conf_doc.DocumentElement.property | Where-Object {$_.name -eq $topology_property_name})
If ($topo_node) {
    # Element found so ensure the property is correctly set
    write-host "$topology_property_name Element Found, so updating value..."
    $topo_node.Value = $topology_property_value
} else {
    # No Element found so add a new one to the document
    write-host "$topology_property_name Element Not Present, adding new element..."
    $rack_property = $conf_doc.CreateElement("property")
 
    $rack_property_name = $conf_doc.CreateElement("name")
    $rack_property_name.AppendChild($conf_doc.CreateTextNode($topology_property_name))
    $rack_property.AppendChild($rack_property_name)
 
    $rack_property_value = $conf_doc.CreateElement("value")
    $rack_property_value.AppendChild($conf_doc.CreateTextNode($topology_property_value))
    $rack_property.AppendChild($rack_property_value)
 
    $conf_doc.DocumentElement.AppendChild($rack_property)
}
 
$conf_doc.Save($hadoop_conf_file)

This PowerShell script basically takes in the name of the topology.script.file.name property and a list of supporting files. It then copies these files to the necessary Hadoop directory and makes the necessary Xml configuration file changes.

The Xml file configuration changes are smart enough that if a previous topology.script.file.name property has been created then it will just update the current value with the newly specified one.

Calling this script can be achieved through a simple CMD file:

set working_path=%~dp0
if %working_path:~-1%==\ set working_path=%working_path:~0,-1%
 
set script_path=%working_path%\hadoop-rack-configuration-deployment.ps1
set files_list=hadoop-rack-configuration.cmd,hadoop-rack-configuration.ps1,hadoop-rack-configurations.txt
set script_filename=hadoop-rack-configuration.cmd
 
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& '%script_path%' -working_path '%working_path%' -script_filename '%script_filename%' -files_list '%files_list%'
 
pause

Hopefully these script will make deploying the necessary rack awareness assets to the cluster a much simpler and error free process.

Hadoop divides the data into multiple file blocks and stores them on different machines. By default all machines are deemed to be on the same rack, and thus if rack awareness is not configured there the possibility that all replicated copies of a block end up in same rack. Although rare, this could result in data loss when that rack fails. This can be avoided by explicitly configuring Rack Awareness.

Rack Awareness is implemented using the property “topology.script.file.name” in the core-site.xml configuration file. The script accepts a list of IP addresses and return the corresponding list of rack identifiers.

If this “topology.script.file.name” is not configured the default value “/default-rack” is returned for any IP address; thus all nodes considered to be on the same rack.

This sample implementation uses PowerShell to relate a machine name to a predefined rack identifier. One can of course implement any logic one needs. The implementation consists of three files:

• hadoop-rack-configuration.cmd – Stub CMD file called by Hadoop
• hadoop-rack-configuration.ps1 – PowerShell file called by the CMD stub
• hadoop-rack-configurations.txt – Mapping file from machine names to rack identifiers

Each of these files will now be defined.

hadoop-rack-configuration.cmd

The stub CMD file that calls a PS1 file, as follows:

@echo off
 
set cmd_args=%*
set root_dir=%HADOOP_HOME%\bin
 
set script_path=%root_dir%\hadoop-rack-configuration.ps1
set config_path=%root_dir%\hadoop-rack-configurations.txt
 
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& '%script_path%' -config_path '%config_path%' %cmd_args%"

This file acts as a stub to pass in the IP addresses and a mapping file, which in this case is a mapping from machine name to rack, to the PowerShell file that implements the necessary mapping logic.

The @echo off commands are needed to ensure that the output is limited to the write-output commands.

hadoop-rack-configurations.txt

This sample implements a mapping of IP address to a rack identifier using the actual machine name through a mapping file. A sample file line would be:

HDP-DATA01.hdp.lab	/hdp/rack1
HDP-DATA02.hdp.lab	/hdp/rack2
HDP-DATA03.hdp.lab	/hdp/rack1
HDP-DATA04.hdp.lab	/hdp/rack2

The machine name and rack identifier are just separated with a TAB. In this implementation the machines are part of a domain “hdp.lab” and as such fully qualified names are used in the mapping.

hadoop-rack-configuration.ps1

The PowerShell file can then use the mapping file to map the IP address to a rack definition.

A sample PowerShell file that uses the configuration mapping file would be as follows:

param([string] $config_path ) # followed by $cmd_args
 
$listofIPs = $args | % {$_.Trim()};
$configurationFile = $config_path;
 
$DEFAULT_RACK = "/default/rack";
 
function Get-LookupTable {
 
    $hash = New-Object Collections.Hashtable([StringComparer]::InvariantCultureIgnoreCase);
 
    if (Test-Path $configurationFile) {
        Get-Content $configurationFile | ForEach-Object { $split = $_ -Split "\s+|\t+"; $hash[$split[0].Trim()] = $split[1].Trim() };
        Trap { Continue; }
    }
 
    return $hash;
}
 
function Get-ComputerNameByIP {
    param($ipAddress = $null)
    
    begin { }
 
    process {
        if (($ipAddress -and $_) -or (!$ipAddress -and !$_)) {
            throw "Please use either pipeline or input parameter";
            break;
        } else {
            $result = $null;
            $lookup = if ($ipAddress) { $ipAddress.Trim(); } else { $_.Trim(); }
            if ($lookup -as [ipaddress]) {
                $result = [System.Net.Dns]::GetHostbyAddress($lookup);
                Trap { Continue; }
            }
            if ($result) 
            { 
                [string]$result.HostName;
            } 
            else
            {
                $lookup;
            }
        }
    }
 
    end  { }
}
 
function Get-ComputerRack
{
    begin
    {
        $lookupTable = Get-LookupTable;  
    }
 
    process
    {
        $lookup = ($_).trim();
 
        if ($lookupTable.ContainsKey($lookup)) {
            $lookupTable[$lookup];
        } else {
            $DEFAULT_RACK;
        }
    }
 
    end { }
}
 
 
$listofNames = $listofIPs | Get-ComputerNameByIP;
$listofRacks = $listofNames | Get-ComputerRack
write-output ($listofRacks -join " ")

Of course this does mean one can implement any logic you desire now in a PS1 file.

Deploying

Once you have written your PowerShell implementation the final step is to deploy to you cluster.

As mentioned, the core-site.xml is updated to include this additional node:

<property>
	<name>topology.script.file.name</name>
	<value>hadoop-rack-configuration.cmd</value>
</property>

Finally the files need to be deployed to the cluster. One has to copy the CMD and PS1 files to the %HADOOOP_BIN_PATH% directory.

Once deployed the nodes can then query the script to find out their non-default rack identifier.

The HDInsight cmdlets no longer exist as these have now been integrated into the latest release of the Windows Azure Powershell Tools. This integration means:

• You don’t need to specify Subscription parameter
• If needed, you can use AAD authentication to Azure instead of certificates

Also, the cmdlets are fully backwards compatible meaning you don’t need to change your current scripts. The Subscription parameter is now optional but if it specified then it is honoured.

As such the cluster creation script will now be:

The only changes are the selection of the subscription and the removal of the Subscription option when creating the cluster. Of course all the other scripts can easily be modified along the same lines; such as the cluster deletion script:

This cluster creation script also contains an additional section to ensure that the default storage container does not container any leftover files, from previous cluster creations:

The rationale behind this is to ensure that any files that may be left over from previous clusters creations/deletions are removed.

Before provisioning a cluster one need to ensure the Azure subscription has been correctly configured and a management station has been setup.

Managing Your HDInsight Cluster

Prepare Azure Subscription Storage

Under the subscription in which the HDInsight is going to be running one needs a storage account defined, and a management certificate uploaded.

Create a Storage Account

Next, under the storage account management create a new default container:

At this point the Azure subscription is ready.

Setting Up Your Environment

Once the Azure subscription has been configured, the next step is to configure your management environment.

The first step is to install and configure Windows Azure PowerShell. Windows Azure PowerShell is the scripting environment that allows you to control and automate the deployment and management of your workloads in Windows Azure. This includes provisioning virtual machines, setting up virtual networks and cross-premises networks, and managing cloud services in Windows Azure.

To use the cmdlets in Windows Azure PowerShell you will need to download and import the modules, as well as import and configure information that provides connectivity to Windows Azure through your subscription. For instructions, see Get Started with Windows Azure Cmdlets.

Prepare the Work Environment

The basic process for running the cmdlets by using the Windows Azure PowerShell command shell is as follows:

·         Upload a Management Certificate. For more information about how to create and upload a management certificate, see How to: Manage Management Certificates in Windows Azure and the section entitled “Create a Certificate”.

·         Install the Windows Azure module for Windows PowerShell. The install can be found on the Windows Azure Downloads page, under Windows.

·         Finally set the Windows PowerShell execution policy. To achieve this run the Windows Azure PowerShell as an Administrator and execute the following command:

Set-ExecutionPolicy RemoteSigned

The Windows PowerShell execution policy determines the conditions under which configuration files and scripts are run. The Windows Azure PowerShell cmdlets need the execution policy set to a value of RemoteSigned, Unrestricted, or Bypass.

Create a Certificate

From a Visual Studio Command Prompt run the following:

makecert -sky exchange -r -n "CN=<CertificateName>" -pe -a sha1 -len 2048 -ss My "<CertificateName>.cer"

Where< CertificateName> is the name that you want to use for the certificate. It must have a .cer extension. The command loads the private key into your user store, indicated by the “-ss My” switch. In the certmgr.msc, it appears in the path of Certificates -Current User\Personal\Certificates.

Upload the new certification to Azure:

If one needs to connect from several machines the same certificate should be installed into the user store on that machine also.

Configure Connectivity

There are two ways one can configure connectivity between your workstation and Windows Azure. Either manually by configuring the management certificate and subscription details with the Set-AzureSubscription and Select-AzureSubscription cmdlets, or automatically by downloading the PublishSettings file from Windows Azure and importing it. The settings for Windows Azure PowerShell are stored in: <user>\AppData\Roaming\Windows Azure PowerShell.

The PublishSettings file method works well when you are responsible for a limited number of subscriptions, but it adds a certificate to any subscription that you are an administrator or co-administrator for. This can be achieved by:

·         Execute the following command:

Get-AzurePublishSettingsFile

·         Save the .publishSettings file locally.

·         Import the downloaded .publishSettings file by running the PowerShell command:

Import-AzurePublishSettingsFile< publishSettings-file>

Security note: This file contains an encoded management certificate that will serve as your credentials to administer all aspects of your subscriptions and related services. Store this file in a secure location, or delete it after you follow these steps.

In complex or shared development environments, it might be desirable to manually configure the publish settings and subscription information, including any management certificates, for your Windows Azure subscriptions. This is achieved by running the following script:

$mySubID = "<subscritionID>"

$certThumbprint = "<Thumbprint>"

$myCert = Get-Item cert:\CurrentUser\My\$certThumbprint

$mySubName = "<SubscriptionName>"

$myStoreAcct = "<StorageAccount>"

Set-AzureSubscription -SubscriptionName $mySubName -Certificate $myCert -SubscriptionID $mySubID

Select-AzureSubscription -Default $mySubName

Set-AzureSubscription –SubscriptionName $mySubName -CurrentStorageAccount $myStoreAcct

You can view the details of the Azure Subscription run the command:

Get-AzureSubscription –Default

Or:

Get-AzureSubscription -Current

This script uses the Set-AzureSubscription cmdlet. This cmdlet can be used to configure a default Windows Azure subscription, set a default storage account, or a custom Windows Azure service endpoint:

Set-AzureSubscription -DefaultSubscription $mySubName

You can also just select a subscription for the current session using:

Select-AzureSubscription -SubscriptionName $mySubName

When managing your cluster and you have multiple subscription it is advisable to set the default subscription.

Install HDInsight cmdlets

Ensure the HDInsight binaries are accessible and loaded. They can be downloaded from:

http://www.microsoft.com/en-gb/download/details.aspx?id=40724 

Once downloaded and installed they are ready to be loaded into Azure PowerShell environment:

Import-Module "C:\Program Files (x86)\Windows Azure HDInsight PowerShell\Microsoft.WindowsAzure.Management.HDInsight.Cmdlet\Microsoft.WindowsAzure.Management.HDInsight.Cmdlet.dll"

Assuming the Windows Azure HDInsight PowerShell Tools were installed to the default directory.

Cluster Provisioning

There are various ways one can provision a new cluster. If using the Management Portal there is a simple UX process once can follow:

However for an automated process it is recommended that PowerShell is utilized. In running the script to be presented one has to remember load the HDInsight management binaries, as outlined in the “Install HDInsight cmdlets” step.

The following sections present a series of scripts for automating cluster management. Further examples can also be found at Microsoft .Net SDK for Hadoop - PowerShell Cmdlets for Cluster Management.

Provision Base Cluster

Firstly one need access to the Subscription Id and Certificate, for the current Azure Subscription, which was registered in the “Setting Up Your Environment” step.

When running these scripts the italicized variables will first need to be defined.

1.    In this version you need to explicitly specify subscription information for cmdlets. The plan is after the cmdlets are integrated with Azure PowerShell tools this step won’t be necessary. Once can get the Default or Current subscription:

    $subscriptionInfo = Get-AzureSubscription -Default

2.    Once the Subscription Information has been obtained the properties of the Subscription are accessible:

$subName = $subscriptionInfo | %{ $_.SubscriptionName }

$subId = $subscriptionInfo | %{ $_.SubscriptionId }

$cert = $subscriptionInfo | %{ $_.Certificate }

$storeAccount = $subscriptionInfo | %{ $_.CurrentStorageAccountName }

3.    Get the Storage Account Key so you can use it later.

$key = Get-AzureStorageKey $storeAccount | %{ $_.Primary }

4.    Next get the Storage Account information:

$storageAccountInfo = Get-AzureStorageAccount $storeAccount

$location = $storageAccountInfo | %{ $_.Location }

5.    Define the PS Credentials to be used when creating the cluster:

$hadoopUsername = "Hadoop"

$clusterUsername = "Admin"

$clusterPassword = "MyPassword"

$secpasswd = ConvertTo-SecureString $clusterPassword -AsPlainText -Force

$clusterCreds = New-Object System.Management.Automation.PSCredential($clusterUsername, $secpasswd)

6.    Define the final cluster creation parameters:

$clusterName = $Cluster

$numberNodes = $Hosts

$containerDefault = $StorageContainer

$blobStorage = "$storeAccount.blob.core.windows.net"

7.    Finally create new cluster. The cmdlet will return Cluster object in about 10-15 minutes when cluster is finished provisioning:

New-AzureHDInsightCluster -Subscription $subId -Certificate $cert -Name $clusterName -Location $location -DefaultStorageAccountName $blobStorage -DefaultStorageAccountKey $key -DefaultStorageContainerName $containerDefault -Credential $clusterCreds -ClusterSizeInNodes $numberNodes

Once the cluster has been created you will see the results displayed on the screen:

Name               : carlhdinsighttest

ConnectionUrl     : https://carlhdinsighttest.azurehdinsight.net

State              : Running

CreateDate         : 07/11/2013 14:16:37

UserName           : Username

Location           : Location

ClusterSizeInNodes : 4

You will also be able to see the cluster in the management portal:

If the subscription one is managing is not the configured default, before creating the cluster, one should execute the following command:

Set-AzureSubscription -DefaultSubscription $mySubName

To create a script that enables one to pass in parameters, such as the number of required hosts, you can define a Param statement as the first line of the script:

Param($Hosts = 4, [string] $Cluster = $(throw "Cluster Name Required."), [string] $StorageContainer = "hadooproot")

The complete script would thus be as follows:

The script can then be executed as follows:

. "C:\Scripts\ClusterCreate.ps1" -Hosts 4 -Cluster "carlhdinsighttest" -StorageContainer "hadooproot"

Of course other variables can also be passed in.

Manage Cluster

One can view your current clusters within the current subscription using the following command:

Get-AzureHDInsightCluster -Subscription $subId -Certificate $cert

To view the details of a specific cluster use the following command:

Get-AzureHDInsightCluster $clusterName -Subscription $subId -Certificate $cert

To delete the cluster run the command:

Remove-AzureHDInsightCluster $clusterName -Subscription $subId -Certificate $cert

Thus a script one could use to delete clusters would be:

The script can then be executed as follows:

. "C:\Scripts\ClusterDelete.ps1" -Cluster "carlhdinsighttest"

There are few word that need to be mentioned around versioning and cluster name.

When a cluster is currently provisioned you will always get the latest build. However if there are breaking changes between your current build and latest build currently executing code could break. Future versions of the cluster provisioning may allow for a version number to be specified.

Also, when you delete a cluster there is no guarantee that when you recreate the cluster the name will be available. To ensure that your name is not going to be used abstract cluster names should be used.

Provision Customized Cluster

You can also provision cluster and configure it to connect to more than one Azure Blob storage or custom Hive and Oozie meta-stores.

This advanced feature allows you to separate the lifetime of your data and metadata from the lifetime of the cluster. This not only includes job submission details but also allows one to retain Hive metadata, and hence table definitions, as long as Hive External tables are used.

Using the same principal of defining the italicized variables first, the first step is to get the subscription details and the storage key of the storage accounts you want to connect your cluster to, in this case 2 storage accounts:

$subscriptionInfo = Get-AzureSubscription -Default

$subName = $subscriptionInfo | %{ $_.SubscriptionName }

$subId = $subscriptionInfo | %{ $_.SubscriptionId }

$cert = $subscriptionInfo | %{ $_.Certificate }

$storeAccount = $subscriptionInfo | %{ $_.CurrentStorageAccountName }

$storeAccountBck = $storeAccount + "bck" # Assume same name with bck suffix

$key = Get-AzureStorageKey $storeAccount | %{ $_.Primary }

$keyBck = Get-AzureStorageKey $storeAccountBck | %{ $_.Primary }

$storageAccountInfo = Get-AzureStorageAccount $storeAccount

$location = $storageAccountInfo | %{ $_.Location }

$hadoopUsername = "Hadoop"

$clusterUsername = "Admin"    

$clusterPassword = "Password"

$secpasswd = ConvertTo-SecureString $clusterPassword -AsPlainText -Force

$clusterCreds = New-Object System.Management.Automation.PSCredential($clusterUsername, $secpasswd)

$clusterName = $Cluster

$numberNodes = $Hosts

$containerDefault = $StorageContainer

$blobStorage = "$storeAccount.blob.core.windows.net"

$blobStorageBck = "$storeAccountBck.blob.core.windows.net"

In this instance I have assumed that the secondary (backup) storage account has the same name as the original with a “bck” prefix. This however can be configured for your environment.

The next step is to create your custom SQL Azure Database and Server following this blog post or get credentials to the existing one.

To create a SQL Database one can easily use the management portal:

Using this connectivity information you can create cluster config object, pipe in additional storage and meta-stores configuration, and then finally pipe this config object into the New-AzureHDInsightCluster cmdlet to create a cluster based on this custom configuration.

New-AzureHDInsightClusterConfig -ClusterSizeInNodes $numberNodes

  | Set-AzureHDInsightDefaultStorage -StorageAccountName $blobStorage -StorageAccountKey $key -StorageContainerName $containerDefault

  | Add-AzureHDInsightStorage -StorageAccountName $blobStorageBck -StorageAccountKey $keyBck

  | Add-AzureHDInsightMetastore -SqlAzureServerName $sqlServer -DatabaseName $databaseOozie -Credential $databaseCreds -MetastoreType OozieMetastore

  | Add-AzureHDInsightMetastore -SqlAzureServerName $sqlServer -DatabaseName $databaseHive -Credential $databaseCreds -MetastoreType HiveMetastore

  | New-AzureHDInsightCluster -Subscription $subId -Certificate $cert -Credential $clusterCreds -Name $clusterName -Location $location

In this instance I have again assumed that the Oozie and Hive database servers are the same and that the credentials are shared. S before this can easily be configured for your environment.

As before one can use the PowerShell Param option to create a script that enables passing in parameters; such as number hosts, cluster name, etc.

A complete configurable script would be as follows:

The script can then be executed as follows:

. "C:\Scripts\ClusterCreateCustom.ps1" -Hosts 4 -Cluster "carlhdinsighttest" -StorageContainer "hadooproot"

Of course other variables can also be passed in.

Job Submission

In addition to managing the HDInsight cluster the Microsoft .Net SDK for Hadoop also allows one to write MapReduce jobs utilizing the .Net Framework. Under the covers this is employing the Hadoop Streaming interface. The documentation for creating such jobs can be found at:

http://hadoopsdk.codeplex.com/wikipage?title=Map%2fReduce

MRRunner Support

To submit such jobs there is a command-line utility called MRRunner that should be utilized. To support the MRRunner utility you should have an assembly (a .net DLL or EXE) that defines at least one implementation of HadoopJob<>.

If the Dll contains only one implementation of Hadoop<>, you can run the job with:

MRRunner -dll MyDll

If the Dll contains multiple implementations of HadoopJob<>, then you need to indicate the one you wish to run:

MRRunner -dll MyDll -class MyClass

To supply options to your job, pass them as trailing arguments on the command-line, after a double-hyphen:

MRRunner -dll MyDll -class MyClass -- extraArg1 extraArg2

These additional arguments are provided to your job via a context object that is available to all methods on HadoopJob<>.

Local PowerShell Submission

Of course one can submit job directly from a .Net executable and thus PowerShell scripts.

MapReduceJob is a property on Hadoop connection object. It provides access to implementation of IStreamingJobExecutor interface which handles the creation and execution of Hadoop Streaming jobs. Depending on how the connection object was created it can execute jobs locally using Hadoop command-line or against remote cluster using WebHCat client library.

Under the covers the MRRunner utility will invoke LocalStreamingJobExecutor implementation on your behalf.

Alternatively one can invoke the executor directly and request it execute a Hadoop Job:

      var hadoop = Hadoop.Connect();

     hadoop.MapReduceJob.ExecuteJob<JobType>(arguments);

Utilizing this approach one can thus integrate Job Submission into cluster management scripts:

The only challenge becomes managing the generic objects within PowerShell.

In this case I have used a version of the WordCount sample that comes with the SDK. I have removed the Driver entry point and compiled the assembly into a DLL rather than an EXE.

Remote PowerShell Submission

Of course the ultimate goal is to submit the job from your management console. The process for this is very similar to the Local submission, with the inclusion of the connection information for the Azure cluster.

To provide flexibility in the job configuration allowing for a configuration object to be defined in the PowerShell script and for the mappers and reducers to be defined is desirable. This the full submission script would be as follows:

The HadoopJobConfiguration allows for the definition of input and output files, along with other job parameters.

To execute the job one has to provide the types for the Mapper, Reducer, and Combiner. If a Reducer or Combiner are not needed then the parameter value of $Null can be used:

. "C:\Scripts\JobSubmissionAzure.ps1" -Cluster "carlhdinsighttest" -StorageContainer "hadooproot" -InputPath "/wordcount/input" -OutputFolder "/wordcount/output" -Dll "C:\Projects\WordCountSampleApplication\ \Debug\WordCount.dll" -Mapper "WordCountSampleApplication.WordCountMapper" -Reducer "WordCountSampleApplication.WordCountReducer" -Combiner "WordCountSampleApplication.WordCountReducer"

One has to remember in this case as we are submitting the job to an Azure cluster the default file system will be WASB.

To run this sample I uploaded some text files to the appropriate Azure storage location:

The output then will show up in the wordcount/output container.

After running this job, from within the management portal, you should see the metrics for the number of mapper and reducers executed be updated: 

Unfortunately at this point in time you will not see your job within the job history when managing the cluster. For this you will have to use the standard Hadoop UI interfaces; accessible by connecting to the head node of the cluster.

As an alternative to his approach one could define a HadoopJob class configuration within the code; in addition to the mappers and reducers, such as:

class WordCount : HadoopJob<WordCountMapper, WordCountReducer, WordCountReducer>

To execute this job one would just need to define the type for the HadoopJob, which provides the necessary configuration details:

[Type] $jobType = $submitterAssembly.GetType("WordCountSampleApplication.WordCount", 1)

...

$result = $job.Execute($jobType)

The code in either case is very similar but the former approach is probably more flexible.

Specifying WASB Paths

As mentioned the default schema when submitting MapReduce jobs is Azure Blob Storage. This means the default container specified when creating the cluster is used as the root.

In the example above the path “/wordcount/input” actually equates to:

wasb://hadooproot@mystorageaccount.blob.core.windows.net/wordcount/input

Thus if one wanted to operate out of a different default container such as “data” fully-qualified names could be specified such as:

-InputPath "wasb://data@ mystorageaccount.blob.core.windows.net/books"

-OutputFolder "wasb://data@mystorageaccount.blob.core.windows.net/wordcount/output"

This would equate to the “data” container:

The containers being referenced just need to be under the same storage account.

Elastic MapReduce Submissions

The idea behind an Elastic Service is that the cluster can be brought up, with the necessary hosts, when job execution is necessary. To achieve this, with the provided scripts, one can use the following approach:

To execute the script one just has to execute:

. "C:\Scripts\JobSubmissionElastic.ps1" –Hosts 4 -Cluster "carlhdinsighttest" -StorageContainer = "hadooproot" -InputPath "/wordcount/input" -OutputFolder "/wordcount/output" -Dll "C:\Projects\WordCountSampleApplication\ \Debug\WordCount.dll" -Mapper "WordCountSampleApplication.WordCountMapper" -Reducer "WordCountSampleApplication.WordCountReducer" -Combiner "WordCountSampleApplication.WordCountReducer"

This will create the cluster, run the required job, and then delete the cluster. During the creation the cluster name and number of hosts is specified, and during the job submission the input and output paths are specified.

One could of course roll all the script elements into a single reusable script, as follows:

In this instance the cluster password is randomly generated, as it only needs to exist for the lifetime of the job execution. In reality one could also randomly generate the name of the cluster.

Of course one could also customize these scripts to including additional parameters such as additional job arguments, etc. The script presented here does take into consideration the number of mappers; specified with the MapTasks parameter.

This Elastic Job Submission process is possible because the storage used for input and output is Azure Blob Storage. As such the cluster is only needed for compute operations and not storage. In addition the job history is persisted, across cluster creations, in the specified SQL databases.

Of course other parameters could be included in this model to make the PowerShell script more applicable to more job types; such as number of mappers, additional job arguments, etc.

References

Windows Azure PowerShell

Get Started with Windows Azure Cmdlets

Microsoft .Net SDK for Hadoop - PowerShell Cmdlets for Cluster Management

How to: Manage Management Certificates in Windows Azure

Windows Azure Downloads

HDInsight cmdlets

Windows Azure SQL Database Management with PowerShell

Microsoft .Net SDK for Hadoop - Map/Reduce

Conclusion

This document hopefully explains how best to manage an HDInsight cluster and submit .Net MapReduce jobs using a management console and Windows PowerShell. The ultimate goal was to provide an approach one could use for managing Elastic Job Submissions where the cluster only exists for the lifetime of the job.

This Elastic approach to job submissions is possible when using Windows Azure Storage Blobs for data retention and Windows Azure SQL Database for job history. Configuring the cluster such that there is a separate of compute and storage processes allows for the cluster to exist when and only when it is needed for computational processing.

•                    Split the LOB values into multiple rows

•                    Store the LOB values in a regular non-memory optimized (file based) table

This document will provide a mechanism for implementing the split of LOB values into multiple rows, and outline several mechanisms for retrieving the data.

Associated with this post is a set of scripts covering all the TSQL code presented, along with some simple testing scripts, which can be downloaded from here.

Approach Overview

The approach to be taken will allow for the processing of LOB data such that when the data length is greater than a predefined upper limit, segment size, it is broken down into smaller chunks.

If the data fits into a single segment then the process will default to operating on just a primary table with a single row for the key identifier. In this instance reading the data means just returning the data solely from the primary table.

If the data is too large for a single segment then it will be broken down into multiple segments and each part saved as a row with an associated part number; linked to the primary element via the key identifier. In this instance, reading the data then becomes a determination of how many segments the data is stored in, and either concatenating them into a single field or return the individual parts through a SELECT statement.

The procedures to support this approach would be as follows:

·         Insert a LOB column when the data length is less than a predefined segment size

·         Insert a LOB column when the data length is greater than the predefined segment size and a (max) column definition is needed

·         Provide Upsert operations that allow an entry to be made after an insert has previously been made; or to insert data if the current state is not known

·         Provide a SELECT process that returns the collection of LOB segments; taking into consideration the fact the data may not be segmented

·         Read a LOB column when the size is less than a predefined segment size

·         Read a LOB column when the size is greater than the predefined segment size, returning the data in a (max) datatype definition

When implementing your own solution consideration should be given to the predefined segment size. The goal should be such that this size is as small as possible but big enough that a large percentage of the applications data can possibly fit into a single segment. For small transient data, such as session data and small encrypted data elements, adjusting this value will impact the overall performance.

The other added advantage of having the data fit into a single segment is that the data handling Stored Procedures can all be natively compiled. In this case the application would need to know the size of the data it is processing, which may or may not be possible.

Of course, if you are saving MBs of data this optimization will not be feasible so the size should probably be set to 8000 bytes.

However, the primary approach outlined may not be suitable for storing very large quantities of LOB data which are accessed infrequently; say document or images. In this instance a standard file based table approach may be more suitable. However for highly transient data such as session data, encrypted authentication codes, etc. using this approach can dramatically speed up access.

Implementation

For this implementation I have created a new database, called [MemBinaryStore]. In reality the object definitions below would be placed into your own application database:

CREATE DATABASE [MemBinaryStore]

     CONTAINMENT = NONE        

     ON PRIMARY (NAME = [MemBinaryStore_Primary], FILENAME = 'C:\MSSQL\DATA\MemBinaryStore_Prim.mdf', SIZE = 10 MB, FILEGROWTH = 10 MB),

     FILEGROUP [MemBinaryStore_Mod] CONTAINS MEMORY_OPTIMIZED_DATA (NAME = [MemBinaryStore_Mod], FILENAME = 'C:\MSSQL\DATA\MemBinaryStore.dir'),

     FILEGROUP [MemBinaryStore_Data] (NAME = [MemBinaryStore_Data1], FILENAME = 'C:\MSSQL\DATA\MemBinaryStore_Data1.ndf', SIZE = 10 MB, FILEGROWTH = 10 MB),

           (NAME = [MemBinaryStore_Data2], FILENAME = 'C:\MSSQL\DATA\MemBinaryStore_Data2.ndf', SIZE = 10 MB, FILEGROWTH = 10 MB)

     LOG ON (NAME = [MemBinaryStore_Log1], FILENAME = 'C:\MSSQL\DATA\MemBinaryStore_Log1.ldf', SIZE = 5 MB, FILEGROWTH = 5 MB)

     COLLATE Latin1_General_100_CI_AS;

GO

ALTER DATABASE [MemBinaryStore]

    MODIFY FILEGROUP [MemBinaryStore_Data] DEFAULT;

GO

One must remember Memory Optimized tables have a restriction on the code page for (var)char columns. These columns must use code page 1252, hence the reason the collation has been specified.

Table Objects

This implementation uses two base tables. A Primary table that holds the base binary information and also all the necessary data when the LOB data can fit into a single segment. An Extended table that holds any necessary LOB data segments; multiple records existing for each primary entry when the data is split into segments.

CREATE TABLE [dbo].[BinaryItem]

(

     [BinaryItemId]             binary(16)      NOT NULL            

          CONSTRAINT [PK_BinaryItem] PRIMARY KEY NONCLUSTERED HASH

           WITH (BUCKET_COUNT = 1000000),

     [ItemCreated]              datetime        NOT NULL,

     [ItemAccessed]             datetime        NOT NULL,

     [BinaryData]               varbinary(6000) NULL,

     [BinaryExParts]            int             NOT NULL

)

WITH (MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_AND_DATA);

GO

CREATE TABLE [dbo].[BinaryItemEx]

(

     [BinaryItemId]             binary(16)      NOT NULL

           INDEX IDX_BINARYITEMEX_ID HASH

           WITH (BUCKET_COUNT = 1000000),

     [BinaryPart]               int             NOT NULL,

     [BinaryDataEx]             varbinary(6000) NOT NULL,

          CONSTRAINT [PK_BinaryItemEx] PRIMARY KEY NONCLUSTERED HASH

           ([BinaryItemId], [BinaryPart])

           WITH (BUCKET_COUNT = 20000000)

)

WITH (MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_AND_DATA);

GO

When defining your tables care should be taken in defining the correct bucket count. As a very rough guideline, the bucket count should be set to 1-2X the maximum expected cardinality of the table.

In this instance I have chosen a key type of binary(16), rather than a varchar() or uniqueidentifier datatype. This is mostly to avoid specifying a binary collation that is needed for the primary key and index specifications; such as “nvarchar(34) collate Latin1_General_100_BIN2”.

This collation requirement is necessary because indexes on (n)(var)char columns can only be specified with BIN2 collations.

Write Procedures

The process for inserting or updating LOB items is dependent on the length of the LOB data. Calling application should determine what procedure to call based on the size of the data being passed in.

If the data is less than the selected segment size then it is just a simple INSERT operation, which can be performed using a natively compiled Stored Procedure:

CREATE PROCEDURE [dbo].[InsertBinaryItem]

(

    @BinaryItemId          uniqueidentifier,

    @BinaryData            varbinary(6000)

)

WITH NATIVE_COMPILATION, SCHEMABINDING, EXECUTE AS OWNER

AS

BEGIN ATOMIC WITH

(

     TRANSACTION ISOLATION LEVEL = SNAPSHOT,

     LANGUAGE = N'English'

)

     DECLARE @now datetime = GETUTCDATE();

     DECLARE @binaryId binary(16) = CAST(@BinaryItemId AS binary(16));

     INSERT INTO [dbo].[BinaryItem]

     ([BinaryItemId], [ItemCreated], [ItemAccessed], [BinaryData], [BinaryExParts])

     VALUES(@binaryId, @now, @now, @BinaryData, 0);

END

You can of course define the primary key to be whatever datatype is applicable for your application.

In the instance that the data length is larger than the defined segment size you have to pass in the data using a (max) datatype definition. In this case the procedure will have to chunk the data down and INSERT it into the extensions table:

CREATE PROCEDURE [dbo].[InsertBinaryItemEx]

(

     @BinaryItemId         uniqueidentifier,

     @BinaryData                varbinary(max)

)

AS

BEGIN

    DECLARE @now datetime = GETUTCDATE();

     DECLARE @binaryId binary(16) = CAST(@BinaryItemId AS binary(16));

     DECLARE @maxSegmentLength int = 6000;

    DECLARE @binaryDataLength int = DATALENGTH(@BinaryData);

     DECLARE @maxSegment int = CEILING(@binaryDataLength / 1.0 / @maxSegmentLength);

     DECLARE @segmentValue varbinary(6000);

     -- Insert the header primary

     INSERT INTO [dbo].[BinaryItem] WITH (SNAPSHOT)

           ([BinaryItemId], [ItemCreated], [ItemAccessed], [BinaryData], [BinaryExParts])

           VALUES(@binaryId, @now, @now, NULL, @maxSegment);

     -- Now insert the chunked binary data

     DECLARE @part int = 1;

     WHILE (@part <= @maxSegment)

     BEGIN

           SET @segmentValue = SUBSTRING(@BinaryData, ((@part -1) * @maxSegmentLength + 1), @maxSegmentLength);

           INSERT [dbo].[BinaryItemEx] WITH (SNAPSHOT)

                ([BinaryItemId], [BinaryPart], [BinaryDataEx])

                VALUES (@binaryId, @part, @segmentValue);

           SET @part += 1;

     END   

END

In this case the binary header element is still written but the binary block is NULL. Also the part count, [BinaryExParts], is defined as the number of segments the data has been split into.

Breaking down the LOB column is merely a simple case of using the SUBSTRING, which can be used on binary data, to define each segment.

For the INSERT/UPDATE, upsert operations, the process follows the same pattern. In the case of the data being less than the selected segment size an UPDATE operation is first performed. If no entry is found then an INSERT operation is performed:

CREATE PROCEDURE [dbo].[UpsertBinaryItem]

(

     @BinaryItemId         uniqueidentifier,

     @BinaryData                varbinary(6000)

)

WITH NATIVE_COMPILATION, SCHEMABINDING, EXECUTE AS OWNER

AS

BEGIN ATOMIC WITH

(

     TRANSACTION ISOLATION LEVEL = SNAPSHOT,

     LANGUAGE = N'English'

)

     DECLARE @now datetime = GETUTCDATE();

     DECLARE @binaryId binary(16) = CAST(@BinaryItemId AS binary(16));

     DECLARE @itemIdFound binary(16) = NULL;

     -- Delete any extended data

     DELETE FROM [dbo].[BinaryItemEx]

     WHERE [BinaryItemId] = @binaryId;

     -- First try an update as the primary may already exist

     UPDATE [dbo].[BinaryItem]

     SET  @itemIdFound = [BinaryItemId],

           [ItemAccessed] = @now,

           [BinaryData] = @BinaryData,

           [BinaryExParts] = 0

     WHERE [BinaryItemId] = @binaryId;

     -- If no primary found the perform the insert

     IF (@itemIdFound IS NULL)

     BEGIN

           INSERT INTO [dbo].[BinaryItem]

           ([BinaryItemId], [ItemCreated], [ItemAccessed], [BinaryData], [BinaryExParts])

           VALUES(@binaryId, @now, @now, @BinaryData, 0);

     END

END

In this case one has to remember to ensure that any spurious extended binary segments are first deleted before the new values are upserted.

For the case of the data being larger than the selected segment size the process is very similar, except that the data is chunked as in the case of an insert operation:

CREATE PROCEDURE [dbo].[UpsertBinaryItemEx]

(

     @BinaryItemId         uniqueidentifier,

     @BinaryData           varbinary(max)

)

AS

BEGIN

     DECLARE @now datetime = GETUTCDATE();

     DECLARE @binaryId binary(16) = CAST(@BinaryItemId AS binary(16));

     DECLARE @itemIdFound binary(16) = NULL;

     DECLARE @maxSegmentLength int = 6000;

     DECLARE @binaryDataLength int = DATALENGTH(@BinaryData);

     DECLARE @maxSegment int = CEILING(@binaryDataLength / 1.0 / @maxSegmentLength);

     DECLARE @segmentValue varbinary(6000);

     IF (@binaryDataLength <= 6000)

     BEGIN

           SET @segmentValue = SUBSTRING(@BinaryData, 1, @binaryDataLength);

           EXEC [dbo].[UpsertBinaryItem] @BinaryItemId, @segmentValue

     END

     ELSE

     BEGIN

           -- Delete any existing binary segments

           DELETE FROM [dbo].[BinaryItemEx] WITH (SNAPSHOT)

           WHERE [BinaryItemId] = @binaryId;

           -- Update the header

           UPDATE [dbo].[BinaryItem] WITH (SNAPSHOT)

           SET  @itemIdFound = [BinaryItemId],

                [ItemAccessed] = @now,

                [BinaryData] = NULL,

                [BinaryExParts] = @maxSegment

           WHERE [BinaryItemId] = @binaryId;

           -- If no header found then create one

           IF (@itemIdFound IS NULL)

           BEGIN

                INSERT INTO [dbo].[BinaryItem] WITH (SNAPSHOT)

                      ([BinaryItemId], [ItemCreated], [ItemAccessed], [BinaryData], [BinaryExParts])

                      VALUES(@binaryId, @now, @now, NULL, @maxSegment);

           END

           -- Insert the binary segments

           DECLARE @part int = 1;

           WHILE (@part <= @maxSegment)

           BEGIN

                SET @segmentValue = SUBSTRING(@BinaryData, ((@part -1) * @maxSegmentLength + 1), @maxSegmentLength);

                INSERT [dbo].[BinaryItemEx] WITH (SNAPSHOT)

                      ([BinaryItemId], [BinaryPart], [BinaryDataEx])

                      VALUES (@binaryId, @part, @segmentValue);

                SET @part += 1;

           END                       

     END

END

As in the case of the insert operations, it is only for when the data is less than the selected segment size that the Stored Procedures can be natively compiled. As such, ensuring that a large portion of the data fits into this selected segment size will give the best performance gains.         

Access Procedures

The most efficient way to access the binary data is just to return binary segments, through a SELECT statement, and have the application layer combine them back into a single element. The process to do this has to look at the primary element for the specified number of elements and return the corresponding data:

CREATE PROCEDURE [dbo].[GetBinaryDataParts]

(

     @BinaryItemId         uniqueidentifier

)

WITH NATIVE_COMPILATION, SCHEMABINDING, EXECUTE AS OWNER

AS

BEGIN ATOMIC WITH

(

     TRANSACTION ISOLATION LEVEL = SNAPSHOT,

     LANGUAGE = N'English'

)

     DECLARE @now datetime = GETUTCDATE();

     DECLARE @binaryId binary(16) = CAST(@BinaryItemId AS binary(16));

     DECLARE @binaryData varbinary(6000);

     DECLARE @binaryParts int = 0;

     -- Get the binary data and see if parts exists

     UPDATE [dbo].[BinaryItem]

     SET  [ItemAccessed] = @now,

           @binaryData = [BinaryData],

           @binaryParts = [BinaryExParts]

     WHERE [BinaryItemId] = @binaryId;

     -- If parts exists then return these otherwise just return the single element

    IF (@binaryData IS NULL AND @binaryParts > 0)

     BEGIN

           SELECT CAST([BinaryItemId] AS uniqueidentifier) AS [BinaryItemId], [BinaryPart], [BinaryDataEx] AS [BinaryData]

           FROM [dbo].[BinaryItemEx]

           WHERE [BinaryItemId] = @binaryId

           ORDER BY [BinaryItemId], [BinaryPart];

     END

     ELSE

     BEGIN

           SELECT @BinaryItemId AS [BinaryItemId], 0 AS [BinaryPart], @binaryData AS [BinaryData];

     END

END

The beauty of this approach is that the called Stored Procedure can be natively compiled.

Also, the application calling this Stored Procedure then has the option of better handling the necessary memory allocations for concatenating the LOB segments into a single element.

The approach I have taken here is to update the primary element with the last accessed time. This would allow for elements to be removed based on a defined life expectancy. If this is not needed one could just perform a SELECT of the primary data, rather than an update.

If a SELECT process is not optimal as in the case that the application requires the data to be concatenated one can use a Stored Procedure with an OUTPUT field.

In the advent that the application knows that data size is less than the specified segment size there is the option of directly calling a natively compiled Stored Procedure:

CREATE PROCEDURE [dbo].[GetBinaryData]

(

     @BinaryItemId         uniqueidentifier,

     @BinaryData           varbinary(6000) OUTPUT

)

WITH NATIVE_COMPILATION, SCHEMABINDING, EXECUTE AS OWNER

AS

BEGIN ATOMIC WITH

(

     TRANSACTION ISOLATION LEVEL = SNAPSHOT,

     LANGUAGE = N'English'

)

     DECLARE @now datetime = GETUTCDATE();

     DECLARE @binaryId binary(16) = CAST(@BinaryItemId AS binary(16));

     -- Get the binary data for output

     UPDATE [dbo].[BinaryItem]

     SET  [ItemAccessed] = @now,

           @BinaryData = [BinaryData]

     WHERE [BinaryItemId] = @binaryId;

END

In the case that the data size is unknown, or it is larger than the specified segment size, the Stored Procedure has to perform the concatenation into a max datatype, and thus cannot be natively compiled:

CREATE PROCEDURE [dbo].[GetBinaryDataEx]

(

     @BinaryItemId         uniqueidentifier,

     @BinaryData           varbinary(max) OUTPUT

)

AS

BEGIN

     DECLARE @now datetime = GETUTCDATE();

     DECLARE @binaryId binary(16) = CAST(@BinaryItemId AS binary(16));

     DECLARE @maxSegmentLength int = 6000;

     DECLARE @blobSegment varbinary(6000);

     DECLARE @binaryParts int;

     DECLARE @count int = 0;

     SET @BinaryData = NULL;

     -- Get the binary primary and see if parts exists

     UPDATE [dbo].[BinaryItem] WITH (SNAPSHOT)

     SET  [ItemAccessed] = @now,

           @BinaryData = [BinaryData],

           @binaryParts = [BinaryExParts]

     WHERE [BinaryItemId] = @binaryId;

     -- If parts exist then build the binary element

     IF (@BinaryData IS NULL AND @binaryParts > 0)

     BEGIN

           WHILE (@count <= @binaryParts)

           BEGIN

                SELECT @blobSegment = [BinaryDataEx]

                FROM [dbo].[BinaryItemEx] WITH (SNAPSHOT)

                WHERE [BinaryItemId] = @binaryId AND [BinaryPart] = @count;

                IF (@BinaryData IS NULL)

                      SET @BinaryData = CAST(@blobSegment AS varbinary(max));

                ELSE

                      SET @BinaryData = @BinaryData + CAST(@blobSegment AS varbinary(max));

                SET @count += 1;

           END

     END

END

In this case the LOB concatenation is performing simply using the plus operator. Of course this concatenation is only necessary if extended binary segments are found.

From the performance perspective it is probably better to take the original SELECT approach and have the application layer handle the necessary data concatenation; as it can better handle the memory allocations necessary to perform this operation.

Alternative Considerations

Error Checking and Retries

The Stored Procedures outlined above do not take into consideration error checking and retry logic. The approach to either can be that it is the responsibility of the calling application or of the actual Stored Procedures.

The associated scripts included with this post also include a version of the Stored Procedures that perform both error checking and retry logic. As an example here is the [UpsertBinaryItem] Stored Procedure:

CREATE PROCEDURE [dbo].[UpsertBinaryItem]

(

    @BinaryItemId     uniqueidentifier,

    @BinaryData       varbinary(6000)

)

WITH NATIVE_COMPILATION, SCHEMABINDING, EXECUTE AS OWNER

AS

BEGIN ATOMIC WITH

(

     TRANSACTION ISOLATION LEVEL = SNAPSHOT,

     LANGUAGE = N'English'

)

     DECLARE @retry int = 3;

     DECLARE @now datetime = GETUTCDATE();

     DECLARE @binaryId binary(16) = CAST(@BinaryItemId AS binary(16));

     DECLARE @itemIdFound binary(16) = NULL;

     WHILE (@retry > 0)

     BEGIN

           BEGIN TRY

                -- Delete any extended data

                DELETE FROM [dbo].[BinaryItemEx]

                WHERE [BinaryItemId] = @binaryId;

                -- First try an update as the primary may already exist

                UPDATE [dbo].[BinaryItem]

                SET  @itemIdFound = [BinaryItemId],

                      [ItemAccessed] = @now,

                      [BinaryData] = @BinaryData,

                      [BinaryExParts] = 0

                WHERE [BinaryItemId] = @binaryId;

                -- If no primary found the perform the insert

                IF (@itemIdFound IS NULL)

                BEGIN

                      INSERT INTO [dbo].[BinaryItem] ([BinaryItemId], [ItemCreated], [ItemAccessed], [BinaryData], [BinaryExParts])

                      VALUES(@binaryId, @now, @now, @BinaryData, 0);

                END

                SET @retry = 0;

           END TRY

           BEGIN CATCH

                SET @retry -= 1;

                IF (@retry <= 0 OR (error_number() <> 41302 AND error_number() <> 41305 AND error_number() <> 41325 AND error_number() <> 41301))

                BEGIN

                      -- insert custom error handling for other error conditions here

                      ;THROW

                END

           END CATCH

     END

END

This code covers the common errors associated with using an optimistic concurrency control mechanism, as used by the In Memory OLTP engine.

In this case I have set the retry count to be 3, but this can easily be configured to suit your application and environment requirements.

Durability

In the samples above I have defined the tables to have a durability option of SCHEMA_AND_DATA. In the case of transient session data one could again improve performance using just the SCHEMA_ONLY option.

File Based Extension

In addition to taking the approach outlined above there is the object that if the data size is larger than the specified segment size a normal file based table can be used for saving the LOB data. In this case the table definitions would be more like the following:

CREATE TABLE [dbo].[BinaryItem]

(

     [BinaryItemId]              binary(16)     NOT NULL            

           CONSTRAINT [PK_BinaryItem] PRIMARY KEY NONCLUSTERED HASH

           WITH (BUCKET_COUNT = 1000000),

     [ItemCreated]              datetime        NOT NULL,

     [BinaryData]               varbinary(6000) NULL,

     [IsExtended]               int             NOT NULL