This blog describe a basic console application written in C# to track Performance Counters in Windows.

Counters are used in windows OS to provide information as to how well the operating system or an application, service, or driver is performing. The counter data can help determine system bottlenecks. The operating system, network, and devices provide counter data that an application can consume to provide user with a graphical view of how well the system is performing.

The System.Diagnostics namespace provides classes that allow you to interact with performance counters. The PerformanceCounter Class has different constructors; in our code we use the constructor with the following format:

public PerformanceCounter( 
 string categoryName, 
 string counterName, 
 string instanceName 
 ) 

where:

• categoryName: it is the name of the performance counter category (performance object) with which this performance counter is associated.
• counterName: The name of the performance counter.
• instanceName: The name of the performance counter category instance, or an empty string (""), if the category contains a single instance.

This constructor initializes the performance counter and associates the instance with an existing counter on the local computer. The values that you pass in for the CategoryName, CounterName, and InstanceName properties must point to an existing performance counter on the local computer.

Below some performance counters to track processor utilization, disk I/O, Memory, and networking

PerformanceCounter("Processor", "% Processor Time", "_Total");
PerformanceCounter("Processor", "% Privileged Time", "_Total");
PerformanceCounter("Processor", "% Interrupt Time", "_Total");
PerformanceCounter("Processor", "% DPC Time", "_Total");
PerformanceCounter("Memory", "Available MBytes", null);
PerformanceCounter("Memory", "Committed Bytes", null);
PerformanceCounter("Memory", "Commit Limit", null);
PerformanceCounter("Memory", "% Committed Bytes In Use", null);
PerformanceCounter("Memory", "Pool Paged Bytes", null);
PerformanceCounter("Memory", "Pool Nonpaged Bytes", null);
PerformanceCounter("Memory", "Cache Bytes", null);
PerformanceCounter("Paging File", "% Usage", "_Total");
PerformanceCounter("PhysicalDisk", "Avg. Disk Queue Length", "_Total");
PerformanceCounter("PhysicalDisk", "Disk Read Bytes/sec", "_Total");
PerformanceCounter("PhysicalDisk", "Disk Write Bytes/sec", "_Total");
PerformanceCounter("PhysicalDisk", "Avg. Disk sec/Read", "_Total");
PerformanceCounter("PhysicalDisk", "Avg. Disk sec/Write", "_Total");
PerformanceCounter("PhysicalDisk", "% Disk Time", "_Total");
PerformanceCounter("Process", "Handle Count", "_Total");
PerformanceCounter("Process", "Thread Count", "_Total");
PerformanceCounter("System", "Context Switches/sec", null);
PerformanceCounter("System", "System Calls/sec", null);
PerformanceCounter("System", "Processor Queue Length", null);

2 Performance counters CategoryName

Here a short description of some performance counters.

2.1 CategoryName: Processor

 PerformanceCounter("Processor", "% Processor Time", "_Total");

The Processor\% Processor Time counter determines the percentage of time the processor is busy by measuring the percentage of time the thread of the Idle process is running and then subtracting that from 100 percent. This measurement is the amount of processor utilization

PerformanceCounter("Processor", "% Interrupt Time", "_Total");

The rate, in average number of interrupts in incidents per second, at which the processor received and serviced hardware interrupts. It does not include deferred procedure calls, which are counted separately.

PerformanceCounter("Processor", "% DPC Time", "_Total");

The percentage of time that the processor spent receiving and servicing deferred procedure calls during the sample interval. Deferred procedure calls are interrupts that run at a lower priority than standard interrupts.

PerformanceCounter("Processor", "% Privileged Time", "_Total");

The percentage of non-idle processor time spent in privileged mode. Privileged mode is a processing mode designed for operating system components and hardware-manipulating drivers. It allows direct access to hardware and all memory. The alternative, user mode, is a restricted processing mode designed for applications, environment subsystems, and integral subsystems. The operating system switches application threads to privileged mode to gain access to operating system services. This includes time spent servicing interrupts and deferred procedure calls (DPCs). A high rate of privileged time might be caused by a large number of interrupts generated by a failing device. This counter displays the average busy time as a percentage of the sample time.

2.2 CategoryName: Memory

PerformanceCounter("Memory", "Available MBytes", null);

This measures the amount of physical memory, in megabytes, available for running processes. If this value is less than 5 percent of the total physical RAM, that means there is insufficient memory, and that can increase paging activity.

 PerformanceCounter("Memory", "Committed Bytes", null);

it shows the amount of virtual memory, in bytes, that can be committed without having to extend the paging file(s). Committed memory is physical memory which has space reserved on the disk paging files. There can be one or more paging files on each physical drive. If the paging file(s) are expanded, this limit increases accordingly.

PerformanceCounter("Memory", "Commit Limit", null);

it shows the amount of virtual memory, in bytes, that can be committed without having to extend the paging file(s). Committed memory is physical memory which has space reserved on the disk paging files. There can be one or more paging files on each physical drive. If the paging file(s) are expanded, this limit increases accordingly.

PerformanceCounter("Memory", "% Committed Bytes In Use", null);

it shows the ratio of Memory\ Committed Bytes to the Memory\ Commit Limit. Committed memory is physical memory in use for which space has been reserved in the paging file so that it can be written to disk. The commit limit is determined by the size of the paging file. If the paging file is enlarged, the commit limit increases, and the ratio is reduced.

PerformanceCounter("Memory", "Pool Paged Bytes", null);

it shows the size, in bytes, of the paged pool. Memory\ Pool Paged Bytes is calculated differently than Process\ Pool Paged Bytes, so it might not equal Process(_Total )\ Pool Paged Bytes.

PerformanceCounter("Memory", "Pool Nonpaged Bytes", null);

it shows the size, in bytes, of the nonpaged pool. Memory\ Pool Nonpaged Bytes is calculated differently than Process\ Pool Nonpaged Bytes, so it might not equal Process(_Total )\ Pool Nonpaged Bytes.

PerformanceCounter("Memory", "Cache Bytes", null);

it shows the sum of the values of System Cache Resident Bytes, System Driver Resident Bytes, System Code Resident Bytes, and Pool Paged Resident Bytes.

2.3 CateroryName: PhysicalDisk

PerformanceCounter("PhysicalDisk", "Disk Read Bytes/sec", "_Total");

PerformanceCounter("PhysicalDisk", "Disk Write Bytes/sec", "_Total");

it captures the total number of bytes sent to the disk (write) and retrieved from the disk (read) during write or read operations.

PerformanceCounter("PhysicalDisk", "Avg. Disk sec/Read", "_Total");

PerformanceCounter("PhysicalDisk", "Avg. Disk sec/Write", "_Total");

it captures the average time, in seconds, of a read/write of data from/to the disk.

PerformanceCounter("System", "Context Switches/sec", null);

A context switch occurs when the kernel switches the processor from one thread to another—for example, when a thread with a higher priority than the running thread becomes ready. Context switching activity is important for several reasons. A program that monopolizes the processor lowers the rate of context switches because it does not allow much processor time for the other processes' threads. A high rate of context switching means that the processor is being shared repeatedly—for example, by many threads of equal priority. A high context-switch rate often indicates that there are too many threads competing for the processors on the system. The System\Context Switches/sec counter reports systemwide context switches.

2.4 CategoryName: Process

Each process provides the resources needed to execute a program. A process has a virtual address space, executable code, open handles to system objects, a security context, a unique process identifier, environment variables, a priority class, minimum and maximum working set sizes, and at least one thread of execution. Each process is started with a single thread, often called the primary thread, but can create additional threads from any of its threads.

A thread is the entity within a process that can be scheduled for execution. All threads of a process share its virtual address space and system resources. In addition, each thread maintains exception handlers, a scheduling priority, thread local storage, a unique thread identifier, and a set of structures the system will use to save the thread context until it is scheduled. The thread context includes the thread's set of machine registers, the kernel stack, a thread environment block, and a user stack in the address space of the thread's process. Threads can also have their own security context, which can be used for impersonating clients. (for more information  see "About Processes and Thread" http://msdn.microsoft.com/en-us/library/windows/desktop/ms681917%28v=vs.85%29.aspx)

PerformanceCounter("Process", "Thread Count", "_Total");

The number of threads created by the process. This counter does not indicate which threads are busy and which are idle. It displays the last observed value, not an average.

PerformanceCounter("Process", "Handle Count", "_Total");

the value reports the number of handles that processes opened for objects they create. Handles are used by programs to identify resources that they must access. The value of this counter tends to rise during a memory leak.

PerformanceCounter("Process", "Thread Count", "_Total");

the value reports over time how many threads the processes create.

2.5 CategoryName: System

PerformanceCounter("System", "Context Switches/sec", null);

A context switch occurs when the kernel switches the processor from one thread to another. A context switch might also occur when a thread with a higher priority than the running thread becomes ready or when a running thread must wait for some reason (such as an I/O operation). The Thread\Context Switches/sec counter value increases when the thread gets or loses the time of the processor.

PerformanceCounter("System", "System Calls/sec", null);

This is the number of system calls being serviced by the CPU per second. By comparing the Processor's Interrupts/sec with the System Calls/sec we can get a picture of how much effort the system requires to respond to attached hardware. On a healthy system, the Interrupts per second should be negligible in comparison to the number of System Calls per second. When the system has to repeatedly call interrupts for service, it's indicative of a hardware failure.

PerformanceCounter("System", "Processor Queue Length", null);

The System\Processor Queue Length counter is a rough indicator of the number of threads each processor is servicing.

3 Structure of the application to track Performance Counters

The basic structure of the application to track a set of PerfomanceCounter() is reported in figure:

Figure 1: main structure of application

The main purpose of application is catch a specific sample interval a list of Performance Counters.

When the main program starts, the first action is to read two XML files: SysField.XML file and app.config file. Behavior of application depends on parameter values stored in these two files.

Structured of SysField.XML is shown below:

<?xml version="1.0" encoding="utf-8" ?>
 <AppValues>
   <FolderPath>C:\SysParamsLog2\</FolderPath>
   <FileName>Log.txt</FileName>
   <WriteToLogFile>true</WriteToLogFile>
   <MaxNumberRecordsInLogFile>30</MaxNumberRecordsInLogFile>
   <PollingSysParamsInterval>1</PollingSysParamsInterval>
 </AppValues>

SysField.XML file specifies all the parameters required to store the PerformanceCounter() values in a log file and the sampling time of performance counters. In particular:

   <FolderPath>     : it is the folder where the application store the log file
 <FileName>       : it is the filename fo the log file
 <WriteToLogFile> : it is a Boolean value that specify
 <MaxNumberRecordsInLogFile> : it is the max number of records stored in a single log file
 <PollingSysParamsInterval>  : it is the sampling interval of PerformanceCounter()

To generate a log file per day, we need to set the MaxNumberRecordsInLogFile equal to:

MaxNumberRecordsInLogFile =(Second in a day)/ PollingSysParamsInterval

i.e.

if we sample every 1 second (PollingSysParamsInterval =1),  MaxNumberRecordsInLogFile =86400secondi/ (1 second)=86400

Using the parameters stored in app.config is possible to define which counters parameters we want to be include in the sampling and the structure of the performance counters (order).

The full structure of app.config is reported below:

 <?xml version="1.0"?>
<configuration>
  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5"/>
  </startup>
  <appSettings>
    <add key="SamplingTime" value="true" />
    <add key="NodeName" value="false" />
    <add key="CPUProcessorTime" value="true" />
    <add key="CPUPrivilegedTime" value="true" />
    <add key="CPUInterruptTime" value="true" />
    <add key="CPUDPCTime" value="true" />
    <add key="MEMAvailable" value="true" />
    <add key="MEMCommited" value="true" />
    <add key="MEMCommitLimit" value="true" />
    <add key="MEMCommitedPerc" value="true" />
    <add key="MEMPoolPaged" value="true" />
    <add key="MEMPoolNonPaged" value="true" />
    <add key="MEMCached" value="true" />
    <add key="PageFile" value="true" />
    <add key="ProcessorQueueLengh" value="true" />
    <add key="DISCQueueLengh" value="true" />
    <add key="DISKRead" value="true" />
    <add key="DISKWrite" value="true" />
    <add key="DISKAverageTimeRead" value="true" />
    <add key="DISKAverageTimeWrite" value="true" />
    <add key="DISKTime" value="true" />
    <add key="HANDLECountCounter" value="true" />
    <add key="THREADCount" value="true" />
    <add key="CONTENTSwitches" value="true" />
    <add key="SYSTEMCalls" value="true" />
    <add key="NumProcess" value="true" />
    <add key="NetTrafficSend" value="true" />
    <add key="NetTrafficReceive" value="true" />   
  </appSettings>
</configuration>

The meaning of key value is self-explanatory, and the value associated to every key is boolean:

• the value equal to “true” means that value of the performance counter is tracked
• the value equal to “false” means that value of the performance counter can be skipped   

When you compile your application, Visual Studio will automatically create a file called <your application name>.exe.config in your bin\debug folder. The contents of the app.config will be automatically copied to this new config file when you compile the application. When you deliver the application to the end user, you have to deliver the exe and this new config file called <your application name>.exe.config and NOT the app.config. Users can modify the data in <your application name>.exe.config file and application will read the data from the config file, when restarted. Change the order of (key,value) in app.config determines a different order of performance counter print out to the console and in log file.

The project use a public class named SysParams to store all the Performance counters.

public class SysParams

{

public string NodeName { get; set; }

public float CPUProcessorTime { get; set; }

public float CPUPrivilegedTime { get; set; }

public float CPUInterruptTime { get; set; }

public float CPUDPCTime { get; set; }

public float MEMAvailable { get; set; }

public float MEMCommited { get; set; }

public float MEMCommitLimit { get; set; }

public float MEMCommitedPerc { get; set; }

public float MEMPoolPaged { get; set; }

publicfloat MEMPoolNonPaged { get; set; }

public float MEMCached { get; set; }

public float PageFile { get; set; }

public float ProcessorQueueLengh { get; set; }

public float DISCQueueLengh { get; set; }

public float DISKRead {get; set;}

public float DISKWrite { get; set; }

public float DISKAverageTimeRead { get; set; }

public float DISKAverageTimeWrite { get; set; }

public float DISKTime { get; set; }

public float HANDLECountCounter { get; set; }

public float THREADCount { get; set; }

public int CONTENTSwitches { get; set; }

public int SYSTEMCalls { get; set; }

public float NetTrafficSend { get; set; }

public float NetTrafficReceive { get; set; }

public DateTime SamplingTime { get; set; }

private PerformanceCounter cpuProcessorTime = new PerformanceCounter("Processor", "% Processor Time", "_Total");

private PerformanceCounter cpuPrivilegedTime = new PerformanceCounter("Processor", "% Privileged Time", "_Total");

private PerformanceCounter cpuInterruptTime = new PerformanceCounter("Processor", "% Interrupt Time", "_Total");

private PerformanceCounter cpuDPCTime = new PerformanceCounter("Processor", "% DPC Time", "_Total");

private PerformanceCounter memAvailable = new PerformanceCounter("Memory", "Available MBytes", null);

private PerformanceCounter memCommited = new PerformanceCounter("Memory", "Committed Bytes", null);

private PerformanceCounter memCommitLimit = new PerformanceCounter("Memory", "Commit Limit", null);

private PerformanceCounter memCommitedPerc = new PerformanceCounter("Memory", "% Committed Bytes In Use", null);

private PerformanceCounter memPollPaged = new PerformanceCounter("Memory", "Pool Paged Bytes", null);

private PerformanceCounter memPollNonPaged = new PerformanceCounter("Memory", "Pool Nonpaged Bytes", null);

private PerformanceCounter memCached = new PerformanceCounter("Memory", "Cache Bytes", null);

private PerformanceCounter pageFile = new PerformanceCounter("Paging File", "% Usage", "_Total");

private PerformanceCounter processorQueueLengh = new PerformanceCounter("System", "Processor Queue Length", null);

private PerformanceCounter diskQueueLengh = new PerformanceCounter("PhysicalDisk", "Avg. Disk Queue Length", "_Total");

private PerformanceCounter diskRead = new PerformanceCounter("PhysicalDisk", "Disk Read Bytes/sec", "_Total");

private PerformanceCounter diskWrite = new PerformanceCounter("PhysicalDisk", "Disk Write Bytes/sec", "_Total");

private PerformanceCounter diskAverageTimeRead = new PerformanceCounter("PhysicalDisk", "Avg. Disk sec/Read", "_Total");

private PerformanceCounter diskAverageTimeWrite = new PerformanceCounter("PhysicalDisk", "Avg. Disk sec/Write", "_Total");

private PerformanceCounter diskTime = new PerformanceCounter("PhysicalDisk", "% Disk Time", "_Total");

private PerformanceCounter handleCountCounter = new PerformanceCounter("Process", "Handle Count", "_Total");

private PerformanceCounter threadCount = new PerformanceCounter("Process", "Thread Count", "_Total");

private PerformanceCounter contentSwitches = new PerformanceCounter("System", "Context Switches/sec", null);

private PerformanceCounter systemCalls = new PerformanceCounter("System", "System Calls/sec", null);

private PerformanceCounterCategory performanceNetCounterCategory;

private PerformanceCounter[] trafficSentCounters;

private PerformanceCounter[] trafficReceivedCounters;

private string[] interfaces = null;

public void initNetCounters()

{

// PerformanceCounter(CategoryName,CounterName,InstanceName)

performanceNetCounterCategory = newPerformanceCounterCategory("Network Interface");

interfaces = performanceNetCounterCategory.GetInstanceNames();

int length = interfaces.Length;

if (length > 0)

{

trafficSentCounters = newPerformanceCounter[length];

trafficReceivedCounters = newPerformanceCounter[length];

}

for (int i = 0; i < length; i++)

{

// Initializes a new, read-only instance of the PerformanceCounter class.

//   1st paramenter: "categoryName"-The name of the performance counter category (performance object) with which this performance counter is associated.

//   2nd paramenter: "CounterName" -The name of the performance counter.

//   3rd paramenter: "instanceName" -The name of the performance counter category instance, or an empty string (""), if the category contains a single instance.

trafficReceivedCounters[i] = new PerformanceCounter("Network Interface", "Bytes Sent/sec", interfaces[i]);

trafficSentCounters[i] =new PerformanceCounter("Network Interface", "Bytes Sent/sec", interfaces[i]);

}

// List of all names of the network interfaces

for (int i = 0; i < length; i++)

{

Console.WriteLine("Name netInterface: {0}", performanceNetCounterCategory.GetInstanceNames()[i]);

}

}

public void getProcessorCpuTime()

{

float tmp = cpuProcessorTime.NextValue();

CPUProcessorTime = (float)(Math.Round((double)tmp, 1));

// Environment.ProcessorCount: return the total number of cores

}

public void getCpuPrivilegedTime()

{

float tmp = cpuPrivilegedTime.NextValue();

CPUPrivilegedTime = (float)(Math.Round((double)tmp, 1));

}

public void getCpuinterruptTime()

{

float tmp = cpuInterruptTime.NextValue();

CPUInterruptTime = (float)(Math.Round((double)tmp, 1));

}

public void getcpuDPCTime()

{

float tmp = cpuDPCTime.NextValue();

CPUDPCTime = (float)(Math.Round((double)tmp, 1));

}

public void getPageFile()

{

PageFile = pageFile.NextValue();

}

public void getProcessorQueueLengh()

{

ProcessorQueueLengh = processorQueueLengh.NextValue();

}

public void getMemAvailable()

{

MEMAvailable = memAvailable.NextValue();

}

public void getMemCommited()

{

MEMCommited = memCommited.NextValue() / (1024 * 1024);

}

public void getMemCommitLimit()

{

MEMCommitLimit = memCommitLimit.NextValue() / (1024 * 1024);

}

public void getMemCommitedPerc()

{

float tmp = memCommitedPerc.NextValue();

// return the value of Memory Commit Limit

MEMCommitedPerc = (float)(Math.Round((double)tmp, 1));

}

public void getMemPoolPaged()

{

float tmp = memPollPaged.NextValue() / (1024 * 1024);

MEMPoolPaged = (float)(Math.Round((double)tmp, 1));

}

public void getMemPoolNonPaged()

{

float tmp = memPollNonPaged.NextValue() / (1024 * 1024);

MEMPoolNonPaged = (float)(Math.Round((double)tmp, 1));

}

public void getMemCachedBytes()

{

// return the value of Memory Cached in MBytes

MEMCached = memCached.NextValue() / (1024 * 1024);

}

public void getDiskQueueLengh()

{

DISCQueueLengh = diskQueueLengh.NextValue();

}

public void getDiskRead()

{

float tmp=diskRead.NextValue()/1024;

DISKRead = (float)(Math.Round((double)tmp,1));

}

public void getDiskWrite()

{

float tmp=diskWrite.NextValue()/1024;

DISKWrite = (float)(Math.Round((double)tmp,1)); // round 1 digit decimal

}

public void getDiskAverageTimeRead()

{

float tmp = diskAverageTimeRead.NextValue() * 1000;

DISKAverageTimeRead = (float)(Math.Round((double)tmp, 1)); // round 1 digit decimal

}

public void getDiskAverageTimeWrite()

{

float tmp = diskAverageTimeWrite.NextValue()*1000;

DISKAverageTimeWrite = (float)(Math.Round((double)tmp, 1)); // round 1 digit decimal

}

public void getDiskTime()

{

float tmp = diskTime.NextValue();

DISKTime = (float)(Math.Round((double)tmp, 1));

}

public void getHandleCountCounter()

{

HANDLECountCounter = handleCountCounter.NextValue();

}

public void getThreadCount()

{

THREADCount = threadCount.NextValue();

}

public void getContentSwitches()

{

CONTENTSwitches = (int)Math.Ceiling(contentSwitches.NextValue());

}

public void getsystemCalls()

{

SYSTEMCalls = (int)Math.Ceiling(systemCalls.NextValue());

}

public void getCurretTrafficSent()

{

int length = interfaces.Length;

float sendSum = 0.0F;

for (int i = 0; i < length; i++)

{

sendSum += trafficSentCounters[i].NextValue();

}

float tmp = 8 * (sendSum / 1024);

NetTrafficSend = (float)(Math.Round((double)tmp, 1));

}

public void getCurretTrafficReceived()

{

int length = interfaces.Length;

float receiveSum = 0.0F;

for (int i = 0; i < length; i++)

{

receiveSum += trafficReceivedCounters[i].NextValue();

}

float tmp = 8 * (receiveSum / 1024);

NetTrafficReceive = (float)(Math.Round((double)tmp, 1));

}

public void getSampleTime()

{

SamplingTime =DateTime.Now;

}

}

A description of program workflow is reported in figure 2 and figure 3

Figure 2: program workflow

Figure 3: program workflow

 You can run the program on your Window 8 laptop or Azure IaaS VM with Windows 2012 R2 (you do not need to install any further package, transfer the list of files under the binary folder and run it).

Here how look like a typical output:

Figure 4: Performance counters parameters on the console

Program has small CPU impact; on Azure IaaS VM –A1 (basic tier) with Windows 2012 R2 the CPU consumption is between 1% and 5% as shown in figure 4 and figure 5.

Figure 5: CPU consumption of program

  Figure 6: CPU consumption of program

To control the record format stored in log file we can change the content in app.config file

3.1 Tracking CPU and Network performance counters only

 To monitor the CPU and network traffic counters is enough to change the app.config file structure:

 <?xml version="1.0"?>
<configuration>
  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5"/>
  </startup>
  <appSettings>
    <add key="SamplingTime" value="true" />
    <add key="NodeName" value="false" />
    <add key="CPUProcessorTime" value="true" />
    <add key="CPUPrivilegedTime" value="true" />
    <add key="CPUInterruptTime" value="true" />
    <add key="CPUDPCTime" value="true" />
    <add key="MEMAvailable" value="false" />
    <add key="MEMCommited" value="false" />
    <add key="MEMCommitLimit" value="false" />
    <add key="MEMCommitedPerc" value="false" />
    <add key="MEMPoolPaged" value="false" />
    <add key="MEMPoolNonPaged" value="false" />
    <add key="MEMCached" value="false" />
    <add key="PageFile" value="false" />
    <add key="ProcessorQueueLengh" value="false" />
    <add key="DISCQueueLengh" value="false" />
    <add key="DISKRead" value="false" />
    <add key="DISKWrite" value="false" />
    <add key="DISKAverageTimeRead" value="false" />
    <add key="DISKAverageTimeWrite" value="false" />
    <add key="DISKTime" value="false" />
    <add key="HANDLECountCounter" value="false" />
    <add key="THREADCount" value="false" />
    <add key="CONTENTSwitches" value="false" />
    <add key="SYSTEMCalls" value="false" />
    <add key="NumProcess" value="false" />
    <add key="NetTrafficSend" value="true" />
    <add key="NetTrafficReceive" value="true" />   
  </appSettings>
</configuration>

 or differently we can cut out all the pairs (key, value) that we do not want to track:

 <?xml version="1.0"?>
<configuration>
  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5"/>
  </startup>
  <appSettings>
    <add key="SamplingTime" value="true" />
    <add key="NodeName" value="false" />
    <add key="CPUProcessorTime" value="true" />
    <add key="CPUPrivilegedTime" value="true" />
    <add key="CPUInterruptTime" value="true" />
    <add key="CPUDPCTime" value="true" />
    <add key="NetTrafficSend" value="true" />
    <add key="NetTrafficReceive" value="true" />   
  </appSettings>
</configuration>

we get a log file look like:

To change the sequence of performance counters in the log file is enough to change the order in app.config

<?xml version="1.0"?>
<configuration>
  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5"/>
  </startup>
  <appSettings>
    <add key="SamplingTime" value="true" />
    <add key="NodeName" value="false" />
    <add key="NetTrafficSend" value="true" />
    <add key="NetTrafficReceive" value="true" />
    <add key="CPUProcessorTime" value="true" />
    <add key="CPUPrivilegedTime" value="true" />
    <add key="CPUInterruptTime" value="true" />
    <add key="CPUDPCTime" value="true" />   
  </appSettings>
</configuration>

To reset the default parameters of app.config you can delete the file or set the content to:

<?xml version="1.0"?>
<configuration>
  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5"/>
  </startup>
  <appSettings>
  </appSettings>
</configuration>

4 Azure Files to store performance counter log of multiple Azure Windows VM

If we want to Tracking Performance Counters on single Azure Windows VM, it is enough to copy the binary in the Azure windows VM and run it.

Azure Files is a great feature that exposes file shares using the standard SMB 2.1 protocol.  As pointed out from the Azure storage product team an Azure File share can also be used to save diagnostic files like logs, metrics, and crash dumps. Multiple Azure VMs can mount the same file share an write the log file in different folders.

The diagram in Figure 7 shows how to use Azure File as repository of Performance counters log files generated from different Azure VMs.

Figure 7: store Performance Counters log files in Azure File

Azure Files can be defined through Azure powershell with simple basic command:

Import-Module Azure

Select-AzureSubscription –SubscriptionName [[SubscriptionName]]
Get-AzureSubscription –current
New-AzureStorageAccount -StorageAccountName [[AzureStorageAccountName]] -Label [[AzureStorageAccountName]] -Location "North Europe"


# To find out the version of the module you're using
(get-module azure).version

$ctx=New-AzureStorageContext [[AzureStorageAccountName]] [[AzureStorageAccountKey]]
$ctx = New-AzureStorageContext -StorageAccountName [[AzureStorageAccountName]] -StorageAccountKey [[AzureStorageAccountKey]]

# create a new share
$s = New-AzureStorageShare [[shareName]] -Context $ctx

In our case we define a share name equal to “logshare1”:

$s = New-AzureStorageShare logshare1 -Context $ctx

To mount the shared folder in the Azure Windows VM we can use the command in both azure Windows VMs:

net use z: \\[[AzureStorageAccountName]].file.core.windows.net\logshare1 /u:[[AzureStorageAccountName]] [[AzureStorageAccountKey]]

This command mount the shared folder “logshare1” of Azure Storage account as driver Z:

To store the log files in different folder we can change the value <FolderPath> in  SysField.XML file 

Content of SysField.XML file in test01 Azure VM:

<?xml version="1.0" encoding="utf-8" ?>
<AppValues>
  <FolderPath>Z:\SysParamsLog1\</FolderPath>
  <FileName>fftest01.txt</FileName>
  <WriteToLogFile>true</WriteToLogFile>
  <MaxNumberRecordsInLogFile>20</MaxNumberRecordsInLogFile>
  <PollingSysParamsInterval>1</PollingSysParamsInterval>
</AppValues>

Content of SysField.XML file in test02 Azure VM:

<?xml version="1.0" encoding="utf-8" ?>
<AppValues>
  <FolderPath>Z:\SysParamsLog2\</FolderPath>
  <FileName>fftest02.txt</FileName>
  <WriteToLogFile>true</WriteToLogFile>
  <MaxNumberRecordsInLogFile>20</MaxNumberRecordsInLogFile>
  <PollingSysParamsInterval>1</PollingSysParamsInterval>
</AppValues>

You can download the SysParams project here:

SysParams.zip

5 Reference

PhysicalDisk Object: http://msdn.microsoft.com/en-us/library/ms804035.aspx

Monitoring Threads: http://technet.microsoft.com/en-us/library/cc776966(v=ws.10).aspx

Memory Object: http://msdn.microsoft.com/en-us/library/ms804008.aspx

http://blogs.technet.com/b/askcore/archive/2012/03/16/windows-performance-monitor-disk-counters-explained.aspx

HPC services for Excel support offloading a Microsoft Excel workbook to a cluster. In a burst-to-Windows Azure cluster with Microsoft HPC Pack 2008 R2 onwards, you can now use Windows Azure worker role instances (worker nodes) for Excel offloading instead Azure IaaS VMs.

This blog walks you through the steps to run an Excel workbook in a burst-to-Windows Azure scenario, where the HPC head node runs on Azure IaaS VM while the Compute Nodes are Windows Azure Worker Roles.

This blog covers the steps for running and Excel Workbook in Azure worker role; in summary the list of actions:

1. Create a VHD which contains un unattended installation of Excel office installation files

2. Upload the VHD to the storage which the burst nodes will deploy into using “hpcpack upload”

    Example:

    hpcpack upload OFFICE.VHD /account:[[storageAccount]] /key:[[storageAccessKey]]

3. Create a cmd to install Excel in unattended mode and copying some HPC DDLs on the worker role

    F:\setup.exe /adminfile F:\Updates\ExcelOnly.msp

    robocopy F:\ E:\approot Microsoft.Hpc.Excel.dll Microsoft.Hpc.Excel.ExcelService.dll Microsoft.Hpc.Excel.ExcelService_1.0.config

4. Create and upload the cmd to storage

    Example: 

    hpcpack create InstallExcel.zip InstallExcel.cmd

    hpcpack upload InstallExcel.zip /account:[[storageAccount]] /key:[[storageAccessKey]]

5. Create Azure Node Template to mount the VHD and set the startup script to install excel when the work role start

6. Add Azure worker roles using the HPC Azure Node Template, the worker role will install excel automatically

7. To run the Excel workbook, you should pack and upload the workbook the Azure storage

    Example:

    hpcpack create xxx.zip xxx.xlsm

    hpcpack upload xxx.zip /account:[[storageAccount]] /key:[[storageAccessKey]]

8. When the Azure nodes are ready, start the workbook in client machine.

Note: In HPC2012 and later releases, you need to add an excel.exe.config in the same folder with excel.exe on client machine

2 Install Windows HPC Head Node on Azure IaaS VM

In Azure image gallery is available an image with Windows 2012 R2 Datacenter OS with embedded Windows HPC PACK 2012 R2 Update1. In this image is also pre-installed Microsoft SQL Server 2014 Express, to host the HPC DBs. This image can be used to create the HPC head node in Azure IaaS VM. Recommended VM size for the HPC is A4, but for testing purpose, you can use smaller VM.

Before creating the HPC head node in Azure IaaS VM it is good practice:

• Define an Azure affinity group (logical group associated with a specific Azure datacenter)
• Create an Azure  Storage Account in the same datacenter where you spin up the HPC head node
• Create an Azure Virtual Network associate with the affinity group defined before.

Steps to create a HPC head node in IaaS VM has been described in the blog.

http://blogs.msdn.com/b/faber/archive/2014/04/13/burst-to-azure-with-microsoft-hpc-pack-a-step-by-step-configuration-guide.aspx

At this point of time, this blog assume that you have the HPC head node up and running. If you deploy an HPC head node using the PACK 2012 R2 Update1 available in the Azure image gallery you might check the version of HPC PACK by the HPC Management console:

Figure 1: PACK 2012 R2 Update1

3 Install Excel on Windows Azure nodes

Running an Excel workbook requires Excel to be installed on each compute node. However, an HPC compute node deployed ad Azure worker role is stateless, which means it could be recycled and redeployed in Windows Azure from time to time, and all settings will be discarded. Fortunately, the HPC Windows Azure Node Template provides a way to mount a VHD and execute a startup script when the worker node is started.

3.1 Create an unattended Excel installer

On your local desktop/laptop you can create an unattended Excel installer. To install Excel in unattended mode follow the official guideline Office Customization Tool.

Unattended Office installer produces a .MSP file; we choice as name of the MSP file “ExcelOnly.msp” and store it in root folder with the Office setup.exe file.

In my blog

http://blogs.msdn.com/b/faber/archive/2013/06/14/running-parallel-excel-on-windows-hpc-2008-r2-hosted-in-windows-azure-iaas-vms.aspx   

I have shown how to create a custom unattended installation of Excel.

1. open the command shell with Administrator right, move in the root folder where Office2010 Administrator credential, run the command: setup /admin
2. Follow the instruction as in the blog and store the outcome file ExcelOnly.msp in root installation directory of Office package.

When the operation is completed, you can create a VHD containing all the installation files along with the “ExcelOnly.msp” file and then upload the VHD file to Windows Azure storage blob.

3.2 Create a VHD file

On your local desktop/laptop, open Disk Management (click Start, type diskmgmt.msc, and then press ENTER; this operation open the Disk Management MMC snap-in).

Figure 2: Disk Management snap-in

Select Create VHD from the Action menu

Figure 3: Create a new VHD

Figure 4: specify the format of VHD file

• Location: This is the location and file name where the new VHD will be saved when it is created
• Virtual hard disk size: This is the size of the VHD; keep the default (Fixed size)

Figure 5: define location and size of VHD

At this stage, the VHD is attached and appears as an uninitialized disk.

Figure 6: Volume before initialization

To initialize the disk, right-click the disk name in the Disk Management pane and click Initialize Disk

Figure 7: initialize the VHD disk

Figure 8: define the partition style of VHD

After initializing the VHD, you can format and assign a drive letters to the volume.

Figure 9: VHD volume after initialization

Right-click on the unallocated space and choice New Simple Volume. Follow the instructions in the New Simple Volume Wizard to select the volume size, drive letter, format, and disk block size.

Figure 10: create a new Simple Volume

Figure 11: define volume size

Figure 12:  assign a drive letter to the volume

Figure 13: specify the file system for the volume

Figure 14: new volume online

3.3 Copy the Office package in the VHD Volume

Copy all the Office installation package (including the file ExcelOnly.msp) inside the VHD Volume.

Figure 15: copying the Office installation package in the VHD volume

3.4 Move the VHD volume offline and detach

The volume needs to be move in offline and therefore detached

Figure 16: Detach the VHD volume

3.5 Copy the .VHD file in Azure storage blob

To copy the OFFICE.VHD file (containing the Office installation package) in Azure storage, you can use AZCOPY command line https://aka.ms/AzCopy:

AzCopy /Source:C:\tmp /Dest:https://<NameStorageAccount>.blob.core.windows.net/excel /DestKey:<AzureStorageKey> /pattern:Office.VHD

 To run the command you need to replace with name and key of your Azure Storage Account:

• <NameStorageAccount> : your Azure Storage account name
• <AzureStorageKey>: primary key of your Azure Storage account

3.6 Move the VHD in the HPC head node an add some .DLL to it

Now that OFFICE.VHD is in Azure Storage account use AZCOPY to transfer the VHD file from the Azure storage blob to the HPC head node. (Note: AZCOPY is already installed by default in the HPC head node, in the folder C:\Program Files (x86)\Microsoft SDKs\Azure\AzCopy ):

AzCopy /Source:https://<NameStorageAccount>.blob.core.windows.net/excel /Dest:C:\Users\toor\Desktop\Office1 /SourceKey:<AzureStorageKey> /pattern:Office.VHD

3.7 Add some .DLL to the VHD file

To run Excel you need to copy some .DLLs in the VHD.

On the HPC head node you can simply mount the OFFICE.VHD with simple click-on from Windows explorer.

Figure 17: double click on the OFFICE.VHD to mount the volume

Figure 18: list of files in OFFICE.VHD mounted on the head node

The OFFICE.VHD is automatically mounted ad driver F:

Here a list of DDL files for Excel to be considered:

%systemroot%\Microsoft.NET\assembly\GAC_MSIL\Microsoft.Hpc.Excel\v4.0_2.0.0.0__31bf3856ad364e35\Microsoft.Hpc.Excel.dll
%ccp_home%bin\Microsoft.Hpc.Excel.ExcelService.dll
%ccp_home%bin\Microsoft.Hpc.Excel.Internal.dll
%ccp_home%bin\Microsoft.Hpc.Excel.tlb
%ccp_home%bin\Microsoft.Hpc.Trace.dll
%ccp_home%bin\CosmosLoggingManaged.dll
%ccp_home%bin\XLL64\Microsoft.Hpc.Excel.XllContainer64.dll
%ccp_home%bin\XLL64\Microsoft.Hpc.Excel.XllConnector.dll
%ccp_home%ServiceRegistration\Microsoft.Hpc.Excel.ExcelService_1.0.config
%ccp_home%ServiceRegistration\Microsoft.Hpc.Excel.XllContainer64_1.2.config

Only three DLLs need to be copied in root of VHD volume:

%systemroot%\Microsoft.NET\assembly\GAC_MSIL\Microsoft.Hpc.Excel\v4.0_2.0.0.0__31bf3856ad364e35\Microsoft.Hpc.Excel.dll
%ccp_home%bin\Microsoft.Hpc.Excel.ExcelService.dll
%ccp_home%ServiceRegistration\Microsoft.Hpc.Excel.ExcelService_1.0.config

To copy the DLLs you might use the robocopy command:

robocopy %systemroot%\Microsoft.NET\assembly\GAC_MSIL\Microsoft.Hpc.Excel\v4.0_2.0.0.0__31bf3856ad364e35\ F:\ Microsoft.Hpc.Excel.dll
robocopy "%ccp_home%bin" F:\ Microsoft.Hpc.Excel.ExcelService.dll 
robocopy "%ccp_home%ServiceRegistration" F:\ Microsoft.Hpc.Excel.ExcelService_1.0.config

Figure 19: list of DLL files copied in OFFICE.VHD

We can now dismount the OFFICE.VHD volume by eject:

Figure 20: Eject the volume OFFICE.VHD

3.8 Create a script to install Excel

You need to create a script to install OFFICE.VHD when the worker nodes start up. If “setup.exe” and “ExcelOnly.msp” are in the root directory of the OFFICE.VHD you can simply use the following command line:

%CCP_APPDRIVE%setup.exe /adminfile %CCP_APPDRIVE%ExcelOnly.msp
robocopy %CCP_APPDRIVE% %CCP_HOME% Microsoft.Hpc.Excel.dll Microsoft.Hpc.Excel.ExcelService.dll Microsoft.Hpc.Excel.ExcelService_1.0.config

 Uses the environment variables %CCP_APPDRIVE% and %CCP_HOME%

Save the above command line to installExcel.cmd.

3.9 Use the hpcpack to package the script

On the HPC head node use the hpcpack command to package the script “installExcel.cmd”:

hpcpack create installExcel.zip installExcel.cmd

3.10 Use the hpcpack to upload the script and VHD file in Azure Storage account

Upload the “installExcel.zip” to the Azure storage account that is used for deploying the Azure nodes:

hpcpack upload installExcel.zip /account:<NameStorageAccount> /key:<AzureStorageKey>

Upload the “OFFICE.VHD” file containing the Office installation package in Azure Storage account

hpcpack upload Office.VHD /account: <NameStorageAccount> /key:<AzureStorageKey>

3.11 Create an Azure Cloud Services to guest HPC worker role

To host the HPC worker role is required a new Azure Cloud Services.

Figure 21: Create a new Cloud Services to host the HPC worker roles

Figure 22: name and affinity associated with the new cloud Services

Figure 22: name and affinity associated with the new cloud Services

Figure 23: new available Azure Cloud Service

3.12 Create an HPC Azure node template to mount the VHD and execute the startup script

Open the HPC management console and create a Windows Azure node template (i.e. assign the name “AzureNodeExcel”). You can just follow the guidance of the Create Node Template Wizard as usual, as long as you specify the same storage account that you used for uploading the Excel installer. When configuring the HPC worker role in the HPC Node Template, you should choose to mount the VHD you just made.

Figure 24: choice “New” in actions pane to create a new Node template

Figure 25: select the Windows Azure Node Template

Figure 26: specify a name for the new Node Template

Figure 27: browse in the list of management certificate

Figure 28: select a digital certificate loaded in the Azure subscription

Figure 29: specify the Azure subscription ID where we want spin up the HPC worker roles

The same digital certificate selected in the Azure node template need to be loaded in the Azure subscription

Figure 30: select the cloud service and storage account used to deploy Azure worker roles

Figure 31: specify the size and number of proxy nodes

Figure 32: select the OFFICE.VHD available in Azure storage and the OS version used to deploy the HPC worker roles

On the Specify startup script page, fill in “installExcel.cmd”.

Figure 33: specify the command to install on the HPC worker roles Excel in unattended mode

Figure 34: specify an optional virtual network where HPC worker roles needs to be attached

Figure 35: specify the administrator credential for the HPC worker roles

Figure 36: select the way to spin-up the Azure worker roles

Figure 37: summary of the new Node Template for the worker roles

Figure 38: Node template used to deploy worker roles with Excel

After Azure Node template is created, you can add HPC worker roles to the HPC cluster using the template.

3.13 Add Node

In the HPC Management Console select Node Management and therefore in Actions pane choice Add Node

Figure 39: add a new Compute Nodes to the HPC cluster

Figure 40: option to add new HPC worker roles to the cluster

Figure 41: select the Azure Node Template, number and size of the worker roles

Figure 42: worker roles appear as un-deployed in list of nodes

Figure 43: click-on Start command to deploy the worker roles with policy define in Azure Node Template

Figure 44: after successful deployment of HPC worker roles the nodes are move in Offline state

Figure 45: list of nodes belonging to the cloud service (proxies and worker roles)

In the table below a successful deployment of an HPC worker role.

Figure 46: Bring the HPC worker roles Online

When the entire HPC node are Online, the cluster is ready to run HPC jobs.

Figure 47: all the HPC nodes are Online

Figure 48: connect to the Azure worker role through Remote Desktop

Figure 49: GUI to connect via RDP to the HPC worker roles

3.14 Install Excel 2010 on the HPC client

In production environment is recommended run HPC head node and HPC client node on different hosts. In our case HPC client runs on the HPC head node.

To install Excel mount the OFFICE.VHD, open a command shell and change the driver (F:)

Run the command for unattended Excel 2010 installation:

After unattended installation, eject the OFFICE.VHD volume

3.15 Create a file in HPC client

For HPC 2012 and later version, you need to add an excel.exe.config in the same folder with excel.exe on HPC client.

The content of excel.exe.config is:

excel.exe file is located by default in the folder: C:\Program Files\Microsoft Office\Office14

Run our Excel workbook without this file causes the following message:

Figure 50: error in Excel worksheet due to missing excel.exe.config file in the HPC client

3.16 Deployment of Excel workbook on the worker role

hpcpack is a tool provided with Windows HPC Server which packages files for staging on Azure worker roles.  You can use it to package HPC services or regular files.  A “package” is just a zip file with some additional metadata used for installation.  The hpcpack tool is used both to create packages and to upload those packages to Azure storage.

When you start Azure worker nodes in the HPC Cluster Manager, any software which has been previously staged – including SOA services and file packages – will be automatically installed on the nodes.  This is done when the nodes are started, using a tool called hpcsync.  Any package which has been created and uploaded using hpcpack will be automatically installed on startup.

If the nodes are already running, using hpcpack you can create or modify a package, and therefore by hpcsync you can ensure that the latest package version is installed on your HPC worker Roles.  We will use hpcpack and hpcsync to move files to the Azure cloud and configure the Azure worker roles.

After you create an Excel workbook to run on the HPC cluster, you should pack it (by the command “hpcpack create”), upload it to Windows Azure storage (by the command “hpcpack upload”), and sync it to all Windows Azure worker roles (by the command “clusrun hpcsync”).

3.16.1 Package the Excel file

Packaging is done on the HPC client. In our setup the HPC client is the head node, but in production you can use a different host join to the AD DC. In our case the Excel workbook is named CP1.xlsb

3.16.2 Transfer the package in Azure Storage

 Command to transfer the package to the Azure Storage

3.16.3 Synchronization between Azure Storage and HPC Worker Roles

To sync all HCP worker roles with Windows Azure Storage runs the command:

Then you can run the Excel workbook in the usual way and the calculation will be distributed on the cluster. Whenever if you modify the workbook, or create a new one, you need to repeat the steps to package, upload to Azure Storage and synchronizing the package from the Azure Store to HPC worker roles.

3.17 Run a basic multiplication table on HPC cluster

The primary benefit of this model is that it requires no extra software (just Excel and the HPC client utilities). The main limitation of this model is that it requires Administrator permissions to move the spreadsheet to the cloud.  That means that regular users won’t be able to install new or modified spreadsheets themselves. Another benefit of this approach is that in many cases it will be the fastest way to use Excel and Azure. Because the spreadsheet is already present on the worker roles, the calculation can start almost immediately – it does not have to copy the spreadsheet before it starts running.

As basic example to run an Excel cluster on HPC, we use the same Excel workbook described in the blog:

http://blogs.msdn.com/b/faber/archive/2013/07/04/running-parallel-excel-on-windows-hpc-2008-r2-hosted-in-windows-azure-iaas-vms-part-2.aspx

 To run the Excel workbook in Azure worker roles, you need to change the VM macro: HPCControlMacros

'==========================================================
 ' This is a skeleton macro file for using the HPC/Excel
 ' VBA macros with an HPC cluster.
 '
 ' Be sure to add references to the required component:
 ' Microsoft_Hpc_Excel
 ' using the VBA editor menu Tools -> References.
 '
 '==========================================================
 Option Explicit
 
 '----------------------------------------------------------
 ' This is the cluster scheduler, or head node. 
 '----------------------------------------------------------
 Private Const HPC_ClusterScheduler = "HEADNODE"
 
 '----------------------------------------------------------
 ' Job Template
 '----------------------------------------------------------
 Private Const Azure_JobTemplate = "Default"
 
 '----------------------------------------------------------
 ' This object is our client for connecting to the HPC
 ' cluster and running calculations.
 '----------------------------------------------------------
 Private HPCExcelClient As IExcelClient
 
 
 '----------------------------------------------------------
 '
 ' This is the main calculation function, which connects
 ' to the client object and runs a calculation. The method
 ' can run both desktop and cluster calculations, controlled
 ' by the function parameter "CalculateOnDesktop".
 '
 ' You'll see below two functions that run calculations
 ' by calling this method, with the parameter set for either
 ' cluster or desktop calculation.
 '
 '----------------------------------------------------------
 Private Sub CalculateWorkbook(CalculateOnDesktop As Boolean)
 
 Dim HPCWorkbookPath As String 
 On Error GoTo ErrorHandler
 
 ' Create a new excelCient instance per session
 Set HPCExcelClient = New ExcelClient
 
 ' Initialize the excel client object with the current workbook
 HPCExcelClient.Initialize ActiveWorkbook
 
 If CalculateOnDesktop = False Then 
 ' Create a cluster session with the desired options. Here, we're
 ' just using the scheduler name and (optionally) a job template. 
 If Azure_JobTemplate <> "" Then
 HPCExcelClient.OpenSession HPC_ClusterScheduler, "Multi.xlsb", 1, 128, SessionUnitType_Core, jobTemplate:=Azure_JobTemplate
 Else
 HPCExcelClient.OpenSession HPC_ClusterScheduler, "Multi.xlsb"
 End If
 End If
 
 ' Run on local machine or cluster as chosen in workbook
 HPCExcelClient.Run CalculateOnDesktop
 Exit Sub
 
 ErrorHandler:
 ' Notify user of error and clean up any allocated resources
 MsgBox Prompt:=Err.Description, Title:="HPC Calculation Error"
 If Not HPCExcelClient Is Nothing Then
 HPCExcelClient.Dispose
 End If
 End Sub
 
 '----------------------------------------------------------
 ' This is a public method for running a calculation on the desktop.
 '----------------------------------------------------------
 Public Sub CalculateWorkbookOnDesktop()
 CalculateWorkbook (True)
 End Sub
 
 '----------------------------------------------------------
 ' This is a public method for running a calculation on the cluster.
 '----------------------------------------------------------
 Public Sub CalculateWorkbookOnCluster()
 CalculateWorkbook (False)
 End Sub
 
 '----------------------------------------------------------
 ' This method is used to clean up a calculation after it's finished; 
 '----------------------------------------------------------
 Public Sub CleanUpClusterCalculation()
 On Error Resume Next
 HPCExcelClient.CloseSession
 HPCExcelClient.Dispose
 On Error GoTo 0
 End Sub
 

Before run the Multi.xlsb file you need to deploy the Excel on the worker roles:

hpcpack create Multi.zip Multi.xlsb
hpcpack upload Multi.zip /account:<AzureStorageAccount> /key:<AzureStorageKey>
lusrun /nodegroup:AzureNodes hpcsync

Figure 52: multiplication table in Excel workbook running on the HPC cluster

You can download the Excel workbook here:

Multi.xlsb 
Table multiplication

3.18 Run Excel bond price model file on HPC cluster

As a second example we can run an HPC cluster an Excel workkook with bond price model. This example is more complex of the simple multiplication table, but has the same structure in HPC VB macro.

You can download the Excel workbook here:

CP1.xlsb 
Bond price model

Figure 53: press the button “Calculate on Cloud” to run on HPC worker roles

Figure 54: job is composed by two tasks (task 1.1 and task 1.2)

Figure 55: every task runs on one HPC worker role

Figure 56: Task1.1 runs on AzureCN-0004

Figure 57: task 1.2 runs on the AzureCN-005

3.19 Description of the Excel workbook

The primary benefit of this model is that it requires no extra software (just Excel and the HPC client utilities).  The main limitation of this model is that it requires Administrator permissions to move the spreadsheet to the cloud.  That means that regular users won’t be able to install new or modified spreadsheets themselves. 

Another benefit of this approach is that in many cases it will be the fastest way to use Excel and Azure.  Because the spreadsheet is already present on the worker roles, the calculation can start almost immediately – it does not have to copy the spreadsheet before it starts running.

3.19.1 Configuring the Example

To run the example, open the spreadsheet file “CP1.xlsb” on your desktop.  Before running on the cloud, we need to make one change to the spreadsheet.  The spreadsheet VBA code connects to the cluster using the name of the cluster head node.  Open the VBA editor (From Excel, press Alt+F11).  In the VBA editor, in the tree view on the top of the left-hand side, double-click the module named “HPCControlMacros” to open it.

In this file, locate the line

Private Const HPC_ClusterScheduler = "HEADNODE"

Change the value of HEADNODE to match your cluster head node.  This should be the machine name. 

The file also specifies the name of the Job Template, which we used above when installing the spreadsheet on the cloud.  The file uses the name we suggested, “Default”.  If you used a different job template name, then find the line

Private Const Azure_jobTemplate = "Default"

and change the value “Default” to match your HPC job template name; remember that this is the job template, not the node template.

3.19.2 Note

In Excel burst scenario, the max allowed length of Excel workbook file name is 39.

The root cause of this issue is Excel file path can’t exceed 218 characters:when you save or open a file if the path to the file (including the file name) exceeds 218 characters an error occurs. This limitation includes three characters representing the drive, the characters in folder names, the backslash character between folders, and the characters in the file name. The file path pattern of the hpcpack is very long, the following is one example:

C:\Resources\Directory\c49f08fea9464edc901ed3c52ae8cb98.HpcWorkerRole1.Microsoft.Hpc.Azure.LocalStorage.Application\convertiblepricing_azurecloud_static\2013-06-04T083747.0000000Z\ConvertiblePricing_AzureCloud_Static.xlsb.

4 Reference

http://technet.microsoft.com/en-us/library/jj899632.aspx

http://technet.microsoft.com/en-us/library/hh162018.aspx

1 Introduction

In the Azure VM gallery are available a list of Linux distributions, including Oracle Linux. One of effective way to access remotely the Linux VM desktop is through VNC (Virtual Network Computing). VNC is used to display an X windows session running on another computer. Unlike a remote X connection, the xserver is running on the remote computer, not on your local workstation. Your workstation (Linux or Windows ) is only displaying a copy of the display that is running on the remote machine. In this note are reported the configuration steps required to connect to the Oracle Linux VM through VNC (Virtual Network Computing). The present note shows up two configurations:

• direct connection to the VNC Server
• tunneling VNC connections over SSH.

2 Create an Oracle Linux IaaS VM

Login in Azure Management portal and create an IaaS VM through the VM gallery

Figure 1: Create a new IaaS VM from the image gallery

 Select the Oracle Linux image available in the VM gallery.

Figure 2: Oracle Linux 6.4.0.0.0 available in Azure image gallery

Define the name of VM, size and administrator credentials (username and password) to access to the Oracle Linux VM.

Figure 3: define IaaS size and administrator credentials

Select the affinity group and storage account to store the .VHD file of the Linux VM

Figure 4: specify an Affinity Group and Azure Storage account

Install the VM agent in the Linux VM.

Figure 5: keep the default option to install the VM Agent

3 Install the desktop package in Oracle Linux

When the Oracle Linux VM is up and running connect to the VM through SSH session.

Figure 6: Oracle Linux VM up and running

To login in the system use the same credential specified in the creation of VM.

It is easy to work with group of packages; there are some useful commands:

 it lists the installed and available groups for your system in two separate lists.

 It provides you detailed information for each group including: description, mandatory, default and optional packages.

When you create an Azure Oracle Linux IaaS VM, by default you do not have GUI and desktop environment.

To install X Window and fonts run the command:

You can ignore the reference to the new Linux kernel uek (unbreakable enterprise kernel)

If you install GNOME desktop, using the command:

 you get the message:

Installation procedure aborts because it tries to install NetworkManager package as part of this install, but the already installed WALinuxAgent is conflicting with NetworkManager package.

To skip the dependent packages and proceed GNOME Desktop installation use --skip-broken as

You can also install the following optional GUI packages.

It is useful to add other packages to our Oracle Linux VM:

• xterm is the standard terminal emulator for the X Window System. A user can have many different invocations of xterm running at once on the same display
• twm is a window manager for the X Window System. It provides titlebars, shaped windows, several forms of icon management, user-defined macro functions, click-to-type and pointer-driven keyboard focus, and user-specified key and pointer button bindings.
• xsetroot program allows you to tailor the appearance of the background ("root") window on a workstation display running X.

You can install optional xorg fonts packages through the command:

4 Install the VNC server in Oracle Linux VM

In this paragraph, you have the procedure to install the VNC server in Oracle Linux VM.

For setting up the VNC server, we perform the following steps:

1. Install the VNC server package
2. Create the VNC users accounts
3. Edit the VNC server configuration
4. Set your users' VNC passwords
5. Confirm that the VNC server can start and stops cleanly
6. Starting VNC server service automatically at boot
7. Customize xstartup scripts
8. Test each VNC user through direct connection to the VNC Server

4.1 Installing the required VNC server package

Login as administrator in Oracle Linux VM and install the VNC Server.

Our VM runs with Oracle Linux 6, the server is tigervnc-server (and not vnc-server).

4.2  Create the VNC users accounts.

By default the Oracle Linux VM install an oracle user in the system.

To reset the password of Oracle user:

4.3  Edit the VNC server configuration

Edit the "/etc/sysconfig/vncservers" file to configure the required displays.

Add to the file /etc/sysconfig/vncservers the following entries to enable VNC for display number “.1” and “.2”

Notice multiple "display:user" pairs are defined on a single line, but the arguments for each display are defined separately.

With these entries we define that a VNC server instance should be started as user “atoor” on display 1 and the user “oracle” on display 2 and we also set some options for this server such as resolution and color depth. Each VNC server instance listens on port 5900 plus the display number on which the server runs. In our case,

• aroot’s vnc server would listen on port 5901
• oracle’s vnc server would listen on port 5902

4.4  Set your users' VNC passwords

Set the VNC password for any users defined in the "/etc/sysconfig/vncservers" file.

Switch user into the account for each user, and as noted below, run: vncpasswd

We are prompted for a password. This is the password that we will use when we connect to userid’s vnc server instance.

This will create the~/.vnc directory for that userid:

[aroot@fftest5 ~]$ ls -al /home/aroot/.vnc/

total 12

drwxrwxr-x. 2 aroot aroot 4096 Oct 16 08:54 .

drwx------. 3 aroot aroot 4096 Oct 16 08:54 ..

-rw-------. 1 aroot aroot    8 Oct 16 08:54 passwd

4.5  Confirm that the VNC server can start and stop cleanly

[aroot@fftest5 ~]$ sudo /sbin/service vncserver start

[sudo] password for aroot:

Starting VNC server: 1:aroot xauth:  creating new authority file /home/aroot/.Xauthority

New 'fftest5:1 (aroot)' desktop is fftest5:1

Creating default startup script /home/aroot/.vnc/xstartup

Starting applications specified in /home/aroot/.vnc/xstartup

Log file is /home/aroot/.vnc/fftest5:1.log

2:oracle

New 'fftest5:2 (oracle)' desktop is fftest5:2

Creating default startup script /home/oracle/.vnc/xstartup

Starting applications specified in /home/oracle/.vnc/xstartup

Log file is /home/oracle/.vnc/fftest5:2.log

                                                           [  OK  ]

Shutting down VNC server: 1:aroot 2:oracle                 [  OK  ]

Start again the vncserver:

4.6  Starting VNC server service automatically at boot

To enable the vncserver service to be automatically started at boot:

Checking correct automatic boot:

[aroot@fftest5 ~]$ sudo chkconfig vncserver --list

vncserver       0:off   1:off   2:on    3:on    4:on    5:on    6:off

4.7 Customize xstartup scripts

After the VNC service is started, some new files are created in /home/aroot/.vnc/ directory. These include aroot’s vnc server log file, pid file and an X startup script. As user “aroot” we edit the script in order to customize some settings. The default /home/aroot/.vnc/xstartup script contains some commands that are executed when the VNC server is started.

• xsetroot in this case sets the background color.
• vncconfig is a supplementary program that can be used to control the VNC server. Apart from this, when run without arguments it acts as a helper application and its main purpose is to provide support for clipboard transfers between the client (vncviewer) and the vnc server.
• xterm starts an xterm terminal.
• twm starts the X server’s default window manager. We probably want to change that to a more user-friendly window manager.

The presence of the "&" symbol after each command, so that it goes to the background and the xstartup script continues on.

4.8  Test each VNC user through direct connection to the VNC server

You should now be able to use a VNC viewer to connect to system using the display numbers and passwords defined. In our case, we have two VNC users: aroot and oracle.

To allow direct VNC connection we need to open the related TCP ports on Oracle Linux VM.

Open the Azure Management portal, select the Oracle Linux VM and select ENDPOINTS to control traffic incoming on public VIP (Virtual IP Address).

 Figure 7: ENDPOINTS to control incoming network traffic in IaaS VM

Add  a new ENDPOINT

Figure 8: first step to add a new public ENDPOINT

We can example map a public TCP port like 6901 in the port 5901:

 Figure 9: public port 6901 is mapped into private TCP port 5901

We need to add a second ENDPOINT to the Oracle Linux VM to allow to the user oracle to connect to the  private TCP port 5902.

Figure 10: add a second ENDPOINT to connect to the VM through oracle user

After adding the two public Azure ENDPOINTS we have:

Figure 11: endpoints mapping on Oracle Linux VM

4.9  Install VNC client on your local host

Once your VNC server is configured, you can connect to it from any VNC client. On your local linux host (my local CentOS client run in hyper-V in Windows 8), install the VNC client using the following command.

We are now ready from our local linux VM to open a VNC connection to the Oracle Linux VM running in Azure. To manage the remote Oracle Linux VM through the user too we use the command:

A GUI ask for the VNC password assigned to the user toor.

Figure 12: vnc client ask for toor password to connect to remote Oracle Linux VM

Figure 13: successful connection to the Oracle Linux VM through the VNC client

We can establish a second VNC connection to the Oracle Linux VM through oracle user:

Figure 14: remote connection via VNC client as oracle user

5 Tunneling VNC connections over SSH

The VNC protocol is not a secure communication protocol. Presence of a vnc password provides security to access to the server, but the whole VNC session is transmitted in the clear, without encryption. A way to secure our connection to the VNC server is to connect through an encrypted SSH tunnel. This way the whole session get encrypted.

5.1 Open the incoming SSH port in the Oracle Linux VM

In the last paragraph, we have used a direct connection from VNC client to VNC server using the TCP ports above 5900. First of all we do not need any more of public ENDPOINTs on Oracle Linux VM, and we can remove it. The only Endpoint we need is the TCP port 22, to accept incoming SSH traffic.

Figure 15: the only endpoint required to tunnel VNC on SSL is the TCP port 22

We want to configure a local port forwarding for a VNC session

Figure 16: local port forwarding for a VNC session

The syntax for local port forwarding as used in the example:

ssh username@RemoteServerHost -L localport:LocalHost:RemotePort

The operating systems on client and on the server are irrelevant. What matters is that local client has a SSH client and VNC viewer, and the remote machine (Azure Oracle Linux VM) has a SSH server and VNC server.

5.2 Change the VNC server configuration

Edit the file /etc/sysconfig/vncservers and modify the entry:

VNCSERVERS="1:toor 2:oracle"

VNCSERVERARGS[1]="-geometry 1024x768 –depth 24 -nolisten tcp -localhost"

VNCSERVERARGS[2]="-geometry 800x600 –depth 16 -nolisten tcp -localhost"

where:

"-localhost" option prevents remote VNC clients connecting except when doing so through a secure tunnel

“-depth 16” option to specify the color depth in the VNC session

To get the new configuration Restart the VNC service:

Shutting down VNC server: 1:aroot 2:oracle                 [  OK  ]

Starting VNC server: 1:aroot

New 'fftest5:1 (aroot)' desktop is fftest5:1

Starting applications specified in /home/aroot/.vnc/xstartup

Log file is /home/aroot/.vnc/fftest5:1.log

2:oracle

New 'fftest5:2 (oracle)' desktop is fftest5:2

Starting applications specified in /home/oracle/.vnc/xstartup

Log file is /home/oracle/.vnc/fftest5:2.log

                                                           [  OK  ]

5.3  Open an SSL connection with port forwarding

The syntax for local port forwarding as used in the example:

ssh RemoteUsername@RemoteServerHost -L localport:LocalHost:RemotePort

We want to create an encrypted tunnel, and connect to our VNC server through it, but we also want the tunnel to be ”automatically closed”, as soon as we shut down vncviewer:

where:

• toor@ffora1.cloudapp.net = SSH host to connect to
• -f forks the SSH session to the background, while sleep is being executed on the remote machine. This ssh option is needed because we want to execute the following command (vncviewer) in the same local machine’s terminal.
• -L = Option to enable local port forwarding
• 6901 = Port on the client PC
• localhost = Host server to connect to (the remote PC, same as 127.0.0.1)
• 5901 = Port on the remote host (forwarded from the client PC)
• The sleep command is of major importance in the above line as it keeps the encrypted tunnel open for 10 seconds. If no application uses it during this period of time, then it is closed. Contrariwise, if an application uses it during the 10 sec period, then the tunnel remains open until this application is shutdown. This way the tunnel is automatically closed at the time we close vncviewer’s window, without leaving any SSH processes running on our workstation.

To connect to the VNC server, open a new Xterm session and enter the following command:

Figure 17: connection to the VNC Server

Instead to open two Xterm sessions to connect to VNC server, you can run all in unique command:

A second SSH tunnel can be opened from local Linux host to the Oracle Linux IaaS VM, using the remote port 5902:

On the local linux host to remote VNC server, run the command in a new XTERM session:

In VNC authentication, specify the oracle’s user credential.

 Figure 18: vnc client ask for oracle’s user credential

Figure 19: connection to remote Oracle Linux VM through oracle’s user credential

You can connect to the VNC server with "oracle" user credential using a unique command (in single Xterm session):

6 Reference

• HowTo VNC: http://wiki.centos.org/HowTos/VNC-Server
• Configuring VNC Server on Linux: http://www.oracle-base.com/articles/linux/configuring-vnc-server-on-linux.php

7 Glossary

The Microsoft Azure Storage team has introduced an improvement in asynchronous copy blob http://blogs.msdn.com/b/windowsazurestorage/archive/2012/06/12/introducing-asynchronous-cross-account-copy-blob.aspx

Making copy asynchronous is a major change that greatly differs from previous versions. Previously, the Azure Blob service returns a successful response back to the user only when the copy operation has completed. With new version, the Azure Blob service will instead schedule the copy operation to be completed asynchronously: a success response only indicates that the copy operation has been successfully scheduled. In our example we can focus on  Block Blob only. Asynchronous Blob copy is implemented through StartCopyFromBlob, a method of CloudBlockBlob class belonging to the library Microsoft.WindowsAzure.Storage.Blob

When you call StartCopyFromBlob operation on a blob, Blob Storage Service puts the copy operation in a queue; since it's an asynchronous operation you wouldn't know when it will be processed.

To track the progress of the copy can be done a polling the property CopyState on CloudBlockBlob.

CopyState retuns one of values:

Aborted:   The copy operation has been aborted.
Failed:    The copy operation encountered an error.
Invalid:   The copy status is invalid.
Pending:   The copy operation is pending.
Success:   The copy operation succeeded.

With asynchronous copy, we can copy content from Amazon S3 to Azure blob basically with the logic:

1. get the link of the Amazon S3 source container.
2. set a StartCopyFromBloboperation to a destination Azure blob container.
3. track the copy status; if it has failed or has been aborted, start a new copy operation.

2. Copy from Amazon S3 to Azure Block Blob in .NET

A blog post has been written on how to copy from Amazon S3 to Azure blob in .NET. http://gauravmantri.com/2012/06/14/how-to-copy-an-object-from-amazon-s3-to-windows-azure-blob-storage-using-copy-blob/

I use roughly the same code running in Visual Studio C# console application, uptodate to run on current Azure storage .NET SDK.

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using Microsoft.WindowsAzure.Storage;

using Microsoft.WindowsAzure.Storage.Auth;

using Microsoft.WindowsAzure.Storage.Blob;

using Microsoft.WindowsAzure.Storage.Blob.Protocol;

using System.IO;

using System.Configuration;

namespace AzureBlobStorage01

{

    class Program

    {

        // amazonObjUrl          : it is the Amazon S3 source URL

        // azureStorageAccoutName: it is the name of destination Azure storage account

        // azureStorageAccoutName: it is the key of destination Azure storage account

        // azureBlobContainerName: it is the name of destination blob container 

        // azureBlobContainerName: it is the name of destination blob

        privatestaticreadonlystring amazonObjectUrl =

                  ConfigurationManager.AppSettings["AmazonSourceObjectURL"];

        privatestaticreadonlystring azureStorageAccountName =

                   ConfigurationManager.AppSettings["DestinationStorageAccountName"];  

        privatestaticreadonlystring azureStorageAccountKey =

                    ConfigurationManager.AppSettings["DestinationStorageAccountKey"];    

        privatestaticreadonlystring azureBlobContainerName =

                    ConfigurationManager.AppSettings["DestinationBlobContainerName"];        

        privatestaticreadonlystring azureBlobName =

                    ConfigurationManager.AppSettings["DestinationBlobName"];

        staticvoid Main(string[] args)

        {

            CloudStorageAccount storageAccount =

                      new CloudStorageAccount(new StorageCredentials(azureStorageAccountName, azureStorageAccountKey), true);

            CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();

            CloudBlobContainer blobContainer = cloudBlobClient.GetContainerReference(azureBlobContainerName);

            Console.WriteLine("Trying to create the blob container....");

            blobContainer.CreateIfNotExists();

            Console.WriteLine("Blob container created successfully");

            Console.WriteLine("------------------------------------");

            CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference(azureBlobName);

            Console.WriteLine("Created a reference for block blob in Windows Azure....");

            Console.WriteLine("Blob Uri: " + blockBlob.Uri.AbsoluteUri);

            Console.WriteLine("Now trying to initiate copy....");

            blockBlob.StartCopyFromBlob(new Uri(amazonObjectUrl), null, null, null);

            Console.WriteLine("Copy started....");

            Console.WriteLine("Now tracking blob's copy progress....");

            DateTime startTime = DateTime.UtcNow;

            bool continueLoop = true;

            while (continueLoop)

            {

                Console.WriteLine("");

                Console.WriteLine("Fetching lists of blobs in Azure blob container....");

                IEnumerable<IListBlobItem> blobsList = blobContainer.ListBlobs(null, true, BlobListingDetails.Copy);

                foreach (var blob in blobsList)

                {

                    var tempBlockBlob = (CloudBlockBlob)blob;

                    var destBlob = blob as CloudBlockBlob;

                    if (tempBlockBlob.Name == azureBlobName)

                    {

                        var copyStatus = tempBlockBlob.CopyState;

                        if (copyStatus != null)

                        {

                            Console.WriteLine("Status of blob copy...." + copyStatus.Status);

                            float percentComplete = 100*

                                float.Parse(copyStatus.BytesCopied.ToString()) /

                                float.Parse(copyStatus.TotalBytes.ToString());

                            Console.WriteLine("Total bytes to copy...." + copyStatus.TotalBytes);

                            Console.WriteLine("Total bytes copied....." + copyStatus.BytesCopied);

                            Console.WriteLine("Perc. byte copied......{0:N1}", percentComplete);

                            if (copyStatus.Status != CopyStatus.Pending)

                            {

                                continueLoop = false;

                            }

                        }

                    }

                }

                Console.WriteLine("");

                Console.WriteLine("==============================================");

                System.Threading.Thread.Sleep(1000);

            }

            DateTime endTime = DateTime.UtcNow;

            TimeSpan diffTime = endTime - startTime;

            Console.ForegroundColor = ConsoleColor.Yellow;

            Console.WriteLine("time transfer (D HH:mm:ss):  " +

                  diffTime.Days + " " + diffTime.Hours + ":" +

                  diffTime.Minutes + ":" + diffTime.Seconds);

            Console.ResetColor();

            Console.WriteLine("Press any key to terminate the program....");

            Console.ReadLine();

        }

    }

}

The content of App.config file In Visual Studio project is shown below:

<?xmlversion="1.0"encoding="utf-8" ?>

<configuration>

  <startup>

    <supportedRuntimeversion="v4.0"sku=".NETFramework,Version=v4.5" />

  </startup>

  <appSettings>

    <add key="AmazonsourceObjectURL"value="http://AMAZON_URL"/>

    <add key="DestinationStorageAccountName"value="AZURE_DESTINATION_STORAGE_ACCOUNT_NAME"/>

    <add key="DestinationStorageAccountKey"value="AZURE_DESTINATION_STORAGE_ACCOUNT_KEY"/>

    <add key="DestinationBlobContainerName"value="AZURE_BLOB_CONTAINER_NAME"/>

    <add key="DestinationBlobName"value="BLOB_NAME"/>

  </appSettings>

</configuration>

Before running the project set the right values in Visual Studio App.config file.

To compile and run the program you need to reference the Windows Azure Storage Library for .NET available on  NuGet:  http://www.nuget.org/packages/WindowsAzure.Storage/

Azure Storage library can be installed easily through Visual Studio Package Management; from Visual Studio open TOOLS -> NuGet Package Manager -> Package Manager Console

 Figure1 Visual Studio menu navigation to access to NuGet Package Manager 

In the windows below is shown the Visual Studio Package Management (PM) Console

Figure2 Visual Studio Package Management Console

In Package Management (PM) Console write the command:

PM> Install-Package WindowsAzure.Storage

NuGet install following libraries (from log):
Attempting to resolve dependency 'Microsoft.Data.OData (≥ 5.6.0)'.
Attempting to resolve dependency 'System.Spatial (= 5.6.0)'.
Attempting to resolve dependency 'Microsoft.Data.Edm (= 5.6.0)'.
Attempting to resolve dependency 'Newtonsoft.Json (≥ 5.0.6)'.
Attempting to resolve dependency 'Microsoft.Data.Services.Client (≥ 5.6.0)'.
Attempting to resolve dependency 'Microsoft.WindowsAzure.ConfigurationManager (≥ 1.8.0.0)'.

.........

The list of all included libraries included through NuGet is reported in Figure 3

Figure3 list of libraries included through NuGet

To read the pairs (key, value) in App.config we need to add to the Visual Studio project the system library: System.Configuration

3. Copy from Amazon S3 to Azure Block Blob in Java

Microsoft has published the Windows Azure Storage SDK for Java at GitHub  https://github.com/Azure/azure-storage-java and the related documentation is available at the link :http://dl.windowsazure.com/storage/javadoc

Using the Windows Azure Storage SDK for Java we can port the code from.NET (see code above) to Java:

// This java program copies the content from Amazon S3 to Azure Storage block blob

// Before running fill up the right input values for:

//   azureStorageAccountName : Name of the Azure Storage Account 

//   azureStorageAccountKey  : Primary Key of the Azure Storage Account

//   azureBlobContainerName  : Name of Azure blob destination container name

//   amazonObjectUrl         : URL of input file

//   azureBlobName           : Name of destination blob in Azure Storage Account

package test2;

import com.microsoft.windowsazure.storage.*;

import com.microsoft.windowsazure.storage.blob.*;

import java.net.URISyntaxException;

import java.security.InvalidKeyException;

import java.io.FileNotFoundException;

import java.io.IOException;

import java.net.URI;

import java.util.*;

publicclassS3toBlob {

  privatestatic final String amazonObjectUrl = "http://AMAZON_URL/";

  privatestatic final String azureStorageAccountName = "AZURE_DESTINATION_STORAGE_ACCOUNT_NAME";

  privatestatic final String azureStorageAccountKey = "AZURE_DESTINATION_STORAGE_ACCOUNT_KEY";   

  privatestatic final String azureBlobContainerName = "AZURE_BLOB_CONTAINER_NAME"; 

  privatestatic final String azureBlobName = "AZURE_BLOB_NAME";

  privatestatic String storageConnectionString ;

  publicstaticvoid main(String[] args) throws

      URISyntaxException,

      StorageException,

      InvalidKeyException,FileNotFoundException,

      IOException {            

      try

      {

           //Define the connection-string with your values

           StringBuilder _sb = new StringBuilder();

           _sb.append("DefaultEndpointsProtocol=http;").append("AccountName=");

           _sb.append(azureStorageAccountName);

           _sb.append(";");

           _sb.append("AccountKey=");

           _sb.append(azureStorageAccountKey);

           storageConnectionString = _sb.toString();

           // Retrieve storage account from connection-string

           // Retrieve storage account from connection-string

           CloudStorageAccount storageAccount = CloudStorageAccount.parse(storageConnectionString);

           CloudBlobClient blobClient = storageAccount.createCloudBlobClient();

           CloudBlobContainer container = blobClient.getContainerReference(azureBlobContainerName);

           // Create the container if it does not exist

           container.createIfNotExists();                   

           CloudBlockBlob blockBlob = container.getBlockBlobReference(azureBlobName);

           System.out.println("Created a reference for block blob in Windows Azure....");

           System.out.println("Blob Uri: " + blockBlob.getUri());

           System.out.println("Now trying to initiate copy....");

           OperationContext op = new OperationContext();                   

           blockBlob.startCopyFromBlob(new URI(amazonObjectUrl), null, null, null,op);

           System.out.println("Copy started....");

           System.out.println("Now tracking blob's copy progress....");

           Date startTime = new Date();

           boolean continueLoop = true;

           while (continueLoop)

           {

                System.out.println("");

                System.out.println("Fetching lists of blobs in Azure blob container....");

                Iterable<ListBlobItem> blobsList = container.listBlobs(null, true, EnumSet.of(BlobListingDetails.COPY), null, op);

                for (ListBlobItem blob : blobsList)

                {

                   CloudBlockBlob tempBlockBlob = (CloudBlockBlob) blob;                    

                   System.out.println("Name  blockblob in cast: "+ tempBlockBlob.getName().toString());

                   System.out.println("URI                    : "+ tempBlockBlob.getUri().toString());

                   if ((tempBlockBlob.getName()).equals(azureBlobName))

                   {            

                      CopyState copyStatus =  (CopyState) tempBlockBlob.getCopyState();                        

                      System.out.println("Getting CopyState......");

                      if (tempBlockBlob != null)

                      {

                         System.out.println("Status blob copy......" + tempBlockBlob.getCopyState().getStatus().toString());

                         System.out.println("Total bytes..........." + tempBlockBlob.getCopyState().getTotalBytes());

                         System.out.println("Total bytes to copy..." + tempBlockBlob.getCopyState().getBytesCopied());

                         float percentComplete = 100*

                                ( tempBlockBlob.getCopyState().getBytesCopied()).floatValue() /

                                ( tempBlockBlob.getCopyState().getTotalBytes()).floatValue();

                         System.out.print("Perc. byte copied......");

                         System.out.format("%.2f\n", percentComplete);                          

                          if (copyStatus.getStatus() != CopyStatus.PENDING)

                          {

                              continueLoop = false;

                          }

                      }

                   }

              }

              System.out.println("==============================================");

              Thread.sleep(1000);

          }

          Date endTime = new Date();        

          long diffTime = endTime.getTime() - startTime.getTime();

          long diffSeconds = diffTime / 1000 % 60;

          long diffMinutes = diffTime / (60 * 1000) % 60;

          long diffHours = diffTime / (60 * 60 * 1000) % 24;

          long diffDays = diffTime / (24 * 60 * 60 * 1000);

          System.out.println("time transfer (D HH:mm:ss):  "+

                   diffDays + " " +

                   diffHours +":"+

                   diffMinutes+":"+

                   diffSeconds);

          System.out.println("Press any key to terminate the program....");

          System.in.read();                

         } catch (StorageException storageException) {

                   System.out.print("StorageException encountered: ");

                   System.out.println(storageException.getMessage());

                   System.exit(-1);

         } catch (URISyntaxException uriSyntaxException) {

                   System.out.print("URISyntaxException encountered: ");

                   System.out.println(uriSyntaxException.getMessage());

                   System.exit(-1);

         } catch (Exception e) {

                   System.out.print("Exception encountered: ");

                   System.out.println(e.getMessage());

                   System.exit(-1);

        }

  }

}

This blog shows you how to use Microsoft HPC Pack 2012 R2 and Microsoft Azure to set up an High Performance Computing (HPC) cluster. Microsoft HPC Pack 2012 R2 is a middleware free of change, available at Microsoft download center: http://www.microsoft.com/en-gb/download/details.aspx?id=41630

This blog assumes no prior experience with HPC Pack and it is intended only to help you deploy a cluster quickly for demonstration purposes. The cluster will consist of an  head node running in Azure IaaS Virtual Machine and some compute nodes you deploy on-demand as Azure worker role instances in Azure Cloud Service. Some of technical choices used in this paper are not recommended in a production environment, and they are used for demonstration purposes only. The examples in this paper refer to Windows Server 2012 R2. Here the table of content:

2 Lab topology architecture for Windows HPC in Azure

Our Windows HPC Server architecture consists of one head node a some compute nodes run in Azure (the topology is shown in Figure 1):

• The head node is the master node in the HPC cluster and runs in Azure IaaS Virtual Machine (VM). It controls and manages all resources that are available in HPC system.
• All HPC compute nodes (slaves of the cluster) run in Azure as Worker Role (PaaS).
• In the midst of deployment of HPC compute nodes in Azure, two or more Compute Proxies (Azure Worker Roles) are created automatically to control communications between the head node and the compute nodes.

  Figure 1   Architecture of the Windows HPC Server cluster fully in Azure

In Windows HPC Server, the cluster authorization and authentication is based on an Active Directory domain controller. For stability and security in a production environment, it is recommended that you install a domain controller on a dedicated Azure IaaS VM that is different from the HPC head node. However, in our lab to save up some resources, the domain controller is installed on the head node of the HPC cluster as the root domain hpcdom.local.  The Windows domain is set in Azure and it is not propagated to on-premises; in case were necessary integrate the user’s credential with on-premises credential an Azure site-to-site VPN will be needed to interconnect the Azure VNET (Virtual Network) with existing AD DC on-premises. Our HPC head node in IaaS VM is attached to the Azure Virtual Network with network belonging to the subnet 10.1.1.0/24.

The first step to create a cluster in Windows HPC Server is to configure the HPC head node.

3 Pre installation tasks in Azure

Few basic pre installation tasks has to be completed thought the Azure Management portal before setting up the HPC cluster fully in Azure.

  3.1  Definite an Azure Affinity Group

Affinity groups are the way to group the services in your Windows Azure subscription that need to work together in order to achieve optimal performance. When you create an affinity group, it lets Azure know to keep all of the services that belong to your affinity group running at the same data center cluster. This reduces latency and increases performance.

Login in Azure Management portal, select “SETTING” tab, “AFFITINITY GROUPS” and therefore click-on “ADD AN AFFINITY GROUP”

Figure 2 Add a new Affinity Group

 Figure 3  Selection of specific Azure datacenter associated with affinity group

  3.2 Define an Azure Storage Account

 An Azure Storage Account is required to store the image (.VHD file) of HPC head node running in Azure IaaS VM.

Figure 4   create a new Azure Storage account

In our lab we haven’t any interest for geo resiliency of the head node, for this reason the Azure storage account is created with Local Redundant; in production is recommended usage of Azure Geo-Redundant  storage account to guarantee data recovery in case of major disaster on the Azure primary datacenter.

 Figure 5   specify the URL for the storage account and reference the affinity group created in previous step

  3.3  Define an Azure VNET

Azure Virtual Network (VNET) is a virtual segment in Azure datacenter to interconnect Virtual Machines (IaaS and PaaS); we proceed now in the creation of HPC Virtual VNET.

Figure 6  add a new VNET

 Figure 7 specify a mnemonic name for the VNET and the affinity group created in previous step

In our configuration the head node is a root domain AD and runs in IaaS VM connected to the VNET. A right setup requires that the DHCP serves associated to the VNET assigns as DNS the IP Address of Domain Controller. The head node will borrow the IP address .4 (see Figure 8A and Figure 8B).

Figure 8A DNS server and DHCP server associated to the Azure VNET

Figure 8B the HPC head node is the DNS server for the Azure VNET

Figure 9   specify the major network and a subnet associated to the VNET

  Figure 10   VNET configuration summary

3.4 Add an Azure IaaS VM to run the HPC head node

We are now ready to define the Azure IaaS VM to guest the HPC head node.

 Figure 11 add a new Azure IaaS VM through the Azure Management portal

  Figure 12 selection of the OS in images library

 Figure 13 choice Windows Server 2012 R2 OS image

 Figure 14 define the hostname, size and local administrator password for the Azure IaaS VM

 Figure 15 define the cloud service, virtual network and storage account associated with IaaS VM

Cloud Services is a logical container for one or more Virtual Machines. You can create a cloud service for a single VM or you can load balance multiple VMs by placing them in the same cloud service.

Figure 16 define the VM Agent and endpoints

The VM Agent is used to install and manage extensions that help you interact with the VM (for example password recovery).

 4 Configuring the Windows HPC head node

The Windows HPC Server head node is the mandatory host for our lab; use the following process to install Windows Server 2012 R2 on Azure IaaS VM and configure the head node.

  4.1 Promote the HPC head node as the domain controller

Login in via RDP to the IaaS VM by local administrator credential and promote the HPC head node as the domain controller:

1. In the server manager click-on Add role and features:

 Figure 17   Add a new Role

  Figure 18   selection of Role-based installation

 Figure 19   Add the Active Directory Domain Services role

 Figure 20   AD DC role installation component

 Figure 21 to complete installation of AD DC role access to the system tasks

 When the AD Directory Domain Services are installed, select the icon flag and choice “Promote this server to domain controller”

 Figure 22   Promotion of IaaS VM as AD Domain Controller

2. Specify the option to create a new domain in a new forest:

 Figure 23   define the IaaS VM as root domain controller and assign the FQDN to be associated with the root domain

 Figure 24   specify the forest level and domain level in the root domain

Leave the properties of AD DC as specified by default; the domain controller needs to have Domain Name System (DNS) server capabilities.

Figure 25   leave the NetBIOS domain name as is

 Figure 26  a warning message inform that it is not best practice use dynamic IP address on AD DC

A warning message advice that the IP address on the NIC is set as dynamic, but in our setup it is fine because the IP addresses assigned to all IaaS VMs attached to the Azure VNET are managed by the Azure fabric.

We are ready to install HPC Pack 2012 R2 on the head node (winhpchn).

  4.2 Install Windows HPC Pack 2012 R2  on the head node

Windows HPC Pack 2012 R2 enables you to create an HPC cluster. Log on as a domain administrator in the head node, download from Microsoft web site the Windows HPC Pack 2012 R2 (version 4.2.4400.0) and run setup. Select the option New installation or add features to an existing installation.

 Figure 27   Install HPC Pack 2012 R2

 In the HPC Pack 2012 R2 Installation Wizard, select Create a new HPC cluster by creating a head node.

Figure 28   Create a new head node

Window HPC Pack 2012 R2 creates five HPC databases on the head node:

• Cluster management database (HPCManagement): Stores node performance counter data
• Job scheduling database (HPCScheduler): Stores job properties, job allocation, and history
• Reporting database (HPCReporting): Stores historical data about the cluster such as cluster utilization, node availability, and aggregation of job statistics
• Diagnostic database (HPCDiagnostics): Stores diagnostic test results
• Monitoring database

Windows HPC Pack 2012 R2 supports features to install the HPC databases locally or remotely. In our setup, all HPC databases are installed on the SQL Server running in the head node, as shown in the Figure 29.

Figure 29   Install all HPC databases on the local SQL Server instance

 Figure 30   default SQL Server instance named COMPUTECLUSTER

 The Installation Wizard enables administrator to specify a folder to install SQL Server databases and log files.

Figure 31   Specify a directory to store databases and logs

 The administrator of the HPC cluster can choose the folder location for installing the Windows HPC Pack 2012 R2 files and system data (including executables, .dll files, and script files), as shown in Figure 32.

 Figure 32   Folder location to store Windows HPC Pack 2012 R2 files and data

 Figure 33 shows the list of role services and software components that are needed to install HPC Pack 2012 R2 .

Figure 33   Software components required to run HPC middleware

 Figure 34  start installation of Microsoft SQL Server 2012 Express edition

Figure 35  setup of HPC PACK 2012 R2

 After the Windows HPC Pack 2012 R2 installation, the following HPC services are installed on the head node (winhpchn):

Figure 36   HPC services on the head node

  4.3 Configure the head node

Setup of Windows HPC Pack 2012 R2 on head node requires the follow steps.

Open the HPC Management Console; a list of configuration tasks appears in the Deployment To-do-list pane.

 Figure 37   To-do list in the HPC management console

The first task in the Deployment To-do List is to select a network topology for the HPC cluster. There are five topologies to choose from, and we choose Topology 5: All nodes only on an enterprise network.

Note: Topology 5 does not allow a bare-metal installation of on-premises compute nodes. In our lab, we have only Azure PaaS compute nodes , where baremental is not required.

 Figure 38   Topology 5 with all nodes connected on the same network

 The Figure 39 shows how the network adapter on the head node connected to segment named in HPC term as enterprise network.

Figure 39   Select the network adapter for the HPC head node

 In our system, all HPC nodes run in Azure where a front-end firewall is present to control traffic on IP Public Endpoints; presence of further Windows Firewall inside the HPC nodes is not required and we disable it.

Figure 40   Disable the Windows Firewall in the HPC nodes

 The Figure 41 shows a review of the setup for the Enterprise Network of the HPC cluster.

Figure 41  Setup review

The second task on the to-do list is to define the domain user who is responsible for managing the HPC cluster.

Figure 42   Provide credentials for HPC cluster administration

 In our configuration, bare-mental installation is not used. However, the HPC Management Console requires a mandatory naming convention for compute nodes that are installed through a bare-metal procedure.

Figure 43   Naming convention for HPC compute nodes installed through bare-metal procedure

  4.4  Define a node template to complete the Deployment to-do List

The last step to complete the tasks in Deployment to-do List is  the definition of a HPC node template. Nevertheless it is not used, we choose to create a node template for an on-premises compute node.

 Figure 44   Create an HPC node template

 Figure 45   add the on-premises node template

Fill up the name for the new node template.

Figure 46   Name the node template

 We will not use this template to add an on-premises compute node to the cluster but only to complete the Deployment to-do-List.

Figure 47   Create a node template without an operating system

We do not add security updates to the node template (we do not care of it).

 Figure 48   Do not request updates

 The Figure 49 is the Deployment To-do List after completion of the mandatory tasks.

Figure 49   Completed mandatory tasks

5 Prepare the Azure subscription

These steps are needed to Azure nodes from the head node.

• Upload a management certificate (needed for secure connections between the head node and the Azure Cloud Services)
• Create an Azure Cloud service in which Azure nodes (worker role instances) will run
• Create a Windows Azure storage account

  5.1 Upload the HPC default management certificate

HPC Pack installs by default a self-signed certificate on the head node, called the Default Microsoft HPC Azure Management certificate (the file is stored in %CCP_HOME%\Bin\hpccert.cer), which you can upload as an Azure management certificate in Azure subscription. This certificate is provided for testing purposes and proof-of-concept deployments. The management certificate (also referred to as a subscription certificate)

 Figure 50   location of Default Microsoft HPC Azure Management certificate (hpccert.cer)

To upload the hpccert.cer in the Azure subscription:

1. Connect to the HPC head node computer, open the web browser and sign in to the Management Portal.
2. In Azure Management portal click Settings, and then click Management Certificates.
3. On the command bar, click Upload.

 Figure 51   upload the Default Microsoft HPC Azure Management certificate in Azure subscription

 Figure 52   selection of hpccert.cer certificate in the HPC head node

Figure 53   Default HPC Azure Management certificate loaded in the Azure subscription

 You will see Default HPC Azure Management in the list of management certificates.

  5.2  Create an Azure Cloud Services

Define an Azure Cloud Service to run HPC PaaS Compute Nodes:

1. In the Management Portal, on the command bar, click New.
2. Click Compute, click Cloud Service, and then click Quick Create.
3. Type a URL for the cloud service, and then click Create Cloud Service.

 Figure 54   define a new Cloud Service

 For best performance, the cloud service and the storage account are created in the same affinity group.

  5.3 Create a Windows Azure storage account

Create a new Azure storage account to run the HPC PaaS Compute nodes:

1. In the Management Portal, on the command bar, click New.
2. Click Data Services, click Storage, and then click Quick Create.
3. Type a URL for the account, and then click Create Storage Account.

 Figure 55   define a new Azure Storage Account

6  Deploying Windows Azure nodes in Windows HPC Server 2012 R2

The following paragraph describes the full process for deploying Windows Azure worker nodes to a cluster running Windows HPC Server 2012 R2.

  6.1 Create a Windows Azure Worker Node Template

A worker node template specifies the information configured in your Windows Azure subscription that will be used to add a set of Windows Azure nodes to the cluster. This information includes:

• Azure Subscription ID. Azure Subscription ID is a 32-character hexadecimal string, in the dashes form xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
• certificate thumbprint for your management certificate. The thumbprint of a subscription certificate (also referred to as a management certificate) provides the required authentication for clients to connect to the Windows Azure subscription. This is a 40-character hexadecimal string
• name of an Azure Cloud Service
• name of an Azure Storage Account.

1. Open the HPC Cluster Manager

2. In Configuration, in the Navigation Pane, click Node Templates.

3. In the Actions pane, click New, which will bring up The Create Node Template Wizard.

 Figure 56 Creation of a new template

 4. On the Choose Node Template Type page, click Windows Azure node template, and then click Next.

Figure 57 Selection of a Windows Azure node template

5. On the Specify Template Name dialog, type a name for the worker node template and click Next.

Figure 58  Name the new template

6. On the Provide Subscription Information page, provide the Windows Azure subscription that will be used to add the nodes:

a. In the Subscription ID text box, type or paste the ID of a valid Windows Azure subscription account.

Figure 59  The Azure subscription ID

b. In the Management certificate text box browse in the Certificates store of the local computer account and select the Default Microsoft HPC Azure Management certificate, then click Next.

Figure 60    selection of the Default Microsoft HPC Azure Management certificate

Figure 61   Specification of the Azure Subscription ID and certificate thumbprint

7. On the Provide Service Information page, select the cloud service and the storage account that you created in a previous step. Then click Next.

Figure 62    select the Azure Cloud Service and Storage Account created before

Figure 63    define number of Azure proxy nodes

Figure 64    specify the OS family for the compute nodes (worker roles)

 The Azure Cloud Services are available in three different OS family: Windows 2008 R2, Windows 2012, Windows 2012 R2. Our Azure node template will create worker role with Windows Server 2012 R2 OS.

Figure 65    empty startup script

 Figure 66    template define worker roles attachment to the same VNET of the head node

8. on the Configure Remote Desktop Credentials page, type a user name and password that you can use to make Remote Desktop connections to your Windows Azure nodes. The credentials you provide will be used only to connect to the Windows Azure nodes that are deployed using this template. They should be different than your domain credentials or any local credentials. The password you provide is encrypted in the HPC databases and it is not exported if you export the node template.

Figure 67 Specify Remote Desktop credentials to access the Azure worker node

9. On the Configure Azure Availability Policy page, select how you want the Windows Azure worker nodes to start and stop (manually or by weekly-based calendar scheduler).

Figure 68  Policy associated to the Azure Node template

 10. To modify the worker role template, right click on node template, select Edit…  (Figure 69).

Figure 69 changing parameters and options in Azure node template

 11. To validate the Windows Azure connection information, on the Connection Information tab, click Validate connection information.

Figure 70 Validating the connection of head node to Azure Cloud Service and Storage Account

 During the validation process, Windows HPC Server 2012 R2 connects to the Windows Azure subscription using the configured certificate, and verifies that the Azure Cloud Service and Azure Storage Account can be reached.

  6.2 Change the role in HPC head node

By default, HPC head node has the role of Head Node, Broker Node and Compute Node. it is a good practice to disable the Compute Node role in the HPC head node to avoid CPU overload in case of heavy jobs (main purpose of head node is to control and manage the HPC clusters).

Figure 71   head node in offline

Figure 72   change role of the head node

 Figure 73   deselect Compute Node role

Figure 74   head node has the only Head node and WCFBroker node roles

  6.3 Customize the Heat Map

HPC Heat Map is a powerful embedded tool in HPC Management console to acquire the fully status of cluster. To customize the HPC Heat Map select the tab, right-click on mouse and therefore “Customize Tab…” option.

Figure 75   option to customize the heat map in the HPC Management Console

 Figure 76   selection of three system parameters to include in the heat map

Figure 77   heat map with tracking of CPU, Core in Use and Network Usage

  6.4  Add Windows Azure Worker Nodes to the HPC Cluster

Still now our HPC cluster includes only the head node (master). To add new Azure HPC compute nodes to the cluster follow the procedure described below.

1. In the Actions pane in the Node Management page, click Add Node to bring up the Add Node Wizard.

 Figure 78   Add node to the HPC head node

2. On the Select Deployment Method page (Figure 79), select Add Windows Azure nodes, and then click Next.

 Figure 79 Specifying the type of node to add to the HPC cluster

3. On the Specify New Nodes page (Figure 80), select the Azure node template, specify the number and the size of the worker nodes, and then click Next.

 Figure 80  Assign a number of azure nodes and azure node size

The number of worker nodes is the number of role instances that will be deployed in Windows Azure when you start the nodes.

The size of the worker nodes defines types of predefined PaaS VM sizes in Windows Azure: Small (A1), Medium (A2), Large (A3), or Extra Large (A4), A5, A6, A7, A8 and A9. The size of the virtual machine determines the number of CPU cores, the memory capacity, and the local file system size of each role instance.

Figure 81   define the number and size of Windows Azure nodes

4. On the Completing the Add Node Wizard dialog, click Finish.

 Figure 82   Azure node in Not-Deployed state

Note: If you want to add worker nodes of different sizes, you must run the Add Node Wizard separately for each size. After you add worker nodes to the cluster, they are in the Not-Deployed state, and they have a health state of Unapproved. Before you can use the worker nodes to run jobs, you must start them and then bring them ONLINE. Worker nodes are numbered consecutively in a naming series that begins with the root name AzureCN (the naming series for the worker nodes cannot be configured).

At this stage in the Azure Cloud Service associated with the Azure node template the number of instances are empty.

 Figure 83   Azure Cloud Service is initially empty

  6.5 Start the Windows Azure Worker Nodes

To provision the role instances in Windows Azure, you must start the worker nodes that you added to the HPC cluster; to provision the role instance:

1. Open the HPC Management console
2. In Node Management, in the Navigation Pane, click Nodes.
3. In the List or Heat Map view, select one or more worker nodes.
4. In the Actions pane, click Start to bring up the Start Windows Azure Nodes dialog (Figure 84).

 Figure 84   Starting an Azure Worker Node

5. Click “Start” (Figure 85) to kick off the deployment phase.

 Figure 85 using a specific Node Template to start Azure nodes

6. During the start process, the state of the worker nodes changes from Not Deployed to Provisioning. If you want to track the provisioning progress, select a worker node, and then in the Details Pane, click the Provisioning Log tab (Figure 86).

Figure 86 Provisioning log of one worker node

Figure 87    Worker nodes in the Provisioning state

 The status of Azure Paas VMs as reported in Azure Management portal is shown in Azure management portal (Figure 88-89)

Figure 88   initial state of Azure compute nodes and proxies in deployment phase

Figure 89   creating PasS VMs in deployment phase

7. To bring the nodes online, in the HPC management console select the nodes that are in the Offline state, right-click, and then click Bring Online (Figure 90).

Figure 90  Bring the Azure worker nodes Online

During the provisioning process, Windows HPC Server 2012 R2 automatically retrieves the primary access key for the storage account that is specified in the Azure node template.

After a worker node starts successfully, the node state changes to Offline.

Figure 91 HPC proxies automatically configured from the HPC Head node

 In Figure 92 is shown a deployment with two Azure compute proxies and two Azure worker roles.

Figure 92  HPC Compute Proxies and HPC Azure Worker

Figure 93  HPC heat map with the head node and three Azure nodes

If you want to remove a set of role instances in Windows Azure, stop the nodes by using HPC Cluster Manager (apply the Stop action). This deletes the role instances from the service and changes the state of the worker nodes in the HPC cluster to Not-Deployed.

Figure 94  action to delete Azure node in Azure Cloud Services

Figure 95  Action to stop Azure compute nodes

Figure 96  Azure node in transitional phase from Running to Not-Deployment

Figure 97  Azure nodes in Not-Deployed state

When Azure nodes reach the Not-Deployed state, the corresponding Azure Cloud Service is empty.

7 Reference

• Burst to Windows Azure with Microsoft HPC Pack

http://technet.microsoft.com/library/gg481749.aspx

• Virtual Machine and Cloud Service Sizes for Windows Azure

http://msdn.microsoft.com/en-us/library/windowsazure/dn197896.aspx

• A8 and A9 Compute Intensive Instances: Quick Start with HPC Pack