Many, many companies and individuals use Excel.  Informal estimates have the number of Excel users at over 500 million. Excel is used for everything from making a grocery list to managing a 15 billion dollar portfolio of complex financial instruments to mission planning for the de-orbiting of a satellite.

It is the ubiquitous application in financial services across the Insurance, Capital Markets and Banking verticals is Excel.   Other industries, including Pharmaceutical, Health Care and finance departments within large multinationals also are significant users of Excel. 

Some users of Excel stretch the limits of its capabilities, both of the application itself and also the limitations of the hardware they are using to perform their calculations.  These aren’t typical users of Excel but “Power users” that are creating spreadsheets that are so big and complicated that many of them run for hours, days and sometimes weeks.  They are in great need of a solution that enables them to reach their answer significantly sooner than they are currently able.

Today, such users are going through great pains to optimize their solutions to reduce the time to solution but have limited methods and solutions to do so.  Most will opt for the traditional approach of simply buying a bigger, more powerful machine to run on.  In the past, going from a single core desktop running at 1 GHz and 1 GB of ram and moving to a single core desktop running at 3 Ghz and 3 GB of ram provided a very attractive performance multiplier without ever having to “touch” the spreadsheet that was suffering.

The problem is that clock speeds have leveled off over the past 2-5 years and computers with significantly higher clock speeds just aren’t available.  In 1989, some of the earliest PC-based servers were running on the order of 33Mhz.  Fast forward to 2005, just as dual-core processors were going mainstream, you could buy a Pentium 4 x86 server that ran at 3.4GHz. This increase in clock speed allowed a very simple/affordable path by which a user could easily “scale up”.

While Moore’s law has been going and continues to go strong, clock frequencies have stopped increasing due to the basic physics of power density on silicon.  Moore’s law states that the number of transistors that can be placed on an IC doubles every 2 yrs.  For example, today you could buy an  x86 server running up to four 6-core Intel Xeon processors, at 2.67 GHz, slower than our single-core x86 servers 5 years ago. 

The traditional “scale up” approach has now turned into a parallel, or “scale out” problem.  Scaling out could be running in parallel across multiple cores and/or scaling out beyond the computer/node to running in parallel across multiple computers/nodes and lastly, scaling out could be scaling out from your environment to run in parallel across multiple virtual machines in the cloud.

Many applications like Excel and the applications written for Excel, weren’t originally designed to run in parallel and need to be re-engineered and re-thought in order to take advantage of a parallel architecture.

Many organizations expressed their challenges with long running workbooks and asked Microsoft to look into tools/methods Microsoft could provide in order to reduce run times.  The result was HPC Services for Excel.

HPC Services for Excel is a comprehensive set of tools for running Excel spreadsheet calculations on a Windows HPC 2008 R2 cluster.  With HPC Services for Excel, you can build applications that run Excel workbooks in parallel for faster overall calculation.

The basic premise behind HPC Services for Excel is being able to take calculations that are taking too long to perform in Excel running on a single computer and “offloading” or moving those calculations to run on computers that are part of a High Performance Computer cluster.  Microsoft has a product, Windows HPC 2008 Server, R2 which can be installed and configured on a group of commodity computers aka “nodes”, networked together, to run computationally intense work.  A Windows HPC system is comprised of a head or “master” node and one or more “compute” nodes.  The Master node acts as the brains of the cluster managing what work gets scheduled to be run on the compute nodes and  providing a single point of management and administration of the cluster.

Example:

Below is an example of results from a project for a firm in the capital markets industry.  The firm has spreadsheet based calculation designed to price 250 derivative securities.

The securities are complex.  The models used to price them are also complex – in addition to the spreadsheet calculations, each valuation includes some VBA code, and some third party and external libraries that are called during each pricing calculation.

For 250 securities, the spreadsheet takes over an hour on the desktop.  By moving the calculations to a 64 core cluster, the firm is able to reduce this to 65 seconds.

For each security, the calculation to price it originally took approximately 15 seconds.  The graph below show that by going to a 4 core cluster, on average each security takes about 4 seconds.  You can see as the application is moved to a greater number of cores, scaling is not linear and this is due to overhead and structuring the calculations.  Even though the result isn’t get straight line performance scaling, exceptional performance improvement is gained as cores are added.  The increase achieved by this method permits the firm to change how they use this spreadsheet in practice.

Method 1:  Cluster enabled User Defined Functions (UDFs)

The first method for accelerating an Excel spreadsheet is by “offloading” certain long running, parallelizable calculations via UDFs to external compute resources. 

This method works as follows:  A user has a spreadsheet that has a few long running functions that can be run in parallel as UDFs.   From the client, the user will execute the calculation of the workbook.  When the workbook goes to execute the UDF function that is to be run in parallel, Excel will send a request to the HPC scheduler to execute the UDFs on the cluster.  The scheduler will then dispatch a request to one or more compute nodes.  On each compute node, the UDF will execute and return its value to the users excel spreadsheet.

What is a UDF?  A function created by a user that can be included in an excel extension library which in turn allows it to be called from excel like any other excel function.  UDFs have been around since 1992 in Excel 4.0.  Multi-threading for UDFs was added in Excel 2007 and now Cluster-Enabled UDFs are possilbe in Excel 2010.

How do they work?  In this case, the UDFs are in XLL files (which are windows Dynamic Link Library (DLL) files with extensions to support the excel interface).  When a UDF  XLL file is loaded using the excel XLL add-in manager, the DLL will be loaded into the excel process running on the desktop.  Using the Excel API, Excel will find the names of the functions that can be typed in the spreadsheet and it will get function pointers to those functions in the library code.  XLL use is widespread including many commercially available XLL's for specific uses, particularly prevalent in the financial services industry.  The user defined functions can be very flexible.  The API supports writing functions that take a wide range of parameters (max. 30)  (discrete values, references to cells, ranges, any excel data type) and can return any excel data type.

Overview of running a (cluster enabled) UDF:

Beginning with the 2007 Microsoft Office system, Excel provided support for multi-threaded recalculation (MTR)–enabled UDFs. Office Excel 2007 was the first Excel release to support multi-threaded calculation on client computers with more than one processing core. MTR-enabled UDFs can be run in parallel on multiple processing cores, resulting in better workbook performance.  This was one of Microsoft’s first steps in providing a “scale out” method of parallelizing a spreadsheet albeit on a single computer.

Office Excel 2010 extends this model to the cluster by enabling Office Excel 2010 UDFs to run in a Windows HPC cluster. In the cluster, UDFs work much like traditional UDFs, except that the calculation is performed by one or more servers. This offers much more flexible potential than running multi-threaded MTR enabled UDF’s in that the scaling can be much greater given the ability to scale beyond a single shared memory computer.  Multiple servers can be used to evaluate functions simultaneously. As far as users are concerned, there is no difference between a client computer function and a function running in the cluster.

Pros and Cons of Cluster-Enabled UDFs

Method 2:  HPC Workbook Calculations

The second method of increasing the performance of Excel spreadsheets is to run entire workbooks on cluster compute nodes.

This method works as follows:  A user has a spreadsheet that some portion of which can be run in parallel.  From the client, the user will execute the calculation of the workbook.  Excel will send a request to the HPC scheduler to execute the workbook on the cluster.  The scheduler will then dispatch a request to one or more compute nodes.  The entire work book is sent to each selected compute node.  On each compute node, the service will start an instance of excel (a full excel process will run on the node).  It will load the excel workbook and then it will calculate the portion of the spreadsheet that the user wants to have executed in parallel.  It is essentially using excel on the compute nodes as a calculation engine.

For this method, HPC Workbook Calculations, there are two important things to keep in mind:

1. 1.      The entire workbook will be calculated on each of the cluster compute nodes
2. 2.      For this to work, the calculation(s) must be able to be partitioned in some way so they can be run in parallel and those parallel components can be re-assembled

Offloading entire workbooks can offer some unique advantages and flexibility because a user can basically run anything and do anything that they would do on the desktop - run VBA, add-ins, XLLs, UDFs.  A user can utilize external resources such as databases, files, dependent workbooks, etc. as part of the offloaded calculations.  The integrity of the calculations is assured given the same workbook is run on the cluster as is run on the desktop.  They are calculated the same way and will get the same results

Pros and Cons of HPC Workbook Calculations

The answer to this is both simple and complex.  In a nutshell, the first and most important qualifier is that HPC Services for Excel is likely an appropriate solution if a user has an Excel spreadsheet that takes too long to calculate.  How long is too long?  That completely depends on the use case.  In some cases too long may be days, in others it may be hours, in others it may be minutes.  In any case, this solution is only appropriate if some significant portion of calculation time is being taken by a portion of the spreadsheet that can be run in parallel.  HPC Services for Excel is also best suited for groups that want to support legacy Excel code and legacy models and ideally have models that are used repeatedly i.e. it may not be worthwhile to the time to create a solution that is only executed one time.

UDFs tend to be most appropriate when there is a calculation or function in the spreadsheet that is very computationally intensive and time consuming.  Ideally, it would be mostly independent and does not require interaction with external data.  Workbook calculations tend to be a good solution when the spreadsheet calculation involves complex interaction between the cells, VBA and potentially external resources.  Running Workbook calculations tend to be a bit more flexible in that you can essentially do anything you would do on the desktop on the cluster. 

All methods of offloading Excel to increase performance require a resource to offload to.  Most of what has been described has been offloading to a Microsoft Windows Server High Performance Computing (HPC) cluster.  This is the simplest approach for a user to use HPC Services for Excel, in particular if the user is part of the same corporate network as the cluster.  The implementation is simpler and there is minimal/no concern over security.  Another option is to offload to a local HPC cluster, in particular connect and offload to a local HPC Server headnode which in turn offloads the work to Azure based compute nodes.  This provides a great deal of flexibility for scaling in particular if the local HPC cluster is used heavily and doesn’t provide the resources needed to achieve the user’s desired performance.  A third option is to offload computations directly to Azure.  This requires a HPC cluster to be configured in Azure including a head node and (one or more) compute nodes.  The Excel client would connect to and offload work to the Azure cloud based HPC cluster in a similar manner in which it would utilize a local HPC cluster.  This third method is will become much simpler to implement in the near future, than it currently is, given the availability of persistent Virtual Machines and VPN capabilities in Azure.

It is important in either case that the solution fits in with user’s workflow and user’s environment as seamlessly as possible.  They should be able to run the same spreadsheet as before, with minimal interaction, just simply have it run much faster.

Using Azure for HPC Disaster Recovery

Microsoft Corporation – Azure/HPC

Overview:  The purpose of this document is to provide an overview of using Azure resources to provide an incredibly cost effective method for providing a versatile HPC disaster recovery resource.

Introduction:

Disaster recovery is a serious topic and a thorn in the side of IT organizations and CIO’s.  While being given a shrinking budget, IT organizations are also being told that risks and downtime are unacceptable creating an impossible situation!  “What if…” and “If this site goes down..” scenarios are easy for CEO’s and executive management to put on the table but they typically don’t realize the scope and impact of their need to have each scenario addressed.  Day to day IT operations with a high degree of reliability and resiliency are challenging enough and costly enough without having to have answers to the multitude of disaster “what if” questions that can be, and frankly should be, examined.

The challenge is that designing, implementing, and maintaining a traditional disaster recovery system is incredibly costly and time consuming.  A brute force, “simplistic” (which is never as simple as it sounds), approach of maintaining cold, warm, or hot redundant system(s) at an offsite location is typically out of the realm of possibility for most smaller organizations and for large companies, the expense of such a system is incredible given the actual return on investment, which is essentially a loss.  The capital expense of keeping the failover system up to date constantly to stay on par with the increasing capacity of production systems, ever increasing power and facility costs, increasing labor rates, and the software infrastructure costs can all add up to an amount that will eat up a huge chunk of an IT budget.  The value is never realized until a failure or disaster occurs and even then, its value is at best just maintaining some semblance of continuity of operations.

Just what is disaster recovery and disaster recovery planning? 

Sometimes disaster recovery planning is a subset of a larger concept of business continuity planning.  Business continuity planning will include the resumption of communications, data, hardware, applications and other IT infrastructure after a critical failure.  A disaster recovery plan typically refers to the IT related infrastructure recovery and continuity aspects of the broader business continuity plan.

What needs to be “recovered”? 

The first thing to mind when business heads think of disaster recovery is data, and rightly so.  A business’s data, its financials, internal records, customer information, transactional information, etc. are essentially the backbone of most businesses today and the loss of which could be catastrophic to continued operation and could possibly result in the demise of the business entirely.  Disaster recovery with respect to business critical data has been of such great importance for such a long period of time that most companies have implemented some form or forms of disaster recovery solution for their data whether it be offsite data replication/storage, tape/disk backups created daily then shipped out for archival, or a manager making dvd backups of his/her critical information and taking them home with him/her.  Disaster recovery is so critical and there is such a market demand for solutions that it has spawned a large industry/market segment of its own with many system providers, offsite hosted solutions, and software/hardware products available.  Many businesses have requirements that extend well beyond basic data recovery and have a critical need for some level of operational continuity and this requires a greater degree of redundant offsite IT infrastructure than a basic data recovery solution.

Cloud Computing as an Option:

The advent of cloud computing resources such as the Microsoft Windows Azure environment have provided another option for disaster recovery planners to use as part of their overall disaster recovery solution.  Cloud resources, given their “built in” resiliency, failover, and disaster recovery mechanisms provide an incredibly cost effective and powerful answer as compared to more traditional physical disaster recovery methods for data and additionally for ongoing continuity of business operation.  For companies that require more than simple data recovery, the cloud is providing a plethora of compute options not previously available to meet their IT, computation, and operational needs. 

HPC  and Disaster Recovery:

One system that can be a significant hardware expenditure with respect to the numbers of servers it requires is High Performance Computing (HPC).  Also, unless it is central to an organization’s mainstream business, HPC may not be on the “top of the list” of requirements for disaster recovery.  Between the expense of the hardware required for a traditional physical disaster recovery approach and the priority of HPC most businesses, HPC can sometimes be relegated to an afterthought and/or simply dismissed when it comes to general disaster recovery planning.

The advent of cloud computing as a key component of a strategy for disaster recovery allows a much greater latitude and ability to implement HPC as part of a robust disaster recovery model.  The ability to have essentially unlimited capacity made available immediately only paying for compute cycle’s utilization can change the entire dynamic of HPC disaster recovery (and potentially on premise HPC production). 

Example:

For example, insurance company X runs their entire portfolio risk calculations for compliance purposes at the end of each day.  Their current disaster recovery plan consists of offsite, warm standby systems for all customer data, internal financials, internal communications, and investment data.  Their planning, however, does not include the ability to run their current risk calculations that have been required for them to run as part of new set of government regulations.  Their current, onsite HPC system, includes over 200 servers and is stretched thin already with increasing demands from the actuaries, investment, and other business units.  It is based on Windows HPC Server 2008 and runs a variety of commercial ISV applications and internally developed codes.  To implement a solution that will provide the minimum capacity required to run the calculations will require a major infrastructure upgrade to their current physical disaster recovery site, requiring at least 100 additional servers that aren’t in their current budget. 

Solution:

A simple solution is to leverage Microsoft’s Azure platform as part of their disaster recovery plan to meet their HPC needs.  Microsoft’s HPC system design and Microsoft’s Azure implementation have been architected to work as a continuum of capability seamlessly allowing the use of on-premise servers as HPC compute resources as well as Azure servers as HPC compute resources – all running the same software as part of the same HPC system.  If the on-premise system compute servers fail, more Azure compute nodes could be added via a few clicks on the HPC management console.  If there is a head node failure, Azure can be used to replicate head node functionality or a head node could be kept as a backup system, local to the disaster recovery data stores, as part of an existing disaster recovery solution but connecting to Azure for compute node resources. 

The figure below depicts three basic sites; “On-Premise”, a “Physical DR (Disaster Recovery) Site”, and the “Azure” Cloud.  The dotted lines represent optional components and data paths available for disaster recovery purposes.  Almost every combination of components and paths is functionally possible and the optimal combination of which depends on the system, budgetary, and failure scenario requirements that are trying to be covered.

Benefits of leveraging Azure Cloud for HPC disaster recovery:

1. Cost savings – Probably the most important reason to leverage cloud resources as your primary option for disaster recovery HPC capability.  You pay for what you need when you need it i.e. eliminate the costs of purchasing and maintaining a separate physical plant for disaster recovery.
2. Versatile system configuration – system size, location of management/head node, data location and connectivity to HPC system, data security - levels of encryption and data persistence
3. Continual testing of compute node failure – essentially provide “hot” standby getting the value of exercising the system while performing calculations
4. Can exercise multiple data paths and configurations without major infrastructure changes – as long as a path/pipe via the internet can be established, the HPC portion of the disaster recovery plan can connect to it.  Note, even this not required if your disaster recovery implementation is entirely in the cloud.
5. Can connect to data source from either on-premise location, physical disaster recovery site, and/or cloud based recovery site
6. Leverage the benefits of Azure:  built in resiliency, robustness, updates, failover, physical security, and disaster recovery of disaster recovery (data replication and ability to leverage multiple geographic locations, etc.)

Summary:

Disaster recovery is of tantamount importance and absolutely needs to be considered, at least to a minimal degree, for every organization that leverages IT, which is essentially every modern company/organization.  Traditional Disaster Recovery methods can be clunkey, time consuming to implement, and costly.  Given the cost benefit and the other benefits/flexibility of the Azure cloud, using Azure resources as part of a robust disaster recovery plan is strongly advisable.    The ability, with a minimum of time and expense, to be able to address a multitude of “what if” disaster scenarios, is appealing for most IT managers that need to answer to a broad range of questions from managers while having to operate under a constrained budget.  High Performance Computing is system that can easily leverage Azure cloud computing for disaster recovery and is often “low hanging fruit” when considering using the cloud for disaster recovery.

General Overview:

The MATLAB Compiler uses the MATLAB Compiler Runtime (MCR), a standalone set of shared libraries that enables the execution of MATLAB files on computers without an installed version of MATLAB.  For many HPC users, this provides a means for Matlab to be executed across many nodes of a system for commonly used Matlab applications/codes that do not require interaction and are used repeatedly.

 The other, more traditional method of running parallel Matlab on a cluster, is to use the Mathworks “Parallel Computing Toolkit” which is part of the MATLAB Distributed Computing Server (MDCS) environment.  This type of architecture is more robust in particular for development and single/short time execution of a model(s). It allows the user to run their parallel MATLAB applications from within the MATLAB client.

USING MATLAB ON AN HPC CLUSTER VIA COMPILED MATLAB CODE AND THE
MATLAB Compiler Runtime (MCR):

For many users that want to run a matlab code repeatedly and/or at large scales on an HPC cluster, they choose to run compiled Matlab applications leveraging the MCR. 

The general work flow for this type of operation is as follows:

1. A user will generate, test, and compile their Matlab code using the Matlab client on their desktop/workstation.
2. To run on the HPC cluster, the user will copy (if not available to the cluster via some type of share storage) their Matlab executable and data they wish to execute to the HPC system.
3. They will submit their application to the HPC cluster as a “Job” which will consist of one or more “tasks”  (note, for an Windows HPC cluster a “job” is comprised of one or ore “tasks” which are individual groups of work – usually an application/script that operates on a set of data).
4. The HPC scheduler will execute the user’s “Job” which includes their Matlab executable as a task or a set of tasks (usually just the command line .exe of their Matlab code) on the compute resources the user requested as part of their job submission.
5. The compiled Matlab executables are then initiated and run on each compute node, leveraging the locally installed MATLAB Compiler Runtime (MCR), typically enerating output files to local storage.
6. Depending on configuration of the system, the output data will be collected/moved to a shared storage device (if not already writing to a shared device), the head
   node, and generate another form of output.
7. User collects results, many times copying to their local machine for further analysis and data reduction.

General Setup:

1. Install the MCR on each of the nodes (many ways to do this)
1. Use ClusterCopy to copy installer to each node and then run the installer silently basic on the Mathworks instructions,
2. RDP directly to each node, copy and install,
3. Ceate an HPC job with a script to copy and install the MCR silently,
4. Use a fileshare that is accessible by each node.
2. Copy input data and compiled matlab application to each node (or have them available to each node via a fileshare).
3. Execute via clusrun utility ( as shown in the link http://technet.microsoft.com/en-us/library/cc947685(WS.10).aspx) and/or submit job/task to the scheduler via
   gui, command line, or powershell (as shown in the link http://technet.microsoft.com/en-us/library/ee783562(WS.10).aspx).