Here’s what is new in the refresh:

Improved user experience: through more actionable exception messages, a refactoring of the probability distribution function APIs, and better and more actionable feedback in the deployment utility. In addition, the deployment process time has decreased and the installer supports installation on a on-premises Windows HPC Cluster. All up, this refresh provides for a more efficient way of writing and deploying “Cloud Numerics” applications to Windows Azure.

More scale-out enabled functions: more algorithms are enabled to work on distributed arrays. This significantly increases the breadth and depth of big data algorithms that can be developed using “Cloud Numerics” Lab. Scale-out functionality was added in the following areas: Fourier transforms, linear algebra, descriptive statistics, pattern recognition, random sampling, similarity measures, set operations, and matrix math.

Array indexing and manipulation: a large part of any data analytics application concerns handling and preparing data to be in the right shape and have the right content. With this refresh “Cloud Numerics” adds advanced array indexing enabling users to easily and efficiently set and extract subsets of arrays and to apply Boolean filters.

Sparse data structures and algorithms: much of the real-world big data sets are sparse —not every field in a table has a value. With this refresh of the lab we introduce a distributed sparse matrix structure to hold these datasets and introduce core sparse linear algebra functions enabling scenarios such as document classification, collaborative filtering, etc.

Apply/Sweep framework: in addition to the built-in parallelism the “Cloud Numerics” Lab, this refresh now exposes a set of APIs to enable embarrassingly parallel patterns. The Apply framework enables applying arbitrary serializable .NET code to each element of an array or to each row or column of an array. The framework also provides a set of expert level interfaces to define arbitrary array splits. The Sweep framework performs as its name implies —this framework enables distributed parameter sweeps across a set of nodes allowing for better execution times.

Improved IO functionality: we added more parallel readers to enable out of the box data ingress from Windows Azure storage and introduced parallel writers.

Documentation: we introduced detailed mathematical descriptions of more than half of the algorithms using print-quality formulae and best-of-web equation rendering that help clarify algorithm mathematical definition and method behavior. In addition, we updated the “Getting Started” wiki, and we added conceptual documentation for the “Cloud Numerics” help that includes the programming model, the new Apply framework, IO, and so on.

Stay tuned for upcoming blog posts:

• F#: We’ll be distributing a F# add-in for “Cloud Numerics” soon. The add-in exposes the “Cloud Numerics” APIs in a more functional manner, introduces operators, such as matrix multiply, and F# style constructors for and indexing on “Cloud Numerics” arrays.

• Text analytics using sparse data structures

Do you want to learn more about Microsoft Codename “Cloud Numerics” Lab? Please visit us on our SQL Azure Labs home page, take a deeper look at the Getting Started material and Sign Up to get access to the installer. Let us know what you think by sending us email at cnumerics-feedback@microsoft.com.

The “Cloud Numerics” refresh depends on the newly released Azure SDK 1.7 and Microsoft HPC Server R2 SP4. It does not provide support for the Visual Studio 2012 RC.

In this blog post we’ll introduce Cloud Numerics F# Extensions, a companion post to Microsoft Codename “Cloud Numerics” Lab refresh. Its purpose is to make it easier for you as an F# user to write “Cloud Numerics” applications, and it does so by wrapping “Cloud Numerics” .NET APIs to provide an F# idiomatic user experience that includes array building and manipulation and operator overloads. We’ll go through the steps of setting up the extension as well as a few examples.

The Visual Studio Solution for the extensions is available at the Microsoft Codename “Cloud Numerics” download site.

Set-Up on Systems with Visual Studio 2010 SP1

First, install “Cloud Numerics” on your local computer. If you only intend to work with compiled applications, that completes the setup. For using F# Interactive, a few additional steps are needed to enable fsi.exe for 64-bit use:

In the Programs menu, under Visual Studio Tools, right click Visual Studio x64 Win64 command prompt. Run it as administrator. In the Visual Studio tools command window, specify the following commands:

Open Visual Studio

• From the Tools menu select Options, F# Tools, and then edit the F# Interactive Path.
• Set the path to C:\Program Files (x86)\Microsoft F#\v4.0\fsi64.exe.
• Restart F# Interactive.

Set-Up on Systems with Visual Studio 2012 Preview

With Visual Studio 2012 preview, it is possible to use “Cloud Numerics” assemblies locally. However, because the Visual Studio project template is not available for VS 2012 preview, you’ll have to use following procedure to bypass the installation of the template:

• Install “Cloud Numerics” from command prompt specifying “msiexec –i MicrosoftCloudNumerics.msi CLUSTERINSTALL=1” and follow the instructions displayed by the installer for any missing prerequisites.
• To use F# Interactive, open Visual Studio. From the Tools menu select Tools, Options, F# Tools, F# Interactive, and set “64-bit F# Interactive” to True.

Note that this procedure gives the local development experience only; you can work with 64-bit F# Interactive, and build and run your application on your PC. If you have a workstation with multiple CPU cores, you can run a compiled “Cloud Numerics” application in parallel, but deployment to Windows Azure cluster is not available.

Using Cloud Numerics F# Extensions from Your Project

To configure your F# project to use “Cloud Numerics” F# extensions:

• Create F# Application in Visual Studio
• From the Build menu, select Configuration Manager, and change the Platform attributes to x64

• In the Project menu, select the Properties <your-application-name> item. From your project’s application properties tab, ensure that the target .NET Framework is 4.0, not 4.0 Client Profile.

• Add the CloudNumericsFSharpExtensions project to your Visual Studio solution
• Add a project reference from your application to the CloudNumericsFSharpExtensions project
• Add references to the “Cloud Numerics” managed assemblies. These assemblies are typically located in C:\Program Files\Microsoft Cloud Numerics\v0.2\Bin

• If you plan to deploy your application to Windows Azure, right-click on reference for FSharp.Core, and select Properties. In the properties window, set Copy Local to True.
• Finally, you might need to edit the following in your .fs source file.
  -  The code within the #if INTERACTIVE … #endif block is required only if you’re planning to use F# Interactive.
  -  Depending on where it is located on your file system and whether you’re using Release or Debug build, you might need to adjust the path specified to CloudNumericsFSharpExtensions.

Using Cloud Numerics from F# Interactive

A simple way to use “Cloud Numerics” libraries from F# Interactive is to copy and send the previous piece of code to F# Interactive. Then, you will be able to use create arrays, call functions, and so forth, for example:

Note that when using F# Interactive, the code executes in serial fashion. However, parallel execution is straightforward as we’ll see next.

Compiling and Deploying Applications

To execute your application in parallel on your workstation, build your application, open Visual Studio x64 Win64 Command Prompt, and go to the folder where your application executable is. Then, launch a parallel MPI computation using mpiexec –n <number of processes> <application executable>.

Let’s try the above example in parallel. The application code will look like

You can use the same serial code in parallel case. We then run it using mpiexec to get the result:

Finally, to deploy the application to Azure we’ll re-purpose the “Cloud Numerics” C# Solution template to get to the Deployment Utility:

1. Create a new “Cloud Numerics” C# Solution
2. Add your F# application project to the Solution
3. Add “Cloud Numerics” F# extensions to the Solution
4. Set AppConfigure as the Start-Up project
5. Build the Solution to get the “Cloud Numerics” Deployment Utility
6. Build the F# application
7. Use “Cloud Numerics” Deployment Utility to deploy a cluster
8. Use the “Cloud Numerics” Deployment Utility to submit a job. Instead of the default executable, select your F# application executable to be submitted

Indexing and Assignment

F# has an elegant syntax for operating on slices of arrays. With “Cloud Numerics” F# Extensions we can apply this syntax to distributed arrays, for example:

Operator Overloading

We supply operator overloads for matrix multiply as x *@ y and linear solve of a*x=b as let x = a /@ b . Also, operator overloads are available for element-wise operations on arrays:

• Element-wise power: a.**b
• Element-wise mod a.%b
• Element-wise comparison: .= , .< , .<> and so forth.

Convenience Type Definitions

To enable a more concise syntax, we have added shortened definitions for the array classes as follows:

• LNDA<’T> : Microsoft.Numerics.Local.DenseArray<’T>
• DNDA<’T> : Microsoft.Numerics.Distributed.NumericDenseArray<’T>
• LNSM<’T> : Microsoft.Numerics.Local.SparseMatrix<’T>
• DNSM<’T> : Microsoft.Numerics.Distributed.SparseMatrix<’T>

Array Building

Finally, we provide several functions for building arrays, for example from F# sequences or by random sampling. They wrap the “Cloud Numerics” .NET APIs to for functional programming experience. The functions are within 4 modules:

• LocalDense
• LocalSparse
• DistDense
• DistSparse

These modules include functions for building arrays of specific type, for example:

Building and Running Example Projects

The “Cloud Numerics” F# Extensions has a folder named “Examples” that holds three example .fs files, including:

• A set of short examples that demonstrate how to invoke different library functions.
• A latent semantic analysis example that demonstrates computation of correlations between different documents, in this case, SEC-10K filings of 30 Dow Jones companies.
• A air traffic analysis example that demonstrates statistical analysis of flight arrival and delay data.

The examples are part of a self-contained solution. To run them:

• Copy the four .csv input data files to C:\Users\Public\Documents (you can use a different folder, but you will need to adjust the path string in the source code to correspond to this folder).
• In Solution Explorer, select an example to run by moving the .fs files up and down.
• Build the example project and run it using mpiexec as explained before.

You can also run the examples in the F# interactive, by selecting code (except the module declaration on the first line) and sending it to F# interactive.

This concludes the introduction to “Cloud Numerics”  F# Extensions. We welcome your feedback at cnumerics-feedback@microsoft.com.

You may recall our earlier blog post about Latent Semantic Indexing and Analysis, where we analyzed a few hundred documents. With new features of “Cloud Numerics” Wave 2 we can repeat the analysis: not just for few hundred but for many thousands of documents. The key new features we will use are distributed sparse matrices and singular value decomposition for dimensionality reductions of sparse data. We’ll then use DataParallel.Apply to compute the cosine similarities between documents.

The dataset we use is derived from SEC-10K filings of companies for the year of 2010. There are 8944 documents in total; the number of terms is over 100,000. We have computed the tfidf term-document matrix and stored it as triplets (term number, document number, tfidf frequency). This matrix was computed using Hadoop and Mahout from pre-parsed filings, and then saved into a regular CSV file. We will use the “Cloud Numerics” CSV reader to read in this data. For those interested, “Cloud Numerics” can also read the triplets from the HDFS (Hadoop Distributed File System) format directly. This is a topic for another blog post.

Step 1: Preparation

To run the example, we recommend you deploy a Windows Azure cluster with two compute nodes, which you can do easily with the “Cloud Numerics” Deployment Utility. Alternatively, if you have a local computer with enough memory –-more than 8 GB free-– you can download the files (see next step) and attempt the example on that local computer.

We’ll be creating the application --method by method-- in the steps below.

1.  Open Visual Studio, create a new Microsoft “Cloud Numerics” Application, and call it SparseLSI.

2.  In the Solution Explorer frame, find the MSCloudNumericsApp project and add a reference to Microsoft.WindowsAzure.StorageClient assembly. We’ll need this assembly to read and write data to and from Azure storage.

3.  Next, copy and paste the following piece of code to the Program.cs file as the skeleton of the application.

Step 2: Copy Data to Your Own Storage Account

The input data for this application is stored at cloudnumericslab storage account:

https://cloudnumericslab.blob.core.windows.net/lsidata/SEC_10K_2010.csv 

and:

 https://cloudnumericslab.blob.core.windows.net/lsidata/Docmapping_2010.csv

The methods we will use to read in data to distributed arrays do not support public access to blobs, so you must copy the data to your own Windows Azure storage account.

To do this, let’s add following methods to the “Cloud Numerics” application:

This routine instantiates a CloudBlobClient instance, given an account name and key:

This method enables your application to download and upload blobs between storage accounts:

The third method copies two blobs: the CSV file that holds the term-document matrix, and a list of document names that maps term-document matrix columns to specific companies.

Step 3: Read Data from the CSV File into a Sparse Matrix

Create a method that reads data in “triplet” format (row, column, value), slices it into three vectors, and then assembles it into a sparse matrix. As an input argument, our reader will take a class that implements the IParallelReader interface, because this will give us flexibility to use the same method for data stored locally or on Azure (this point becomes more clear in Step 9). The triplet data is shaped n-by-3 where n is total number of unique words in each document of the corpus. The slicing operation yields first, second, and third column as separate vectors, where the first two correspond to indices of terms and documents, respectively. To convert them into row and column indices, we cast the first two into long integers and then call the distributed sparse matrix constructor. Add the following method to the skeleton program from previous step.

Step 4: Read List of Documents

From the sparse matrix alone we cannot tell how columns map to filings by specific companies. To provide this mapping, we read in an indexed list of documents. This list was originally produced during the pre-processing of documents.

The following method reads the data file from Windows Azure to a string:

This method converts the string into list of strings, one for each document:

Step 5: Compute Sparse Singular Value Decomposition

The dimensionality of the term-document matrix is huge. In a sense, each document can be considered as a vector in t-dimensional space, where t is the number of words --over 100,000-- in the corpus. Also, the term-document matrix is very sparse, each document uses only a small subset of all available words. We, therefore, can reduce the dimensionality of the problem by applying singular value decomposition.

This SVD operation has three major effects that facilitate subsequent analysis by:

The number of SVD components determines the reduced dimensionality. In this example we’ll specify a value of 100; you can experiment with different values. The output is a V*S matrix in reduced-dimensional subspace, which we will use for further analysis.

Step 6: Compute Cosine Similarities Between Documents

At this stage, each document is represented as a row vector in V*S matrix. Now, we’d like to find which documents are most similar to a specific document of interest.

To compare the documents we’ll use a cosine similarity measure. Let’s first add a class with a method that computes cosine similarity between a pair of documents. Note that the first vector is supplied as an argument to the constructor. The reason for this design is that we can then pass the cosine similarity method as an argument to the DataParallel.Apply(), to compute the similarity between a single document of interest and all other documents in parallel. We also mark the class as serializable so it can be passed to DataParallel.Apply().

Then, we’ll add a method that first looks up the row index of document of interest and gets the slice (row vector) corresponding to that document. It then instantiates SimilarityComparer using that vector, and calls DataParallel.Apply with SimilarityComparer.CompareSimilarity as the method to execute in parallel against rows of the V*S matrix. We sort the results by their similarity comparison score, select the top 10, and write the result to a string.

Step 7: Write out Result

We write the results to a Windows Azure blob as a text file. The blob is made public, so you can view the results using URI http:\\<myaccount>.cloudapp.net\lsioutput\lisresult.

Step 8: Put it All Together

Finally, we put together the pieces in a Main method of the application: we specify the input locations, load in data, reduce the term-doc matrix, select Microsoft Corp. as a company of interest, and find similar documents. We also added a Boolean “useAzure” to provide for the optional scenario where local input and output of data is needed for cases where you want to run the application on your workstation, not on Windows Azure.

Note that the strings accountName, accountKey need to be changed to hold the name and key of your own Windows Azure storage account. In case you intend to run on local workstation, ensure that localPath points to the folder that holds the input files.

Once all pieces are together, you should be ready to build and deploy the application. In Visual Studio’s Solution Explorer, Set AppConfigure Project as the StartUp Project, build and run the application, and use the “Cloud Numerics” Deployment Utility to deploy to Windows Azure. After a successful run, the resulting cosine similarities should look like the following:

The algorithm has produced a list of technology companies as one would expect. As a sanity check, note that MICROSOFT_CORP_2010 has similarity score of 1 with itself.

This concludes the example; the full example code is available as a Visual Studio 2010 solution at Microsoft Codename “Cloud Numerics” download site.

You may recall we posted about using Cloud Numerics and F# for distributed computing on Windows Azure a while back. In the tutorial we’ll give hands-on examples of using libraries for math, linear algebra and statistics, and present demos of data analysis using Cloud Numerics and F# on Azure.

This is a great opportunity to learn about Cloud Numerics; don’t worry if you’re not an Azure user yet. You can try out the hands-on examples with distributed arrays and libraries using your local development environment.

Update: Code examples for our tutorial are available for download.

Introduction

The LinearAlgebra namespace in “Cloud Numerics” is divided into 3 main classes:

• Microsoft.Numerics.LinearAlgebra.Decompositions
• Microsoft.Numerics.LinearAlgebra.LinearSolvers
• Microsoft.Numerics.LinearAlgebra.Operations

In this post we’ll look at examples from each of the classes, and how they can be combined to implement interesting algorithms on matrices.

Operations

Let’s kick off the tour with an example using methods from the Operations class. We compute the largest eigenvector of a matrix by power iteration. We use Operations.MatrixMultiply to iterate the result, and Operations.Norm, first to scale the result to unit length, and then to check the convergence 

For normalization we use the L2 norm, by passing the NormType.TwoNorm argument value.

For exact eigenvalues, the identity  A*x=e*x should hold. We check this by comparing  x and xnew: at convergence they should be close to one another. We perform the comparison by using NormType.InfinityNorm that gives the largest magnitude element-wise difference.

The algorithm should reach convergence after about 8 iterations.

Matrix decompositions

The decompositions, broadly speaking, fall into two categories

1. Decompositions that facilitate solving systems of linear equations, for example LU and Cholesky
2. Decompositions aimed at analyzing the spectrum of a matrix, for example EigenSolve and Svd

The workflow for category 1 involves computing the decomposition and passing the result object into the appropriate linear solver function. For example, one can use Cholesky decomposition as

Because this category of methods is closely related to linear solvers, we’ll take a closer look in next section “Linear solvers”

The methods from category 2 can be used to represent a matrix in a form that can be analyzed more easily. Let’s try an example: singular value decomposition to compute a low-rank approximation of a matrix. In the following code, we decompose the matrix and then reconstruct it, first by keeping only the largest singular value, and then adding second largest value, and so forth.

We compare the reconstructed matrix to the original one by computing the Frobenius norm of the difference. As we increase the number of singular values, the norm gets smaller and smaller as the reconstructed matrix gets closer and closer to the original one. For all 10 singular values the result should be exactly the same, apart from small numerical noise.

Note that we are using the Decompositions.SvdThin method. This is more memory-efficient than Svd for non-square matrices. To give an example, if your data was a tall-and-thin 1 million  by 100 matrix of doubles, the resulting U matrix would be a huge million by million array that consumes 8 terabytes of memory, with 999900 rows filled with zeros. However, SvdThin keeps only the 100 non-zero rows, producing a much more manageable 0.8 gigabyte array.

Linear solvers

There are two kinds of operations: general-purpose ones and specialized ones that can be much faster if your data has special symmetries. Consider, for example, Cholesky decomposition. If your data happens to be a symmetric positive-definite matrix, it’s much faster to compute the Cholesky decomposition first, and then use the LinearSolvers.Cholesky method to solve the linear system of equations, compared to using the general-purpose LinearSolvers.General method.

There’s another benefit to using specialized methods. Consider what LinearSolvers.General does under the covers: it first computes a matrix factorization internally and then solves the system of linear equations against the factorized matrix. Often, this internal factorization step is the expensive part of the computation. The factorization only depends on matrix A. If you have to compute linear solve many times against the same matrix - for example, in iteration A*x2 = f(x1), A*x3 = f(x2)  - it would be nice to be able to re-use the same factorization. Using the specialized method allows you to do exactly that: pre-compute the Cholesky decomposition once and then re-use it as input to LinearSolvers.Cholesky.

Let’s look at an example to compare the performance of the two approaches. We’ll first need to generate a matrix that is guaranteed to be symmetric and positive-definite. First, we matrix multiply an n-by-1 and a 1-by-n random number array to create an n-by-n distributed matrix: this is an efficient way to create a large distributed matrix. Second, we add together the matrix and its transpose to create a symmetric matrix. Finally, we increase the values on the diagonal to make the matrix positive-definite: this follows from the Gershgorin circle theorem.

We’ll time 5 repeated computations of LinearSolve.General.

Next, we perform the same computation using Cholesky decomposition: we store the decomposition and re-use it as input to the LinearSolvers.Cholesky method. Note that because the matrix is assumed symmetric, the Cholesky method only uses  values in its upper or lower triangle, as specified by the UpperLower argument.

The Cholesky code should be 6-7 times faster.

Examples as single code

This concludes the our tour of the LinearAlgebra namespace. To try the examples yourself, you can create a Cloud Numerics application and copy-paste the code below:

“Microsoft Codename "Cloud Numerics" is a SQL Azure Lab that lets you model and analyze data at scale. Now when you want to upload a file with Microsoft Codename "Data Transfer" to Windows Azure Blob storage for use in Cloud Numerics, you can choose to have the file converted to the Numerics Binary Format. This only applies to CSV and Excel files that contain numerical data ready for analysis with “Cloud Numerics.”

To run the example application in the blog post you will need the implementation of the SequenceReader used in the example. You can download the sample reader from the “Microsoft Numerics” Connect Site (note that you need to sign up to our Connect program first). See links for Sign-up / Download.

You sit at the airport only to witness your departure time get delayed. You wait. Your flight gets delayed again, and you wonder “what’s happening?” Can you predict how long it will take to arrive at your destination? Are there many short delays in front of you or just a few long delays? This example demonstrates how you can use “Cloud Numerics” to sift though and calculate a big enough cross section of air traffic data needed to answer these questions. We will use on-time performance data from the U.S. Department of Transportation to analyze the distribution of delays. The data is available at:

https://www.transtats.bts.gov/DL_SelectFields.asp?Table_ID=236&DB_Short_Name=On-Time

This data set holds data for every scheduled flight in the U.S. from 1987 to 2011 and is —as one would expect— huge! For your convenience, we have uploaded a sample of 32 months —one file per month with about 500,000 flights in each— to Windows Azure Blob Storage at container URI: https://cloudnumericslab.blob.core.windows.net/flightdata (note that you cannot access this URI directly in a browser, you must use a Storage Client C# APIs as in Step 2, or a REST API query).

Before You Begin

You should use two-to-four compute nodes when deploying the application to Windows Azure. One node might not have enough memory, and for larger-sized deployments there are not enough files in the sample data set to assign to all distributed I/O processes. You should not attempt to run the application on a local system because of data transfer and memory requirements.

Step 1: Create Cloud Numerics project

First, let’s create a Cloud Numerics application using the Cloud Numerics Visual Studio project template.

1.  Start Microsoft Visual Studio and create a solution from the Cloud Numerics project template. Let’s call the solution “OnTimeStats.”

2.  Create a new application, and delete the sample code from within the MSCloudNumericsApp subproject’s Program.cs file. Replace it with the following skeleton of an application.

3.  Add a .NET reference to Microsoft.WindowsAzure.StorageClient. This assembly is used for reading in data from blobs, and writing results.

Step 2: Add Methods for Reading Data

We’ll use the IParallelReader interface to read in the data from blob storage in the following manner:

• In the ComputeAssignment method, we must first list the blobs in the container that holds the flight data (https://cloudnumericslab.blob.core.windows.net/flightdata). Then, we assign blobs to each worker node, in round-robin fashion.
• In the ReadWorker method, we instantiate a list “arrivalDelays”. Each reader will then open the first blob on their list, download the text, break it to lines, break the line to columns, and select column 44 that holds the arrival delays in minutes. Note that if a flight was canceled or diverted, this value is blank and we must skip the row. We convert the result to a double precision type and append it to the arrivalDelays list. Then, we either move to the next blob or –if done– convert the list into NumericDenseArray and return it.

For Step 2, add the following code to the FlightInforReader class in your skeleton application.

Step 3: Implement the Algorithm

After reading in the delays, we compute mean, we center the data to the mean (to make the worse-than-average delays positive and better-than-average ones negative), and then compute sample standard deviation. Then, to analyze the distribution of data, we count how many flights, on average, are more than k standard deviations away from the mean. Also, we keep track how many flights are above or below to mean, so as to detect any skew in data.

For example, if the data were normal distributed one would expect it to be symmetric around the mean, and obey the three-sigma rule —or, equivalently, about 16% of flights would be delayed by more than 1 standard deviation, 2.2 % by more than 2 standard deviations and 0.13% by more than 3.

To implement the algorithm, we add following code as the Main method to the application.

Step 4: Write Results to a Blob

We add the WriteOutput method that writes results to binary large object (blob) storage as .csv-formatted text.  The WriteOutput method will create a container “flightdataresult” and a blob “flightdataresult.csv.” You can then view this blob by opening your file in Excel. For example: https://<myAccountName>.blob.core.windows.net/flightdataresult/flightdataresult.csv

--Where <myAccountName> is the name of your Windows Azure account.

Let’s fill in the last missing piece from the application, the WriteOutput method, with following code.

Note that you’ll have to change "myAccountKey" and "myAccountName" to reflect your own storage account key and name.

Step 5: Deploy the Application and Analyze Results

Now, you are ready to run the application. Set AppConfigure as the startup project, build the application, and use the Cloud Numerics Deployment Utility to deploy and run the application.

Let’s take a look at the results. We can immediately see they’re not normal-distributed at all. First, there’s skew —about 70% of the flight delays are better than average of 5 minute. Second, the number of delays tails off much more gradually than a normal distribution would as one moves away from the mean towards longer delays. A step of one standard deviation (about 35 minutes) roughly halves the number of delays, as we can see in the sequence 8.5 %,  4.0 %, 2.1%, 1.1 %, 0.6 %. These findings suggests that the tail could be modeled by an exponential distribution.

This result is both good news and bad news for you as a passenger. There is a good 70% chance you’ll arrive no more than five minutes late. However, the exponential nature of the tail means —based on conditional probability— that if you have already had to wait for 35 minutes there’s about a 50-50 chance you will have to wait for another 35 minutes.

In this application example, we use “Cloud Numerics” to analyze demographics data such as average household size and median age between different postal (ZIP) codes in the United States.

This blog post demonstrates the following process for analyzing demographics data:

Before You Begin

Because of the memory usage, do not attempt to run this application on your local development machine. Additionally, you will need to run it on at least two compute nodes on your Windows Azure cluster.

Step 1: Subscribe to the Demographics Dataset at Windows Azure Marketplace

For this demonstration, we use the dataset “2010 Key US Demographics by ZIP Code, Place and County (Trial)”

To subscribe to the dataset:

1. Sign up to Windows Azure Marketplace using your Windows Live ID at: https://datamarket.azure.com/
2. Navigate to the page for the dataset: https://datamarket.azure.com/dataset/c7154924-7cab-47ac-97fb-7671376ff656
3. Select Sign Up. Then click the Details tab of the dataset, and take note of the Service Root URI https://api.datamarket.azure.com/Esri/KeyUSDemographicsTrial/ . This URI will be used in creating the service reference in the next step.
4. Finally, go to My Account, Account Keys and take note of the long Value string. You will need this string in the next step to access the dataset from the C# application.

Step 2: Set Up the Cloud Numerics Project

To set up your Cloud Numerics Visual Studio project:

1. Start Microsoft Visual Studio and create a solution from the Cloud Numerics project template. Let’s call the solution “CloudDemographics.”
2. Rename the “MSCloudNumerics” project with more descriptive language. For the purposes of this blog post, we will name it “Demographics.”
3. Next, add a new project to the solution as a C# class library. This project will have the code for reading data. let’s call the project “DemographicsReader.”

We now need to add project dependency from Demographics to DemographicsReader.

1. In Solution Explorer, under the Demographics project, right-click References and select Add References.
2. Click the Projects tab, and browse and add DemographicsReader as a reference.
3. Also add a .NET reference for Microsoft.WindowsAzure.StorageClient. We’ll need it later on when it’s time to write results to blob storage.

Finally, go to the DemographicsReader project, and add references to the following managed assemblies. In the Add References window, click Browse and find the assemblies, typically found at C:\Program Files\Microsoft Numerics\v0.1\Bin

• Microsoft.Numerics.DenseArrays
• Microsoft.Numerics.Distributed.IO
• Microsoft.Numerics.DistributedDenseArrays
• Microsoft.Numerics.ManagedArrayImpl
• Microsoft.Numerics.Runtime
• Microsoft.Numerics.TCDistArrayHandle

Step 3: Add Service Reference to Dataset

To get data into the application, we need to add a service reference to the dataset.

1. In Solution Explorer, right-click the DemographicsReader project and select Add Service Reference.
2. In the dialog window, enter https://api.datamarket.azure.com/Esri/KeyUSDemographicsTrial/ as the address. Also, specify “USDemographics” as the namespace. Adding the reference provides auto-generated flexible query methods that reside in the DemographicsReader.USDemographics namespace and allow us to access the dataset.

3.  Add the following “using” statements to the DemographicsReader project’s Class1.cs file, and declare a namespace “DemographicsReader” that will contain the classes for accessing the demographics data.

4.  Add a class named  “DemographicsReference” to this namespace with the following private members and a constructor.

5.  Replace "myemail@live.com”  with your Live ID and “myAzureMarketplaceKey” with the key from Step 1.

Step 4: Create a Serial Reader for Windows Azure Marketplace Data

Next, let’s build the LINQ query operation for reading in the data. A single transaction returns a maximum of 100 rows. However, we want to read in much more data. Therefore, we create a list where we append and consolidate the individual query results. We structure the query so that we can start from a given row, read in a block of rows, move forward, read in another block, and so forth.

The query, by default, returns objects of type demog1. This is a class that holds the demographics data as fields. However, we want to be able to select an arbitrary subset of columns. We accomplished this by supplying Func<demog1, T> selector function into the query as an argument.

Add the following method to the DemographicsReference<T> class

Step 5: Parallelize Reader Using IParallelReader Interface

Now, we have the basic framework in place to read in data. Next, we implement the Cloud Numerics IParallelReader interface to read in the data to a distributed array in parallel fashion.

Add the following class “DemographicsData” to the DemographicsReader namespace.

The class constructor expects the number of dataset rows (“samples”) as input. The method “ComputeAssignment” divides the total number of rows into groups (referred to here as “blocks”) and attempts to distribute these blocks evenly between ReadWorkers. The “ReadWorker” method is executed in parallel fashion: there is one parallel call in each rank. The method expects as input a start index and a specified number of blocks to read. For example, the first worker might read rows 0-999, the second one rows 1000-1999 and so on. Then, each ReadWorker converts the demographics indicators from the query results into array columns. The arrays from each ReadWorker are then concatenated to form a large distributed array.

See the blog post titled “Cloud Numerics” Example: Using the IParallelReader Interface for more details on how to use the IParallelReader interface.

Step 6: Add Method for Getting Mapping from Rows to Geographies

Because the resulting distributed array is all numeric, we need an auxiliary list to map array rows to geographies. We accomplish this by adding a serial reader named “GetRowMapping.” We also add a struct “ZipCodeAndLocation” that holds the ZIP Code, city name, and state as well as the row index. The GetRowMapping then returns a list of such structs.

1.  First, add the following struct to the top-level of the DemographicsReader namespace

2.  Next, add a method GetRowMapping to the DemographicsData class from the previous step.

Here we use a selector that only picks out the GeographyID (the ZIP code), GeographyName (the city name), and StateAbbreviation from the full query. Then, because the results are in the same sorted order as those from the earlier demographics query, we can use a simple for-loop to add an index.

Step 7: Compute Correlation Matrix Using Distributed Data

Next, we’ll move to the Demographics project. We add a static method that expects a distributed array as input, normalizes the rows and columns, and then computes a correlation matrix. The columns have to be normalized because otherwise they would be on vastly different scales: for comparing per capita income and median age, for example. We do this by subtracting the mean of each column, and dividing by the standard deviation. Note how the KroneckerProduct method is used to tile vectors into an array that matches the data array size. The rows are then normalized to obtain the correlation coefficients between –1 and 1.

To get the correlation matrix between all ZIP codes, we need to compute correlation coefficients between each pair of rows. An efficient way to perform this computation is to multiply the data with its own transpose. This operation is at the heart of the application. Because it produces a very large matrix (30000-by-30000), performing the computation on a distributed cluster yields benefits.

Delete the default template code from Demographics project source code, and add the following:

In addition to the Correlate method, this code includes the necessary “using” statements for Cloud Numerics namespaces.

Step 8: Select Interesting Subset of Cities and Write the Correlations into a Blob

We store results to Windows Azure blob storage as a blob resembling a .csv formatted file. To save time and memory, we only write a small subset of the 30000-by-30000 correlation matrix to a blob. Add the following method to the Demographics Program class. Also, change “myAccountKey” and “myAccountName” to the key and name of your own storage account.This method takes a list of ZipCodeAndLocation objects we’re interested in, finds the corresponding rows and columns from the correlation matrix, and writes the results to a string. Then it creates a container “demographicsresult” and a blob, “demographicsresult.csv”, makes the blob publicly readable, and uploads the string to https://<myAccountName>.blob.core.net/demographicsresult/demographicsresult.csv .

Step 9: Putting it All Together

Finally, we add a main method that instantiates the reader, reads in data, computes correlations, and selects the interesting subset of cities from the row mapping. For example, let’s select Cambridge, MA, Redmond, WA, and Palo Alto, CA. Add the following main method to the Demographics Program class.

Now, the application is ready to be deployed. Set AppConfigure as StartUp project, build and deploy using the Deployment Utility. The application will run for a few minutes. The results can be viewed, for example by opening https://<myAccountName>.blob.core.net/demographicsresult/demographicsresult.csv as a spreadsheet

Here the correlation coefficients above 0.9 between different ZIP codes are highlighted.

Source Code

For your convenience, this section of the blog post contains the entire source code for you to copy. Remember to change the “myAccountKey” and “myAccountName” strings to your storage account key and name, and change “myemail@live.com” and “myAzureMarketplaceKey” to your Live ID and key from Step 1.

Demographics Program.cs

DemographicsReader Class1.cs

Frequently Asked Questions

Best Practices

Known Issues

Frequently Asked Questions (FAQs)

Can I Use “Cloud Numerics” Libraries and Runtime with F#?

Yes. See the blog post titled “Cloud Numerics” Example: Distributed Numerics on Azure with F#.

Can Multiple Applications Use the Same Windows Azure Storage Account?

Yes, but you need to ensure that these applications have distinct project names, otherwise, each will overwrite the other application’s output. The post build event of a Microsoft codename “Cloud Numerics” C# project uses the project name to create an Azure blob. Identical project names using the same Windows Azure storage account will map application output to the same blob.

If multiple users use the same Windows Azure storage account while creating “Cloud Numerics” applications for Azure by way of the “Cloud Numerics” Deployment Utility, the applications must use different project names.

Can I Share an Existing “Cloud Numerics” Solution with Another User?

To share your “Cloud Numerics” Visual Studio solution with another user, a small modification must be made to the solution’s properties in order for the other user to execute the compiled solution locally. The reason for this is that the solution has fully-qualified paths in its application properties, and these paths reflect the directory structure of the development machine where the application was originally developed. For example, the following screen shows fully-qualified directory paths that descend into a user’s home directory.

To change these path specifications so that a different user can run the compiled solution locally on a different machine, follow these steps.

To change the Start Option paths to reflect your local machine:

1. From Solution Explorer, right-click the MSCloudNumericsApp and select Properties.
2. Click the Debug tab.
3. In the Start Options section of the Debug tab, edit the following fields to reflect the paths on your local development machine:

a.  For the Command line arguments field, change the path to reflect the directory on the current machine where the “Cloud Numerics” Projects directory can be found.

For example, given the figure above, you would change: 
     c:\users\roastala\documents\visual studio 2010\Projects…

     to: 
     c:\users\<YourUsername>\documents\visual studio 2010\Projects…

b.  For the Working directory field, change the path to reflect the directory on the current machine where the “Cloud Numerics” Projects directory can be found.

For example, given the figure above, you would change:

     c:\users\roastala\documents\…

     to:
     c:\users\<YourUsername>\documents\…

     --Where c:\users\<YourUsername>\ reflects the home directory of the user on the local development machine where you put the “Cloud Numerics” project.

      4.   Change the Project Configuration from Debug to Release, and repeat the changes for the Release configuration.

Best Practices

Memory Usage and Matrix Size Guidelines

The recommended maximum memory usage is <=30% of physical memory. For a Windows Azure node used by a “Cloud Numerics” job, this equals roughly 4GB per node. All active variables in your application should not consume more than 30% of the physical memory available on the cluster. By default, the “Cloud Numerics” Deployment Utility creates clusters with extra-large Windows Azure nodes (8 cores, 14 GB memory).

Partition the Use of Memory Resources in Your Program with using{}

For improved performance, you can evaluate any memory intensive array operation such as matrix creation and cloning, matrix multiplication, LU decomposition, or SVD decomposition within the C# using{} construct. This construct forces the de-allocation of memory from the managed heap when the program leaves the scope of execution bounded by the using{}. If you do not exercise using{} or similar constructs to free unused allocated memory, your cluster can run with noticeable latency and long processing times.

Possible Issues and Suggested Workarounds

Transport-Level Error When Receiving Results from the Server

If you see the following error when interacting with SQL Azure, shut down all connections with SQL Azure, wait 10 to 20 minutes, and attempt your transaction again.

“A transport-level error has occurred when receiving results from the server (provider: TCP Provider, error:0. An existing connection was forcibly closed by the remote host.”

Superseded Software Packages Not Removed Before You Install the Prerequisites

The software packages referred to in this section must be uninstalled before you install the software prerequisites.

If you have any of the following software packages on your development machine, you need to uninstall them before you install the software prerequisites:

• Microsoft HPC Pack 2008 R2 Azure Edition
• Microsoft HPC Pack 2008 R2 Client Components
• Microsoft HPC Pack 2008 R2 MS-MPI Redistributable Pack
• Microsoft HPC Pack 2008 R2 SDK
• Windows Azure SDK v1.5
• Windows Azure AppFabric v1.5
• Windows Azure Tools for Microsoft VS2010 1.5

Network Error when Creating SQL Server Instance

When you deploy “Cloud Numerics,” for the first time, if you receive an error message when the “Cloud Numerics” Deployment Utility attempts to create your SQL Azure database, it is likely your network rules are blocking SSL traffic on TCP port 8443. In order to create the SQL Azure server instance, the “Cloud Numerics” Deployment Utility makes a network call to the SQL Azure Management API over this port.

To allow the deployment tool to complete the cluster deployment sequence, your site’s firewall rules need to enable SSL traffic over port 8443. The location where this modification needs to be made depends on your network route to SQL Azure. It may involve updating firewall rules on your local machine (where you run the deployment utility) and possibly any proxy or gateway servers through which traffic is filtered.

For details regarding requests sent to the SQL Azure Management API, see the following URL:

https://msdn.microsoft.com/en-us/library/windowsazure/gg715281.aspx

Cannot Establish Connection with SQL Server

Depending on your site IT policy, you might need a gateway client in order to set up a connection with the Azure Storage service.

https://www.microsoft.com/en-us/server-cloud/forefront/threat-management-gateway-trial.aspx

If you see the following message, it is typically a indicator that you need to install a gateway client.

Windows Azure Management Service Reports a Conflict

The management service reports a conflict if you attempt to deploy your cluster with a service name that is already in use.

If you see the following message when you try to deploy a “Cloud Numerics” application using the “Cloud Numerics” Deployment Utility (AppConfigure.exe), this can be  an indicator that the Windows Azure service name you chose is already in use.

To work around this situation, you can deploy your “Cloud Numerics” cluster with a different Windows Azure service name.

For example, if your first attempt specified the service name “contoso,” you could either choose another name or add a few characters to the end of this name to differentiate it from the service that is already deployed and in use. For example, you could try again using “contosopharmaceuticals.” Your fully qualified domain name for your Windows Azure service would then be contosopharmaceuticals.cloudapp.net.

This post demonstrates how to use Microsoft.Numerics C# API to perform statistical operations on data in Windows Azure blob storage. We go through the steps of loading data using IParallelReader interface, performing distributed statistics operations, and saving results to blob storage. As we sequence through the steps, we highlight the code samples from the application.

Before You Run the Sample Application

static string outputAccountKey = "myAccountKey";
static string outputAccountName = "myAccountName";

public object[] ComputeAssignment(int nranks)
{
    Object[] blobs = new Object[nranks];

    var blobClient = new CloudBlobClient(accountName);
    var matrixContainer = blobClient.GetContainerReference(containerName);
    var blobCount = matrixContainer.ListBlobs().Count();
    int maxBlobsPerRank = (int)Math.Ceiling((double)blobCount / (double)nranks);
    int currentBlob = 0;
    for (int i = 0; i < nranks; i++)
    {
        int step = Math.Max(0, Math.Min(maxBlobsPerRank, blobCount - currentBlob));
        blobs[i] = new int[] { currentBlob, step };
        currentBlob = currentBlob + step;
    }
    return (object[])blobs;
}

public int DistributedDimension
{
    get { return 1; }
    set { }
}

public msnl.NumericDenseArray<double> ReadWorker(Object assignment)
{
    var blobClient = new CloudBlobClient(accountName);
    var matrixContainer = blobClient.GetContainerReference(containerName);
    int[] blobs = (int[])assignment;
    long i, j, k;
    msnl.NumericDenseArray<double> outArray;
    var firstBlob = matrixContainer.GetBlockBlobReference("slab0");
    firstBlob.FetchAttributes();
    long rows = Convert.ToInt64(firstBlob.Metadata["dimension0"]);
    long[] columnsPerSlab = new long[blobs[1]];
    if (blobs[1] > 0)
    {
        // Get blob metadata, validate that each piece has equal number of rows
        for (i = 0; i < blobs[1]; i++)
        {
            var matrixBlob = matrixContainer.GetBlockBlobReference("slab" + (blobs[0] + i).ToString());
            matrixBlob.FetchAttributes();
            if (Convert.ToInt64(matrixBlob.Metadata["dimension0"]) != rows)
            {
                throw new System.IO.InvalidDataException("Invalid slab shape");
            }
            columnsPerSlab[i] = Convert.ToInt64(matrixBlob.Metadata["dimension1"]);
        }

        // Construct output array
        outArray = msnl.NumericDenseArrayFactory.Create<double>(new long[] { rows, columnsPerSlab.Sum() });

        // Read data
        long columnCounter = 0;
        for (i = 0; i < blobs[1]; i++)
        {
            var matrixBlob = matrixContainer.GetBlobReference("slab" + (blobs[0] + i).ToString());
            var blobData = matrixBlob.DownloadByteArray();
            for (j = 0; j < columnsPerSlab[i]; j++)
            {
                for (k = 0; k < rows; k++)
                {
                    outArray[k, columnCounter] = BitConverter.ToDouble(blobData, (int)(j * rows + k) * 8);
                }
                columnCounter = columnCounter + 1;
            }
        }
    }
    else
    {
        // If a rank was assigned zero blobs, return empty array
        outArray = msnl.NumericDenseArrayFactory.Create<double>(new long[] { rows, 0 });
    }
    return outArray;
}

static string inputAccountName = @"https://cloudnumericslab.blob.core.windows.net";

Microsoft.Numerics.NumericsRuntime.Initialize();

var dataReader = new AzureArrayReader.AzureArrayReader(inputAccountName, arraySize);

var x = msnd.IO.Loader.LoadData<double>(dataReader);

// Compute summary statistics: max, min, mean, median
output.AppendLine("Summary statistics\n");
var xMin = ArrayMath.Min(x);
output.AppendLine("Minimum, " + xMin);
var xMax = ArrayMath.Max(x);
output.AppendLine("Maximum, " + xMax);
var xMean = Descriptive.Mean(x);
output.AppendLine("Mean, " + xMean);
var xMedian = Descriptive.Median(x);
output.AppendLine("Median, " + xMedian);

// Compute 10% quantiles
var tenPercentQuantiles = Descriptive.QuantilesExclusive(x, 10, 0).ToLocalArray();

// Write output to blob storage
var storageAccountCredential = new StorageCredentialsAccountAndKey(outputAccountName, outputAccountKey);
var storageAccount = new CloudStorageAccount(storageAccountCredential, true);
var blobClient = storageAccount.CreateCloudBlobClient();
var resultContainer = blobClient.GetContainerReference(outputContainerName);
resultContainer.CreateIfNotExist();
var resultBlob = resultContainer.GetBlobReference(outputBlobName);

// Make result blob publicly readable,
// so it can be accessed using URI
// https://<accountName>.blob.core.windows.net/statisticsresult/statisticsresult
var resultPermissions = new BlobContainerPermissions();
resultPermissions.PublicAccess = BlobContainerPublicAccessType.Blob;
resultContainer.SetPermissions(resultPermissions);

resultBlob.UploadText(output.ToString());