Just a survey article on materials on parallel database products and technologies for OLTP/OLAP applications. It mainly covers major commercial/academic efforts on developing parallel dbms to solve the ever growing large amount of relational data processing problem.
 
Part I - Parallel DBMSs

1.1 Parallel Database for OLAP (Shared-Nothing/MPP)

TeraData
- TeraData Home
- Teradata DBC/1012 Paper
- NCR Teradata VS Oracle Exadata (Teradata's perspective)

Vertica
- Vertica Home
- The original research project: C-Strore

Paraccel
- Paraccel Home
- MPP Based Architecture
- Columnar Based Storage
- Flash Based Storage

DataLlegro(now MS Madison)
- Design Choices in MPP Data Warehousing Lessons from DATAllegro V3
- Microsoft SQL Server Parallel Data Warehousing

Netezza
- Netezza Home
- Acquired by IBM
- Hadoop & Netezza: Synergy in Data Analytics (Part 1, Part 2) 
- Netezza Twinfin VS Oracle Exadata (eBook, Blog, Netezza's perspective)

GreenPlum:
- GreenPlum Home
- Combined: PostGreSQL/ZFS/MapReduce
- Acquired by EMC

Oracle ExaData:
- ExaData Home (Technical Overview, White Paper)
>> - OLTP & OLAP Hybrid Orientation
>> - 1 * RAC + N * Exadata Cells (Storage Node) + Infiniband Network
>> - Exadata Cell: Flash Cache + Disk Array + Data Filtering Logic (partial SQL execution)
- Oracle Exadata VS Netezza TwinFin (Oracle Engineer's perspective)

IBM DB2 Data Partitioning Feature (can work with both OLAP/OLTP)
- formerly known as DB2 Parallel Edition (An Shorter Overview)
- DB2 At a Glance - Data Partitioning Feature
- Simulating Massively Parallel Database Processing on Linux

AsterData:
- Supercharging Analytics with SQL-MapReduce
- Aster Data brings Applications inside an MPP Database 

Misc Articles:
- What's MPP?
- Comparison of Oracle to IBM DB2 UDB and NCR Teradata for Data Warehousing
- SMP or MPP for Data Warehouse
- Dividing the data Warehousing work among MPP Nodes
- SANs vs. DAS in MPP data Warehousing
- Three ways Oracle or Microsoft could go MPP

1.2 Parallel Database for OLTP (Shared-Disk/SMP)

Oracle Real Application Cluster
- Oracle RAC Concepts
- Oracle Parallel Database Server Concepts
- Oracle RAC Case Study on 16-Node Linux Cluster

IBM DB2 for z/OS (with Sysplex Technology)
- Share Disk and Share Nothing for IBM DB2
- What's DB2 Data Sharing?

IBM DB2 for LUW (with pureScale Technology)
- IBM DB2 pureScale: The Next Big Thing or a Solution Looking for a Problem?
- What is DB2 pureScale?
- DB2 pureScale Scalability (section 1, section 2)

Part II - Academic Readings

2.1 Overview
1). Parallel Database System: The Future of High Performance Database Processing
2). Survey of Architecture of Parallel Database System
3). The Case for Shared Nothing
4). Much Ado About Shared-Nothing 

2.2 Research System
1). XPS: A High Performance Parallel Database Server
2). The Design of XPRS
3). Prototyping Buuba, H High Parallel Database System
4). The Gamma Database Machine Project
5). NonStop SQL, A Distributed, High-Performance, High-Availability Implementation of SQL
6). Parallel Query Processing in Shared Disk Database System
7). Architecture of SDC, the Super Database Computer

2.3 Commercial System
1). A Study of A Parallel Database Machine and Its Performance - The NCR/TERADATA DBC/1012
2). A Practical Implementation of the Database Machine - Teradata DBC/1012
3). DB2 Parallel Edition
4). Parallel SQL Execution in Oracle 10g
6). Shared Cache - The Future of Parallel Database
7). Cache Fusion: Extending Shared-Disk Clusters with Shared Caches

"Over the last several months in my role here in China, I have given talks at several leading universities and met with many of the leading faculty and students working on technologies related to the Data Platform. I’ve also spoken at several industry conferences, meeting with customers, partners, analysts and other industry folks. There are many topics that come up at these meetings – changing technology trends, distributed development, the tremendous growth of Asia etc. But one topic that seems to come up more than almost any other is the question of how we organize and conduct our product development in Microsoft. I suppose this is only natural – Microsoft is one of the most successful software companies in the world, and the software industry here in this region is poised for tremendous growth, so it makes sense that people in the industry are eager to learn from the our experience over the last quarter century.

This is actually a very big topic and within Microsoft we have an Engineering Excellence group that actually runs courses that can span several days and provide an overview of Microsoft’s software development methodology, our engineering system, organizational structures, best practices, tools and technologies we use internally ensure quality, reliability, security etc and a variety of related topic. By no means would we claim that we have all this figured out perfectly and have a perfect system, but there is indeed a lot of accumulated knowledge and experience that we can share. And we do actually share this information, in appropriate form, with others in our industry, worldwide and also in this region.

As this is indeed a large topic, I don’t want to get too deep into this here, but I do want to address one aspect of our engineering system – the core disciplines that we organize our R&D teams around and the particular roles that each of these disciplines plays. I want to discuss this because I believe Microsoft does this a little bit differently from the rest of the industry even in the US, and especially here in China there is not a good understanding of these core disciplines and what role each of them plays.

Traditionally, the Microsoft engineering system has consisted of 3 “core” disciplines: “Development”, “Test”, and “Program Management”, also known as Dev/Test/PM for short. I’m going to touch on each of these briefly here, but I like to introduce them in a different order:

PM: When we think of engineering disciplines, most people start with “Dev”. For me however, things really start with the Program Management discipline. At Microsoft, “PM” means many different things, but for me the core essence of the PM role is two things:

1. The first part of the PM’s job is to understand the customer’s requirements and translate that into a functional specification of what we should build. This is where it all begins. If we don’t understand the customer, it is not very likely that we’ll end up building the right thing.

2. The second part of the PM’s job is to work with Dev and Test to translate the initial specification into a living, breathing product.

I find that many people, especially here in China, think “Project Management” when they hear PM. Indeed, Project Management is part of a PM’s job (under #2 above), but it is only a part of the PM’s job. The real skill that a PM brings is the expertise to listen to customers, understand the world from their point of view, and then to design a solution for their problem. This does not just mean giving customers what they ask for literally, but to truly understand them and design a solution that solves their problems even if the customers could never imagine the solution – as the famous saying goes, if we had only listened to customers, we would have looked for a faster horse, not come up with the automobile.

Dev: Of all the engineering disciplines, this one is probably the one people think about the most commonly. Dev is short-hand for “Development”, the folks who responsibility it is to actually design and build the software that we ship. The essential job of Dev is to take the functional specification produced by PM and translate that into an actual implementation. In the world of mission-critical system-level software, this implementation better be extremely reliable, secure, manageable, scalable and high-performance. And the designs and implementations Dev produces better stand the test of time and last for several versions and years to come.

Test: The test discipline in Microsoft is much misunderstood, certainly externally, but sometimes internally as well. When I first came to Microsoft many years ago, I was (pleasantly) surprised to find that Microsoft had almost as many, if not more, testers as developers. Coming from a company that had a much less developed testing discipline (and where as a result, quality assurance was considerably weak), it took a little while to get used to what the essence of the Test discipline really is. The reality is that, in Microsoft, how fast we can ship software depends on not how quickly we can design and implement it but rather on how quickly we can test it. This is because every piece of software we ship, especially on the systems-software side, has to pass an extremely high quality bar. The Test discipline is really an complex area, and one where have learned a lot over the years in terms of different types of testing that we employ – unit tests, functional test, integration tests, stress and long-haul tests, performance tests, security tests, localization tests, etc. The set of tools and techniques we employ in test is truly some of the most impressive and complex – automated test harnesses, automated test generators, automated test failure analyzers, automated security “fuzzers”, fail-point and state-machine based testing.

The three “core” engineering disciplines described above are like the 3 legs of a chair – you need all three of them, and in a balance, to have a proper engineering organization. No one leg can dominate the other – otherwise, you get an organization that may not be in touch with customers needs or one that does not pay enough attention to quality. Indeed, the three disciplines are a little bit like the branches of government – they form a system of checks and balances that ensures we understand what customers want, we design and build that with high quality, and we ensure that we deliver a product that meets customer expectations in every regard.

It is also important to emphasize that we aim to attract the best talent to all three core disciplines – the bar is equally high for all the disciplines, it just happens to be that the passion and skill-set for each is a little different:

- PMs usually have a passion for working with customers, conceptualizing what the product should do, and then working with their Dev and Test peers to coordinate all the work to make sure we deliver exactly that.

- Developers have a passion for building top-quality software – software that is innovative, simple, reliable, secure, scalable, high-performance and stands the test of time.

- Testers are passionate about finding all kinds of ways to break software and making sure making sure we find all the issues and bugs before we ship it to customers.

When we interview candidates, a very important part of what we do is find out which discipline the person’s talent and passion really lie in and directs them accordingly. Of course, over the course of one’s career, one’s passion and talent may change, and the person may change disciplines as a result – I myself started in the Dev discipline before switching to PM. This is only natural and we actually encourage that as a way to build better teams.

Other disciplines

It is also important to point out that although the three disciplines mentioned above are what have traditionally been considered the “core” disciplines at Microsoft, there are several other disciplines that are also becoming increasingly important. For example, User Experience (UX) professionals are essential to ensuring that products are intuitive and natural for users to use. A great user experience can make the difference a product that customers love versus one they merely tolerate. UX is certainly very important for products aimed at end consumers, but it is also important for all our audiences – Developers, IT Professionals, Information Workers.

As we move into the Software+Services era, a variety of disciplines related to architecting, building and running extremely large-scale infrastructure becomes increasingly important. Again, while this has been true for some time for our consumer facing web properties such as MSN and Live, it is now becoming increasingly important for all our product groups as more and more of them take steps to evolve their products along the Software+Services model.

Many candidates I talk to often want to discuss what role at Microsoft would be the best fit for them and how they can grow their careers. The best advice I can think of is to work on a technology and a role that they are really passionate about.

As I mentioned above, we value all the disciplines equally and a well-balanced organization needs great people in all the different roles. While different disciplines appeal to people with different passions and skill-sets, all the disciplines offer opportunities for innovation and great work. And all of them offer opportunities for advancement and leadership. Indeed if you look across the senior levels of Microsoft, there are leaders who emerged from various disciplines – what they shared was a passion for what the work they were doing.

I hope this discussion of the different engineering disciplines at Microsoft and the approach we take to them shall be useful for the many people who seem to be interested in this topic. If you have any questions or comments, feel free to post a reply to his entry."

All .Net networking facilities are build on top of Winsock2 subsystem, so concepts such as non-blocking I/O, async I/O and I/O multiplexing can all be applied to the managed world. .Net network framework just wrapped Winsock2 and added some extra abstraction.

Most of the wrapping stuff is done through the System.Net.Sockets.Socket class. If you are familiar with Winsock, you will feel comfortable with this class. Here I will focus on the extra (OO style)abstraction that .Net library team has added on top of this class.

There are 3 abstraction layers for .Net network programming:

-Raw Socket Layer
In this model, you work directly with System.Net.Socket.Sockets class, which is just a lightweight OO and PInvoke wrapper around native Winsock2 functionalities. You can do anything in Winsock2 using methods of this class. It's almost just method to method mapping. More info see .Net Network Programming Using Raw Socket.

- Tcp/Udp Layer
.Net network framework provides three extra classes: TcpClient, TcpListener, and UdpClient to ease the task of Tcp/Udp application developing. They are built on top of raw socket facilities and hide detail works such as connection management, data buffering and parsing with the help of NetworkStream class. More info about Tcp/Udp model and more specific on .Net Tcp Programming, .Net Udp Programming

- Request/Response Layer
It's also known as pluggable protocol layer, in which all communication are simple request/response pattern using standard URI to represent internet resources. This abstraction is build on top of Raw Socket and Tcp/Udp layer and it's also extensible so you can customize it. If you just need to work in web world, this is a great facility. More info about programming pluggable protocols, FTP programming and HTTP programming

Source code for example client/server application using .Net network framework.

BTW, blocking/non-blocking/async model and the upper 3 layer concepts are orthogonal and can be combined arbitrarily. For example, there are both blocking TcpClient/TcpListener and async TcpClient/TcpListener.

[Reference]

1. Network Programming Tutorial
2. Network Programming Sample
3. Core NameSpaces:
- System.Net
- System.Net.Socket
4. Good .Net Networking Books
- Tcp/Ip Sockets in C#
- C# Network Programming

Part II - Java Networking Briefing

Java has a totally different network library taxonomy.

1. Abstraction Layers

Java doesn't have real raw socket facility, which is similar to BSD socket interface and Winsock2 interfaces. But it does contain class named as Socket that is used do network communication. It can be roughly divided into Tcp/Udp layer and URL layer.

- Tcp/Udp Layer
You deal with Tcp/Udp concepts directly when coding at this layer. Java provides 3 core classes:Socket/ServerSocket (Connection Oriented network facility, similar to .Net's TcpClient/TcpListener), DatagramSocket (Connectionless, Message Oriented network facility, similar to .Net's UdpClient). These classes also use I/O Stream to do reading/writing operations.

- URL Layer
Similar to .Net's pluggable protocol layer, it uses URL to identify network resources, provides convenient methods to talk to them. URL and URLConnection are those core classes.

2. Network I/O Model

For network I/O model, JDK provides blocking I/O in Java.Net name space since Java 1.0, multiplexing(a.k.a. non-blocking) I/O is available since Java 1.4 and Async I/O is added in Java 7, both in Java.Nio name space.

Multiplexing I/O consists of three core concepts: SelectableChannel, Selector and SelectionKey.(illustrated below). Common practices using java nio suggests that a network server is composed of acceptor to accept client connections, dispatcher to monitor network ready events and dispatch them to application handler, and application handler with worker thread pool, which handles application logic and generate response data to client.
I had write an time echo server using Java Nio and the upper architecture.

Java Asychronous IO(part of NIO.2) is relatively new and not officially released yet. But its basic idea is very simple: you issue the i/o request without blocking. The system will notify you the completion of the request by meas of:
- calling callback function
- waiting handle to wait
- polling flag to poll

[Reference]

1. Official Network Tutorial
2. Java Scalable I/O
3. NIO based Server Architecture

Packages:
- Java.Net
- Java.NIO

- It's a heavyweight OO communication framework but it goes beyond just computer proress communication. In a high level point view, it contains three lays:

- OS Adapter Layer: it abstracts the underlying native OS's services using raw C wrappers and provide a unified interface to upper layer components and application developers. The OS services include:

　　．Concurrency and Synchronization
　　．Inter-Process Communication
　　．Event Demultiplexing
　　．File System I/O
　　
- OO Facade Layer: it wraps the OS adapter layer using OO, more specifically, the C++ language. Its functionalities are all the same as the adapter layer.

- Communication Framework Layer: it integrates and enhances the lower-level C++ wrapper facades and supports the dynamic configuration of concurrent distributed services into applications. The framework the following components:

　　．Event Demultiplexing - The Reactor and Proactor are extensible, object-oriented demultiplexers .
　　．Service Initialization - The Acceptor and Connector components represents the active and passive initialization roles, respectively.
　　．Service Configuration - The Service Configurator supports the configuration of applications dynamically.

ACE is very heavy-weighted and contains many powerful components. It is also the pioneer of concurrent and communication software architect. It invents many famous design patterns in this area, for example, Reactor and Proactor.

ICE website also contains other two great articles:
- The Rise and Fall of CORBA
- API Design Matters

Behind Avatar's Fancy Special Visual Effect

Since 12/18/2009, Avatar has attracted all people's eyes on this planet. I watched a 3D IMAX version of this film some time ago. After seeing those fancy special visual effect, I can't help wondering how they accomplish that and what computer hardware/software they had used.

The special effect of Avatar is mainly made by Weta Digital(alongside with ILM), who had also made Lord Of The Ring, King Kong etc. Data Center Knowledge and Information Management both have some great articles([01] - [07]) about the IT infrastructure that supports the making of Avatar.

Here, I try to summarize core information gathered from these posts and focus on the IT related aspects. For the making of special effect itself, see "Avatar - The Game Changer" please.

Part I - Hardware Configuration[3][7]:

Cluster
- HP Cluster Platform 3000BL
- 34 Water Cooled Racks(from Rittal), each with 4 HP BladeSystem Chassis
- Each Chassis consists of 16 Blade Servers
Blade Server
- Type: BL 2x220c
- 2 Nodes, each with 2 4-core processors
- totally 34 * 4 * 32 * 2 * 4 = 34, 816 cores
RAM
- 104 T
Network
- 10G Ethernet Network from Foundry Networks(now acquired by Brocade)
Storage System
- 3P Network Area Storage
- FAS6000 and FlexCache from NetApp
- Titan Server from BlueArc
Space Size
- 10,000 square feet
Cooling System
The industry standard, raised floors and forced-air cooling, doesn't work because the blade server is very dense. Weta's innovative solution consists of two parts:
- Enclosed, Water Cooled Racks where the hot air is sucked into a radiator and cycled back through the front of the machines.
- Run the Machine Warm (~25 C), which modern computer doesn't mind.

Part II - Movie Rendering

"The main activity in a visual effects data center is called rendering,the process of turning the digital description of an image into an actual image that can be saved to disk and eventually written to film or another media. And the banks of computer is called render wall."[01]

Software
- OS: Linux
- Job Management: Pixar Alfred

Job System
- In HPC or MPP community, each job is huge. But when doing move rendering, you faces with large amount of relatively small tasks. In any single job, there might be thousands of interdependent tasks. As soon as CPUs on the render wall are freed up, new tasks are fired at idle processors.
- The queuing system is a Pixar product called Alfred, which creates a hierarchical job structure or tree of multiple tasks that have to run in a certain order.

Work Flow
"The artist shows their work, the director wants more of this or that and it goes back and forth many times,” says Wilkie. “It’s the review cycle that drives the way things look and that drives our job work. In that way each tiny bit gets better and better, then perfect, and we move on to the next item.” [01]

Processing Scale
- Job Scale : 10K jobs and 1.3~1.4 M tasks per day
- Data Scale : 12M/Frame, 288M/Second and 17.28G/Minute ...
- Speed : 7~8 G/Second for about one Month

[Reference]

- Data Center Aspects

01. Processing Avatar
02. Data Center Lessons from Avatar
03. Data Crunching Powerhouse behind Avatar
04. One Data Center to Rule Them All
05. Overheard: Avatar's Data Center
06. Data Center Plays Supporting Role in Avatar
07. Avatar, Hollywood and the Data Center
08. Weta Digital's Cluster Ranking among Top500 systems

- Special Effect Aspects

11. Avatar, the Game Changer
12. ILM Steps Into Making Avatar Visual Effect

- Partner's News Report

21. HP - Behind the Magic of Avatar
22. NetApp - Weta Digital and NetApp Bring Avatar to Life
23. BlueArc - BlueArc Storage Solution Powers Visual Effects Behind Avatar

- HP Blade System

31. HP - What is a Blade?
32. HP Cluster Platform 3000BL DataSheet
33. HP Official Page on BL2x220c
34. BL2×220c Video Introduction
35. Blog on Inside BL2X220c

Database Technologies for Decision Support System

Database technologies can be applied into two types of scenarios:
- Transaction Processing(OLTP)
- Analytic Processing, using statistical method(OLAP) or machine/computational learning method(Data Mining)

OLTP, which is based on E.F. Codd's relation model, is the traditional (maybe most popular) application type of DBMS and most people are very familiar with it. This post tries to summarize related technologies in analytic processing, which is widely adopted in decision support systems.

Part I - What Data to Analyze?

In Decision Support(or Business Intelligence) system, data to be analyzed usually comes from operational system, i.e., OLTP relational database. These relational databases are often located at different departments/sites, may be using different DBMS vendor's products, using different data schema and merely contain data within a relatively short time span.

To make a good business decisions, it's strongly desired to hold historical data, view them in a uniformed way and not bothering the daily operational environment. Thus comes out the Data Warehouse, which is a repository of an organization's electronically stored data and is designed to facilitate reporting and analysis.

Operational data (in OLTP system) is extracted, transformed(also cleaned) and loaded(by ETL subsystem) into the data warehouse for further analyzing. OLAP and DM systems read these data, analyze them, produce useful reports and present them to end(business) users. (See diagram below)

Decision Support System Life Cycle, based on MS SQL Tech Doc

Part II - How to Analyze?

There are two ways to analyze data in data warehouse:

1. Data Mining

Data Mining is the extraction of hidden predictive information from large databases. Someone also defines it as knowledge discovery process(using machine learning algorithms) in database.

Technical highlights:
- Typical Data Mining enabled information systems process data in terms of Record(or Case).
- Such system also provides some Language Extension to facilitate composing data mining related queries. (for example, MS SQL Server provides DMX - Data Mining Extensions)

The challenging part of data mining is various mining algorithms. Here is a list of data mining algorithms available in MS SQL Server Analysis Service.

See - A Basic Data Mining Tutorial using MS SQL Server

2. OnLine Analytical Processing(OLAP)

OLAP is the processing of large scale multidimensional data using statistical based methods. A typical OLAP system provides:
- Multidimensional Model
- Analytical Query Language (for example, MS SQL Server provides MDX - Multidimensional Expression)
- Analyzing Server(or Engine) that executes analytical queries

See - A Basic Tutorial for OLAP in MS SQL Server

2.1 Multidimensional Model

Multidimensional model view data as as cubes that generalize spreadsheets to any number of dimensions. It categorizes data either as numerical values(a.k.a. measures) associated with some facts or textual values(a.k.a. dimensions) that characterize the facts.

Facts represent the subject - the interesting event in a enterprise that need to be analyzed.
Dimensions represent context information for facts, perspectives to view facts.
Measures represent those numeric properties of facts that decision makers want to analyze.

For example, in a shoe shop, shoe purchasing events are the facts, the selling price is a measure attribute and the color, the size, the manufacture and the brand are all dimension attributes.

More tutorial explanation of this model can be found in [7][8].

2.2 OLAP Server Architecture

There are three ways to implement multidimensional model:

- ROLAP (Relational OLAP)
Fact data is stored in relation model based storage system and some special induces technologies may be adopted. In this architecture, measures are derived from the records in the fact table and dimensions are derived from the dimension tables.

from star schema example

This architecture can be further divided into two types:
　Star Schema
　Snowflake Schema

from snowflake schema example

Both of them contains fact table and dimension table, but in star schema, there is only one table for one dimension, while in snowflake schema, there are usually multiple tables for one particular dimension.

- MOLAP (Multidimensional OLAP)
Fact data is stored in an optimized multi-dimensional array storage, i.e., the server supports the multidimensional model directly.
It's usually regarded to be faster but less scalable than ROLAP.

- HOLAP (Hybrid OLAP)
It is a combination of ROLAP and MOLAP. HOLAP allows storing part of the data in a MOLAP store and another part of the data in a ROLAP store.

Part III - Other technologies

Other related technologies include data visualization, metadata management, analytical query parallelizting etc.

Analytical database technology is very promising and also complicated, more survey paper can be found at[5][7][8].

[Reference]

General

[01] Course on DM & OLAP
[02] OLAP & Data Mining Links
[03] OLAP & Data Warehouse Bibliography
[04] A Brief Tutorial on Data Mining and OLAP

[05] An Overview of Data Warehousing and OLAP technology
[06] Providing OLAP to User-Analysts: An IT Mandate

[07] An Overview of Data Warehouse, OLAP and Data Mining Technology
[08] Multidimensional Database Technology

Data Warehousing

[21] Data Warehouse Architecture
[22] Oracle Data Warehousing Guide

Data Mining

[31] Data Mining FAQ
[32] Data Mining Introduction
[33] Data Mining Tech Summary

[34] Oracle Data Mining Concept
[35] Microsoft SQL Server Analysis Service - Data Mining

OLAP

[41] OLAP Introduction
[42] OLAP Overview
[43] OLAP Council
[44] OLAP Wiki

[45] Oracle OLAP User Guide
[46] Microsoft SQL Server Analysis Service - OLAP

Parallel Programming - Using MPI

MPI is a message passing programming model standards[2], it defines various Terms/Concepts, Data Structures and Function Signatures that are used to passing messages among computer processes.

1. Terms and Concepts

A MPI application consists of multiple Processes(Tasks), each has a unique identifier called Rank. A Process belongs to some Groups and some Communicators. Processes don't share memory or state, the only way to communicate is sending/receiving message.

Programming Patterns - Usually, multiple Processes are needed to accomplish a task, there are some popular patterns on how processes are cooperated:
- Peer to Peer (a.k.a. Work Crew) Model, each process behaves equally
- Master/Slave (a.k.a. Master/Worker) Model, one process acts as coordinator, others act as common labor force

2. MPI APIs

MPI APIs can be categorized as:
- Environment Management
- Data Type
- Point-to-Point Communication
- Collective Communication
- One Sided Communication
- Process (Topology, Creation and Management)
- Parallel I/O

Point-to-Point communication occurs between two Processes, while Collective communication involves all Processes in a communicator and One-Sided communication only needs on Process's participation.

3. Collective Communication Patterns

Typical collective communication semantics are somewhat hard to describe, Some diagrams (mainly from Rusty Lusk) are listed below to illustrate the semantic of each primitive:

4. Typical MPI Program Structure

#include "mpi.h"

int main(int argc, char** argv)
{
　int nProc;
　int nThisRank;
　MPI_Init(&argc, &argv);
　MPI_Comm_size(MPI_COMM_WORLD, &nProc);
　MPI_Comm_rank(MPI_COMM_WORLD, &nThisRank);

　if (nThisRank == 0)
　{
　　//Core App Logic
　}
　else
　{
　　//Core App Logic
　}

　MPI_Finalize();
　return 0;
}

I had written some non-trivial MPI applicatioins:
- Demo various collective communication semantic
- Parallel Numeric PI Calculation
- Map-Reduce-Merge over MPI

You can try the code to get hands on experiences.

5. Combine Message Passing with Threading

Typical MPI applications run one process per core on a machine node. Since all cores on a node share the same physical memory, we can use a combined model, where one process per node and multiple threads per process(OpenMP, PThreads etc) on each node. I.E.:
- Threading within one node
- Message Passing cross nodes

Notes on MPI programming:

1. MPI's network facilities are message oriented, not connection/stream oriented (as in Tcp Socket programming), so before receive a message, you'd better know how large it will be, other wise, message may be truncated.

2. MPI can dealing communication with both fixed length and variable length data types. For variable length data types, use MPI_Pack/MPI_Unpack and those MPI_xxxv(for instance, MPI_Scatterv, MPI_Gatherv, MPI_Allgatherv, MPI_Alltoallv) functions.
(mpi.net's source contains code that leverages these *v functions, it's good example to learn MPI programming)

[Reference]

0. MPI Official Site
1. MPI V2.2 Official Standard
2. Message Passing Interface on wikipedia

3. MPI on Multicore (PPT)
4. MPI and OpenMP Hybrid Model
5. Combining MPI and OpenMP

Tutorials

11 Parallel Computing Introduction
12 MPI Tutorials @ LLNL (Exercises)
13 Tutorial on MPI by William Gropp (Exercises)
14 C++ MPI Exercises by John Burkardt
15 MPI Hands-On Tutorial by OSC
16 Tutorial on OpenMP and MPI

17 Book online: MPI The Complete Reference
18 Book: Parallel Programming With MPI

19 Clear Message Passing Concept Explanation

MS-MPI/MPI.NET

20 Windows HPC Server 2008 - Using MS-MPI whitepaper
21 Using Microsoft MPI
22 MPI.NET Tutorial

Implementations

- Native
MPICH2 - a MPI implementation
Open MPI (formerly Lam/MPI)
MPI Implementation List (List 1, List 2)

- .Net
Pure MPI.NET (implemented totally using .Net)
MPI.NET (an .Net wrapper around native MPI library)

- Java
Java-MPI binding list
mpiJava
MPJ Express

- Python
PyMPI
MPI4Py

- Matlab
MatlabMPI

Parallel Programming - Using POSIX Threads