Recently I have be scouring resources all over the internet and at work to see what technologies actually exist to help programmers write scalable parallel code for .NET. Over the last few years various teams at Microsoft have been working on technology for parallel code. I have picked three projects that really stood out and I thought I’d share them. I’ll keep the summaries brief as there are very good resources already in existence. You can find out more about the various teams and technologies for concurrent programming at Microsoft at The Parallel Computing Developer Centre.

If you’re looking for Native Libraries I would strong advise looking at the Parallel Patterns Library, you can find out more information at the blog Parallel Programming in Native Code.

Task Parallel Library (TPL) and Parallel LINQ (PLINQ)

Version 4.0 of the .NET Framework ships with several new pieces of technology, namely library improvements, to make parallel programming simpler. The two additions that have made me most excited are the Task Parallel Library (TPL) and Parallel LINQ (PLINQ). You can find out more about both libraries in the Parallel Programming in the .NET Framework docs on MSDN.

I have been playing with both technologies ever since I first got hold of the .NET Framework 4. It makes writing parallel code ridiculously easy and straightforward. I was able to convert a C# test game server which made heavy use of threads to use the new Task system very quickly. Overall I gained an average 15% performance improvement, which is certainly impressive. I don’t claim the server was particularly well written, but the simple task of making use of TPL and PLINQ made significant performance gains, time is always an issue for programmers and these libraries certainly help.
As I don’t work with .NET in my job, I haven’t had the chance to apply these technologies in a large real-world application but it certainly shows promise and once .NET 4.0 comes out of beta I will revisit the technology.

F#

“F# is a functional programming language for the .NET Framework. It combines the succinct, expressive, and compositional style of functional programming with the runtime, libraries, interoperability, and object model of .NET.”

Functional programming is not something I have ever done in any great depth. Although F# is not designed specifically for concurrent programming, when I started my research several friends suggested that I look at functional languages for concurrent development. As I was looking for Microsoft technologies and I have heard so much about F# recently, it seemed like a good place to start.

After playing around with the Visual Studio 2010 Beta which includes F# and reading various tutorials and examples, I wrote a test game server based on my C# version. The code was certainly a lot more simplistic than an equivalent implementation in C# or C++.
Without any experience in functional programming the learning curve was definitely quite steep and I doubt I could implement a very large project in any great speed. Having discussed F# with a functional mad friend who is very excited about it, it definitely seems to offer some great advantages for functional programmers. Naturally the disadvantage of the F# technology is that programmers like me need to make a big investment to learn both the paradigm and the language. It’s certainly something I hope to be able to do though!

Axum

“Axum is a language that builds upon the architecture of the Web and principles of isolation, actors, and message-passing to increase application safety, responsiveness, scalability, and developer productivity.”

Now for my personal favourite. Axum is an incubation project from the Parallel Computing Platform team, it’s a new language derived from C# which is designed to offer an easy and productive programming model for parallel development.

I stumbled across Axum when searching around a few weeks ago. I downloaded it and after reading the Axum Programmers Guide, I gave it a try. Using my C# test game server as a base, I implemented it in Axum.

The paradigm felt a little strange at first and there was certainly a learning curve. I was able to pick up Axum far more rapidly than I did F#, as someone who is used to writing concurrent code in C# and C++ the code felt logical and I was able to very easily understand the flow of the application.

For me, Axum offers the perfect balance between F# with a very steep learning curve and TPL with a very shallow one. I will be following the progress of the project very closely and hope that the team continue to develop it to a fully supported product.

Summary

I should point out that it is possible to use a combination of the above options and they are not mutually exclusive, there are also numerous other technologies that you can investigate.

If you’re looking to quickly parallelise an existing .NET application, TPL and PLINQ certainly looks like an attractive option. It’s very quick to convert existing threading logic, ultimately very cheap to implement and offers some significant performance and scalability gains.

As I mentioned in my Axum section, if you’re looking for a language which feels familiar but offers excellent concurrency functionality and has a reasonable learning curve, you should do some research and testing with Axum and you won’t regret it.

F# certainly seems to be the option which allows you to clearly express what you want to happen and when. My problem with F# is that without any significant experience with functional programming, I found the learning curve very steep. I believe that most programmers have experience in similar languages to me, and that could be barrier to this option.

Ultimately the correct decision will be different for every project and the key is to do the correct research.

That’s all folks.

When I started this blog I promised myself I would post once a week, I have already broken that promise after only two posts.

The last couple of weeks have been extremely busy for me; I tidied my apartment properly for the first time since I moved in, several new games have appeared on the market and I've been trying to keep up with the influx of awesome TV shows.

In my last blog post I covered some of the good and bad things which have been introduced as games become more concurrent. I also stated that games are now expected to have online multiplayer as standard. The problem is, if you don’t have anyone with network programming experience, it’s often a minefield.

The last month or so has seen some problems which still exist with online games resurface in the form of server queues and simple unavailability in a couple of new titles.
Since I began my career in the world of game networking, I have worked with three networking systems which couldn’t be more different from one another.
 

Architecture

The very first system I worked with was for a massively-multiplayer online role-playing game. This system was, as you might expect, strictly client-server.
The second networking system I worked with was for a racing title. Unlike the MMO, this system used a peer-to-peer architecture.
Finally, and most recently, we have the third system for an Action RPG. This system is a hybrid between peer-to-peer and client-server.

One of the most frequent questions I get asked is “Which architecture should I use?” it’s a valid question and a very interesting one. It’s a particularly nasty question for engine developers as there simply is no “one size fits all” solution for game networking. The true answer is “it depends”.

Peer-to-peer architecture allows all peers to communicate directly with one-another, sharing and managing their own resources. You can find out more at http://en.wikipedia.org/wiki/Peer-to-peer.
Client-server architecture on the other hand has one or more central authoritative peers (aka servers). You can find out more at http://en.wikipedia.org/wiki/Client-server. I won’t go into a huge level of detail as those articles describe both architectures very well and it’s somewhat outside the scope of my blog.

Peer-to-peer architecture generally allows peers to propagate information more quickly as they can communicate directly with other relevant peers. One of the big advantages of a peer to peer system is the ability share ownership and responsibility between peers. Ownership of world objects can literally be spread between peers, and when you need to do something to an object, you can request ownership from the peer that currently owns it. Peer-to-peer is also generally quite flexible; host migration is simple as there is no true host.

Peer-to-peer generally starts to fall down when security and control is important. A peer-to-peer system is inherently insecure. There is no authoritative server to make decisions, so in the case of disagreements peers need to arbitrate, which can become quite complicated. Peer-to-peer also tends to use more bandwidth. It is often the case that someone hosting a game has an internet connection that is suitable to do so, but with this architecture all peers share higher bandwidth. Methods to work around high-bandwidth generally include setting up relay systems, this starts to eat into the speed gains you get from the direct connection and is again, complicated.

Client-server architecture generally gives more control, this is extremely important in some kind of games. Client-server is heavily used in most first-person-shooters for this very reason. The server is in the perfect position to make decisions and treat all players fairly. A client-server architecture also gives you more security, having one authoritative point allows you to see exactly what is going on everywhere else and make informed decisions about cheating and other security concerns. Security is even more important in games such as MMOs where cheating is a very large concern and the majority of simulation such as NPCs is running on the server.

It follows that client-server is weaker at the things which peer-to-peer excels at and vice-versa. In some cases it lacks speed, as messages have to pass through the server and then to relevant peer. Client-server also makes it more difficult to implement seamless host migration and good shared ownership.

The above examples are not exhaustive and do not cover every implementation of both architectures so do not take those reasons as the only ones, do some research!

Integration

Not only has the network architecture itself been different, but so has the way it is integrated with the rest of the game. Unfortunately, networking is a very obtrusive system as it really needs to know everything that is happening on relevant peers to keep things ticking over correctly.

Another question that I hear frequently is “how do I write the code?” or “how do I lay out the systems?” again, a valid and interesting question. Again, there is no “one size fits all” solution.

I’ve seen some extremely clever and some extremely stupid things done in relation to game networking over the past few years. Some solutions are over-engineered to the point of insanity and others are tacked on to the side of the game in a very scruffy fashion.

The systems which are over-engineered tend to be written by one programmer throughout the lifetime of the project. That one programmer scurries around after everyone else and make’s their code network friendly. None of the other programmers are involved with the networking system and the network programmer basically has entire control.

The scruffy systems on the other hand, tend to appear when during the vast majority of production, there has been no network code. Very late on a large amount of the team focuses on getting online multiplayer working. This results in code which is essentially tacked on to the side of the game. Nothing in the game is designed to work over the network, so the code is littered with special case exceptions for certain systems.

Naturally you want to hit a balance between code which is too complexed and code which is messy, but this can be hard and needs the co-operation of every programmer on the team.

Solutions and best practices

There are lots of things you need to take into account when writing network code for games. The subject is quite deep and will only get more complicated as games try to do more and more online.

The best thing you can do is thoroughly research the best solution for your title. It’s also very important to get all programmers involved in writing code which is robust enough to work over the network; this helps you to avoid the integration problems I described above and leaves the network programmers to concentrate on the core network tech and architecture.

I plan to follow up this post with some more in-depth details about both the architecture and integration aspects of network code, but going on my previous record... that might be a while.

When I moved to the games industry from the (rather less interesting) business sector, I was therefore very excited to be working on a large-scale MMO as my very first project. With the success of games like World of Warcraft, and even services such as Xbox Live, much attention has been paid to massively scalable server solutions. The number of people specializing in development and technology for multiplayer games such as myself has expanded rapidly in the last few years. Games which have been traditionally single-player are now expected to implement multiplayer as standard.
There are more third-parties offering middleware for networking than ever before and services such as Microsoft’s Xbox Live offer a framework that developers targeting that platform must implement, this simplifies both the development process and the end-user experience. Networking and developing across a large-scale environment is complicated, many developers find the subject tedious at best.  

Most programmers are used to writing applications which run in-order on a single thread. Even in the age of multi-threaded games, most developers only work with a single thread in their day to day tasks. Typically, different systems such as rendering, AI and networking will run on their own thread or will be split logically across a number of threads, in many cases this is hidden away from most of the developers by the engine and code is written as if it were a single-threaded environment.
Most modern programming languages such as C++ are designed in a way which makes their core functionality inherently single-threaded. Understanding code which is running in a multi-thread or multi-core environment is difficult. Debuggers have a hard time presenting the information in a form which can be easily understood by programmers, much multi-threaded code is powered by more abstract concepts such as messaging and queuing systems; it’s simply not possible for the debugger to deduce exactly how the systems hang together in order to present it in a way which would be easily understood.

In many ways, networking in a large-scale environment is similar to developing for a multi-threaded environment. Work is typically spread across multiple peers with each peer doing different bits of work concurrently with other peers.
There are numerous paradigms and implementations for these large-scale systems and going into each one and its advantages and disadvantages would be another post at the very least. In general these solutions fit into one of two categories:

1.       Each peer has a specific purpose, proxy or coordinator servers dish out work to the relevant peers as requests are received.

2.       All peers share the same functionality; work is spread out between the peers based on available resources.

It is not atypical to have a solution which fits into category two but also derives from category one. A website might have numerous web servers which can all deal with the same requests, but also a second layer of servers for database management.

You might, at around this point, wonder what the point of this post actually is.
As consumers expect more and more from online games and services, the games industry going to need more programmers who are familiar with the abstract and concurrent nature of large-scale network solutions. With the rising popularity of CELL, GPGPU and Intel’s Larrabee on the horizon the games industry is also going to need more and more programmers who have a firm grasp of multi-threaded development as consoles and PCs become more and more concurrent. I think we’re at an exciting time of technological development and in many cases games are one of the first places new technology is leveraged.
Most of all, the industry is going to need people who are interested and passionate about this stuff.

When people decide they want to be games programmers, very few of them think “I want to be a multiplayer programmer” or “I want to work on core tech libraries” such as those for multi-threading. Understandably people are more interested in the far more obvious areas of games such as rendering and that's a real shame.
It is my hope that the games industry will be able to get new developers interested in what is still one of the least explored areas of software development which is helping to drive some of the biggest changes the games industry has seen in recent years.  

I should probably say a little about me otherwise nobody will know who I am or what I do and that might be sort of pointless...
My name is Joe Hegarty and I'm a Network Programmer at Lionhead Studios which is part of Microsoft Game Studios, I'm currently working on Fable III and my job is basically to write the code that deals the multiplayer aspect of the game alongside the other Gameplay Programmers who specialise in other areas. This blog is not specifically for game development or Fable, you can check out the news on the Lionhead website for Fable 3. There are lots of sites about game development but I would advise looking into XNA if you're starting out.

As a network programmer a lot of the time I work on quite a low level (memory management, bit packing and the like) and I intend to focus on that in my blog. As I'm a C++ programmer the majority of what I post will naturally be in C++. However, I have been known to use C and ASM where appropiate and I'm always keen to learn something new! Recently I have been working on a TLSF heap allocator in my spare time and if I get it finished that might make up my first "proper" post.

I hope this blog will at least be somewhat informative or interesting.

Cheers,
Joe