The Moth

Like last year, I was going to attend SC this year, but other events are unfortunately keeping me here in Seattle next week. If you are going to be in New Orleans, have fun and be sure not to miss out on the following two opportunities.

MPI Debugging UX Study

Throughout the week, my team is conducting 90-minute studies on debugging MPI applications within Visual Studio. In exchange for your feedback (under NDA) you will receive a Microsoft Gratuity (and the knowledge that you are impacting the development of Visual Studio). If you are interested, sign up at the Microsoft Information Desk in the Exhibitor Hall during exhibit hours. Outside of exhibit hours, send email to tcur@microsoft.com. If you took part in the GPGPU study, this is very similar except it is for MPI.

Microsoft High Performance Computing Summit

On Monday 15th, the Microsoft annual user group meeting takes place. Shuttle transportation and lunch is provided. For full details of this event and to register, please visit the official event page.

Microsoft Research (MSR) researches technologies, incubates projects which many times result in technology that looks like a ready-to-use product (but it is important to understand that these are not the same as products built by the various… actual product teams here at Microsoft).

A very popular MSR project has been DryadLINQ, which itself builds on Dryad. To learn more follow the project pages I just linked to and I also recommend this 1-hour channel 9 video. If you only have 3 minutes, watch this great elevator pitch instead. You can also stay tuned on the official blog, which includes a post that refers to internal adoption e.g by Bing, a quick DryadLINQ code example, and some history on how DryadLINQ generalizes the MapReduce pattern and makes it accessible to regular programmers (see this post and that post).

Essentially, the DryadLINQ framework (building on the Dryad runtime) allows developers to re-use their LINQ skills for creating/generating programs that process large multi-gigabyte/terabyte datasets across 100s-1000s of machines. One way to think about it is that just as Parallel LINQ allows LINQ developers to seamlessly use multiple cores from a single process on a single machine, DryadLINQ allows LINQ developers to seamlessly use multiple machines for their data parallel algorithms. In the former scenario the motivation was speed of execution, in the latter it is speed of execution AND processing large datasets that simply don't fit on a single machine.

Whenever I hear about execution of parallel code on multiple machines on the Microsoft platform, I immediately think of Windows HPC Server. Indeed Dryad and DryadLINQ were made available for Windows HPC Server and I encourage you to watch the PDC session on this topic: Data-Intensive Computing on Windows HPC Server with the DryadLINQ Framework.

Watch this space…

Internally and unofficially we refer to this as "HPC Server v3" and its Beta2 became available last week. Read the full story on this blog post from Ryan and this one from Don.

There has been a lot of excitement on the web for this release with coverage from last Wednesday here, here, here, here, here and here.

Don't forget that Visual Studio 2010 makes it easy to develop for HPC Server including the MPI Cluster Debugger integration that I explained here and here.

I've blogged before about the MPI Cluster Debugger in VS2010 that facilitates launching the application on the cluster and attaching the debugger (btw, a shorter version of the screencast I link to there, is here).

There have been requests for the code I use in the screencast, so please find a ZIP with that code.

There have also been requests for a PowerPoint deck to use when showing this feature to others. Feel free to download some slides I threw together the other day.

Let's assume that you have all the HPC bits installed and that you have existing MPI code (or you created a "Hello World" project using the MPI project template). Of course, you create a single MPI application and at runtime it will correspond to multiple processes (of the same app) launched on multiple nodes (i.e. machines) on the cluster. So how do you debug such a situation by simply hitting the familiar "F5" keystroke (i.e. Debug -> Start Debugging)?

WATCH IT INSTEAD OF READING ABOUT IT
If you can't bear to read through all the details below, just watch this 19-minute screencast explaining this VS2010 feature. Alternatively, or even additionally, keep on reading.

REQUIREMENT
When you debug an MPI application, you would want the copying of resources from your client machine (where Visual Studio is installed) to each compute node (where Windows HPC Server is installed) to take place automatically for you. 'Resources' in the previous sentence includes your application binary, plus any binary or data dependencies it may have, plus PDBs if needed, plus the debug CRT of the correct bitness, plus msvsmon for remote debugging to work. You would also want, after copying is complete, to have your app and msvsmon launched and attached so that you can hit breakpoints back in Visual Studio on your client machine. All these thing that you would want are delivered in VS2010.

STEPS TO F5
1. In your MPI project where you have placed a breakpoint go to Project Properties -> Configuration Properties -> Debugging. Ensure the "Debugger to launch" combo box value is set to MPI Cluster Debugger.

2. There are a whole bunch of properties here and typically you can ignore all of them except one: Run Environment. By default it is set to run 1 process on your local machine and if you change the number after that to, for example, 4 it will launch 4 processes of your app on your local machine.

You want this to run on your cluster though, so go to the dropdown arrow at the end of the Run Environment cell and open it to expose the "Edit Hpc node" menu which opens the Node Selector dialog:

In this dialog you can enter (or pick from a list) the cluster head node name and then the number of processes you want to execute on the cluster and then hit OK and… you are done.

3. Press F5 and watch your breakpoint get hit (after giving it some time for copying, remote execution, attachment and symbol resolution to take place).

GOING DEEPER
In the MPI Cluster Debugger project properties above, you can see many additional properties to the Run Environment. They are all optional, but you may want to understand them in order to fine tune your cluster debugging. Read all about each one of these on the MSDN page Configuration Properties for the MPI Cluster Debugger.

In the Node Selector dialog above you can see more options than just the Head Node name and Number of Process to run. They should be self-explanatory but I also cover them in depth in my screencast showing you an example of why you would choose to schedule processes per core versus per node. You can also read about these options on MSDN as part of the page How to: Configure and Launch the MPI Cluster Debugger.

To read through an example that touches on MPI project creation, project properties, node selector, and also usage of MPI with OpenMP plus MPI with PPL, read the MSDN page Walkthrough: Launching the MPI Cluster Debugger in Visual Studio 2010.

Happy MPI debugging!

So you have installed your cluster and you are done with introductory material on Windows HPC. Now you want to develop an application with the most common programming model: Message Passing Interface.

The MPI programming model is a standard with implementations from many vendors. For newbies (like myself!), I have aggregated below links for getting started.

Non-Microsoft MPI resources (useful even if you are not on the Windows platform)
1. Message Passing Interface on wikipedia.

2. The MPI standard.
3. MPICH2 - an MPI implementation.
4. Tutorial on MPI by William Gropp.
5. MPI patterns presented as a tutorial with sample code.

6. THE official MPI Forum (maintains the standard) including the wiki discussing the MPI future.

7. Great MPI tutorial including at the end the MPI Exercise.

8. C++ MPI Exercises by John Burkardt.

9. Book online: MPI The Complete Reference.


MS-MPI
10. Windows HPC Server 2008 - Using MS-MPI whitepaper (15 page doc).

11. Tracing MPI applications (27 page doc).

12. Using Microsoft MPI (TechNet section).

13. Windows HPC Server MPI forum (for posting questions).


MPI.NET
14. MPI.NET Home Page (not owned by Microsoft).
15. MPI.NET Tutorial.

16. HPC Development using F# using MPI.NET (38 page doc).


Next time I'll post resources for the Microsoft Cluster SOA programming model - happy coding...

I've already described how to setup a Windows HPC Server for development. Before you dive into developing for the cluster, if you are new to this it is probably a good idea to learn the basics by reading some overview material. Below is a list of links.

Direct Links to Windows HPC content
1. Windows HPC Server 2008 Overview Datasheet (4 page pdf).

2. Windows HPC Server 2008 Technical Overview (32 page doc).

3. Windows HPC Server 2008 Getting Started Guide (26 page doc) which actually is available online as part of the TechNet technical library section on Windows HPC Server 2008, which includes much more useful data.

4. Windows HPC Server 2008 Job Scheduler (38 page doc).
5. Windows HPC Server 2008 Job Templates (56 page doc).

6. Developing for the Windows HPC Server 2008 Platform (16 page doc or pdf version).


Windows HPC sites
7. Windows HPC Forums.

8. HPC Developer Resources.

9. Windows HPC Server 2008 Resource Kit - Developer.

10. Windows HPC Server 2008 - TechNet.

11. The Windows HPC Team Blog.


HPC Course
12. High-Performance Computing Fundamentals Course (pluralisight)
13. Classic HPC Development using Visual C++ (course slides and materials in a ZIP). Author's blog post.
14. From sequential to parallel code (course slides and materials in a ZIP). Author's blog post.


Next time I will post resources specific to the most popular programming models for the cluster today: MPI and Cluster SOA - until then, happy reading!

If you are developing MS MPI applications with Visual Studio 2010, you are probably tired of following some tedious steps for every new C++ project that you create, similar to the following:

1. In Solution Explorer, right-click YourProjectName, then click Properties to open the Property Pages dialog box.

2. Expand Configuration Properties and then under VC++ Directories place the cursor at the beginning of the list that appears in the Include Directories text box and then specify the location of the MS MPI C header files, followed by a semicolon, e.g.
C:\Program Files\Microsoft HPC Pack 2008 SDK\Include;

3. Still under Configuration Properties and under VC++ Directories place the cursor at the beginning of the list that appears in the Library Directories text box and then specify the location of the Microsoft HPC Pack 2008 SDK library file, followed by a semicolon, e.g.
if you want to build/debug 32bit application:
C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\i386;
if you want to build/debug 64bit application:
C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\amd64;

4. Under Configuration Properties and then under Linker, select Input and place the cursor at the beginning of the list that appears in the Additional Dependencies text box and then type the name of the MS MPI library, i.e.
msmpi.lib;

5. In the code file
#include "mpi.h"

6. To debug the MPI project you have just setup, under Configuration Properties select Debugging and then switch the Debugger to launch combo value from Local Windows Debugger to MPI Cluster Debugger.

Wouldn't it be great if at C++ project creation time you could choose an MPI Project Template that included the steps/configurations above? If you answered "yes", I have good news for you courtesy of a developer on our team (Qing).

Feel free to download from Visual Studio gallery the MPI Project Template.

Recently I decided to play around with developing for a cluster on the Microsoft platform and below are the steps I had to take with regards to installation.

First I gathered a number of old machines at the office, in my case that is 3 dual-core boxes, but you could have done it with just 2 PCs (or many more of course). On each one of those you must install: Windows HPC Server 2008 (link to trial)

a. This includes a trial of the Windows Server 2008 RTM 64-bit operating system (SRVHPC_EN.iso). I already had Windows Server on a separate disc, so I installed what I had. I did not try R2, which is the same code base as Win7, but that should work too. Note that this is just the vanilla operating system (even Standard Edition is good enough) and the important part is the 64-bit. I will assume that you are all capable of installing an OS, so no more instruction or special consideration needed here (window update, join domain, add users etc).

b. The second part of the link above is an add-on to the operating system, namely the HPC Pack 2008 (HPCEval.iso). It is a wizard with a set of steps to complete which are all fairly explanatory of the "click Next" flavor. After installation of the HPC Pack, the HPC Cluster Manager application runs up automatically (or else you can run it through the Start menu) for further step-by-step configuration in an easy to follow outofthebox To-do List.

Here are some tips for step b from above, for your developer cluster setup:

i) In the HPC Pack wizard, you will create one of the machines as the Head Node.

ii) In the HPC Pack wizard, the remainder machines will be created as the Compute Nodes.

iii) In Cluster Manager, click on "Configure your network" and choose topology 5 "All Nodes only on an Enterprise Network".

iv) In Cluster Manager, click on "Add compute nodes" and then select "Add compute nodes that have already been configured".

v) In Cluster Manager, click "Change the role of the head node" and also make it a WCF Broker Node.

vi) In Cluster Manager, "Validate your cluster" under Diagnostics.

It may be useful to cross-reference the text above with the following screenshot of HPC Cluster Manager


Now that your cluster is installed, you need to set up your development machine (x86 or x64) for developing on this cluster. Here are the things you need to install:

1. Visual Studio 2010

2. HPC Pack 2008. This is essentially the same msi you installed on the server (step b above), but here you will choose the 3rd option Client Utilities (which will probably be your only enabled option as per my screenshot further above).

3. HPC Pack 2008 SDK. This allows writing client applications that interact with the Job Scheduler running on the Head Node. MSDN has a dedicated page to the HPC SDK.

4. (optional) Install the separate MPI Project Template, for VS2010.

You are now ready to develop your cluster applications on your development machine and to debug/deploy them on your cluster. More on that in future blog posts.