The Moth

Just realized that I have not linked from here to a screencast I recorded a couple weeks ago that shows the API, parallel debugger and concurrency visualizer in VS2010. Take a few minutes to watch the VS2010 Parallel Computing Features Tour.

Let's assume that you have all the HPC bits installed and that you have existing MPI code (or you created a "Hello World" project using the MPI project template). Of course, you create a single MPI application and at runtime it will correspond to multiple processes (of the same app) launched on multiple nodes (i.e. machines) on the cluster. So how do you debug such a situation by simply hitting the familiar "F5" keystroke (i.e. Debug -> Start Debugging)?

WATCH IT INSTEAD OF READING ABOUT IT
If you can't bear to read through all the details below, just watch this 19-minute screencast explaining this VS2010 feature. Alternatively, or even additionally, keep on reading.

REQUIREMENT
When you debug an MPI application, you would want the copying of resources from your client machine (where Visual Studio is installed) to each compute node (where Windows HPC Server is installed) to take place automatically for you. 'Resources' in the previous sentence includes your application binary, plus any binary or data dependencies it may have, plus PDBs if needed, plus the debug CRT of the correct bitness, plus msvsmon for remote debugging to work. You would also want, after copying is complete, to have your app and msvsmon launched and attached so that you can hit breakpoints back in Visual Studio on your client machine. All these thing that you would want are delivered in VS2010.

STEPS TO F5
1. In your MPI project where you have placed a breakpoint go to Project Properties -> Configuration Properties -> Debugging. Ensure the "Debugger to launch" combo box value is set to MPI Cluster Debugger.

2. There are a whole bunch of properties here and typically you can ignore all of them except one: Run Environment. By default it is set to run 1 process on your local machine and if you change the number after that to, for example, 4 it will launch 4 processes of your app on your local machine.

You want this to run on your cluster though, so go to the dropdown arrow at the end of the Run Environment cell and open it to expose the "Edit Hpc node" menu which opens the Node Selector dialog:

In this dialog you can enter (or pick from a list) the cluster head node name and then the number of processes you want to execute on the cluster and then hit OK and… you are done.

3. Press F5 and watch your breakpoint get hit (after giving it some time for copying, remote execution, attachment and symbol resolution to take place).

GOING DEEPER
In the MPI Cluster Debugger project properties above, you can see many additional properties to the Run Environment. They are all optional, but you may want to understand them in order to fine tune your cluster debugging. Read all about each one of these on the MSDN page Configuration Properties for the MPI Cluster Debugger.

In the Node Selector dialog above you can see more options than just the Head Node name and Number of Process to run. They should be self-explanatory but I also cover them in depth in my screencast showing you an example of why you would choose to schedule processes per core versus per node. You can also read about these options on MSDN as part of the page How to: Configure and Launch the MPI Cluster Debugger.

To read through an example that touches on MPI project creation, project properties, node selector, and also usage of MPI with OpenMP plus MPI with PPL, read the MSDN page Walkthrough: Launching the MPI Cluster Debugger in Visual Studio 2010.

Happy MPI debugging!

Using Visual Studio 2010 parallel debugging is easy. Two new debugging windows provide a total view of the internals of your PPL and TPL applications with hints on where to start investigations. These are not mere extensions to VS, but tightly integrated with the rest of the debugger experience, so you don't need to learn many new techniques. Use them in your program to eclipse bugs from existence!

One of the most FAQ I receive is links to VS2010 parallel debugging content and rather than keep sending many, I decided to gather them all under one permalink, hence this multi link blog post.

- MSDN Magazine article on Parallel Debugging.
- Screencast of sample code from the article.

- MSDN Walkthrough: Debugging a Parallel Application (VB, C++, C#).
- Screencast of walkthrough for Parallel Stacks.
- Screencast of walkthrough for Parallel Tasks.

- MSDN "How To" on Parallel Tasks.
- MSDN "How To" on Parallel Stacks.

- Detailed blog post on Parallel Tasks.
- Detailed blog post on Parallel Stacks.
- Detailed blog post on Parallel Stacks - Tasks View.
- Detailed blog post on Parallel Stacks - Method View.

- Download slides on Parallel Tasks and Parallel Stacks (pptx).

If you have questions on these, please post to any of the parallel computing forums or the debugging forum (your question will be routed to me if nobody else can answer it).

Over at the Intel Software Network Aaron Tersteeg runs a "Parallel Programming Talk" audio show on which I was invited as a guest (for the 55th episode) to talk about Microsoft's parallelism offerings in Visual Studio 2010. The call started at 7:45AM, so if my voice sounds croaky to you, now you know why ;)

Check out the 20-minute chat (and related hyperlinks) on Aaron's blog.

So you have installed your cluster and you are done with introductory material on Windows HPC. Now you want to develop an application with the most common programming model: Message Passing Interface.

The MPI programming model is a standard with implementations from many vendors. For newbies (like myself!), I have aggregated below links for getting started.

Non-Microsoft MPI resources (useful even if you are not on the Windows platform)
1. Message Passing Interface on wikipedia.

2. The MPI standard.
3. MPICH2 - an MPI implementation.
4. Tutorial on MPI by William Gropp.
5. MPI patterns presented as a tutorial with sample code.

6. THE official MPI Forum (maintains the standard) including the wiki discussing the MPI future.

7. Great MPI tutorial including at the end the MPI Exercise.

8. C++ MPI Exercises by John Burkardt.

9. Book online: MPI The Complete Reference.


MS-MPI
10. Windows HPC Server 2008 - Using MS-MPI whitepaper (15 page doc).

11. Tracing MPI applications (27 page doc).

12. Using Microsoft MPI (TechNet section).

13. Windows HPC Server MPI forum (for posting questions).


MPI.NET
14. MPI.NET Home Page (not owned by Microsoft).
15. MPI.NET Tutorial.

16. HPC Development using F# using MPI.NET (38 page doc).


Next time I'll post resources for the Microsoft Cluster SOA programming model - happy coding...

I've already described how to setup a Windows HPC Server for development. Before you dive into developing for the cluster, if you are new to this it is probably a good idea to learn the basics by reading some overview material. Below is a list of links.

Direct Links to Windows HPC content
1. Windows HPC Server 2008 Overview Datasheet (4 page pdf).

2. Windows HPC Server 2008 Technical Overview (32 page doc).

3. Windows HPC Server 2008 Getting Started Guide (26 page doc) which actually is available online as part of the TechNet technical library section on Windows HPC Server 2008, which includes much more useful data.

4. Windows HPC Server 2008 Job Scheduler (38 page doc).
5. Windows HPC Server 2008 Job Templates (56 page doc).

6. Developing for the Windows HPC Server 2008 Platform (16 page doc or pdf version).


Windows HPC sites
7. Windows HPC Forums.

8. HPC Developer Resources.

9. Windows HPC Server 2008 Resource Kit - Developer.

10. Windows HPC Server 2008 - TechNet.

11. The Windows HPC Team Blog.


HPC Course
12. High-Performance Computing Fundamentals Course (pluralisight)
13. Classic HPC Development using Visual C++ (course slides and materials in a ZIP). Author's blog post.
14. From sequential to parallel code (course slides and materials in a ZIP). Author's blog post.


Next time I will post resources specific to the most popular programming models for the cluster today: MPI and Cluster SOA - until then, happy reading!

One of the powerful aspect of Visual Studio is its ability to be extended and many people do that. You can find numerous extensions at the Visual Studio Gallery. The VSX team links to a 4-part blog series on how to create and share templates. You can also look find extension examples on the vsx code gallery.

With Visual Studio 2010, you can search for items and install them directly from within Visual Studio's new Extension Manager. You launch it from the Tools menu:


When the dialog comes up, be sure to explore the various actionable areas on the left and also note the search on the right. For example, I typed "MP" and it quickly filtered the list to show me the MPI Project Template:


Others have written about this before me, just bing Extension Manager (and note that Beta2 introduced changes, some of which you can witness in the screenshot above).

Since my team offers "parallel debugging", we refer to the team delivering all the other debugging features as the "core debugger" team. They have published a video of new VS2010 debugger features that I encourage you to watch to find out about enhancements with DataTips, breakpoints, dump debugging (inc. IL interpreter) and Threads window.

The raw list of features with short description is also here.

One of the areas where we fixed many bugs for Beta2 in our parallel debugging windows is with regards to managed dump debugging. So it was really cool to see Tess use the Parallel Stacks window in that scenario in her video demo with Scott.

Other than the neat ability to open managed dumps in VS2010, Parallel Stacks was the only debugging feature she needed for diagnosing the issue! Check out the video, definitely worth 10 minutes of your time.

Recently I gave a talk at our Microsoft Shanghai offices on Parallel Programming so I had to update my existing Beta1 deck to Beta2 content. Specifically for Parallel Tasks and Parallel Stacks, I used 5 slides to accompany the demo.

In case you are giving talks on parallelism within Visual Studio 2010, please feel free to download and use the updated parallel debugger slides (pptx).

TIP: The slides have animations so be sure to F5 the deck for the full benefit and they also have text in the Comments section so be sure to see them at design time too.

If you are developing MS MPI applications with Visual Studio 2010, you are probably tired of following some tedious steps for every new C++ project that you create, similar to the following:

1. In Solution Explorer, right-click YourProjectName, then click Properties to open the Property Pages dialog box.

2. Expand Configuration Properties and then under VC++ Directories place the cursor at the beginning of the list that appears in the Include Directories text box and then specify the location of the MS MPI C header files, followed by a semicolon, e.g.
C:\Program Files\Microsoft HPC Pack 2008 SDK\Include;

3. Still under Configuration Properties and under VC++ Directories place the cursor at the beginning of the list that appears in the Library Directories text box and then specify the location of the Microsoft HPC Pack 2008 SDK library file, followed by a semicolon, e.g.
if you want to build/debug 32bit application:
C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\i386;
if you want to build/debug 64bit application:
C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\amd64;

4. Under Configuration Properties and then under Linker, select Input and place the cursor at the beginning of the list that appears in the Additional Dependencies text box and then type the name of the MS MPI library, i.e.
msmpi.lib;

5. In the code file
#include "mpi.h"

6. To debug the MPI project you have just setup, under Configuration Properties select Debugging and then switch the Debugger to launch combo value from Local Windows Debugger to MPI Cluster Debugger.

Wouldn't it be great if at C++ project creation time you could choose an MPI Project Template that included the steps/configurations above? If you answered "yes", I have good news for you courtesy of a developer on our team (Qing).

Feel free to download from Visual Studio gallery the MPI Project Template.