Turn your desktop computer into a high-performance cluster with PelicanHPC


PelicanHPC includes two MPI implementations: LAM/MPI and OpenMPI. When writing parallel programs in C or C++, make sure you include the mpi.h header file (#include <mpi.h>). To compile the programs, you need mpicc for C programs, mpic++ or mpiCC for C++ programs, and mpif77 for Fortran.

Listing 1 has a sample "Hello World" program in C that uses the MPI library to print a message from all the nodes in the cluster. Compile it with mpicc:

mpicc borg-greeting.c -o borg-greeting

To run the programs you need to use mpirun:

mpirun -np 4 borg-greeting

This command tells the MPI library to explicitly run four copies of the hello app, scheduling them on the CPUs in the cluster in a round-robin fashion. Depending on the number of nodes in your cluster, you'll see something like:

We are borg! I am 1 of 4
We are borg! I am 3 of 4
We are borg! I am 0 of 4
We are borg! I am 2 of 4

Several MPI tutorials reside on the web [6]. Professor José Luis at the University of Seville in Spain uses PelicanHPC to teach his parallel programming course. He recommends that new programmers try the examples available online from Peter Pacheco's book, Parallel Programming with MPI [7].

Listing 1

"Hello, World" in C with MPI

01 #include <stdio.h>
02 #include "mpi.h"
04 int main(int argc, char *argv[ ])
05 {
06     int rank, size;
08     MPI_Init(&argc, &argv);
10     MPI_Comm_rank(MPI_COMM_WORLD, &rank);
11     MPI_Comm_size(MPI_COMM_WORLD, &size);
13     printf("We are borg! I am %d of %d\n", rank, size);
15     MPI_Finalize();
16     return 0;
17 }

See the OpenMPI website for additional documentation, including a very detailed FAQ [8].

Build Your Own PelicanHPC

If you're just interested in learning parallel programming, PelicanHPC provides more than enough. But the main goal of the Live CD is to help you get a cluster up and running without much effort. The focus is on maintainability and ease of customization, which is why the releases do not include a lot of packages.

Once you test the Live CD and think it'll work for you, you are encouraged to make your own versions via the Debian live-helper package and Pelican's make_pelican script. Also, you'll need a Debian or Ubuntu installation to produce the Live CD, which can be a minimal installation or even a virtual machine on a host with lots of RAM and a fast dual-core processor, which is what I use.

So to roll out your own ISO or USB image, first install a recent Ubuntu or Debian release. I've used Lenny to create a customized PelicanHPC release. Next, grab the live_helper package from the distro's repository. Finally, grab the latest version of the make_pelican script (currently v1.8) from Pelican's download page [4].

Open the script in your favorite text editor. The script is divided into various sections. After the initial comments, which include a brief changelog, the first section lists the packages that will be available on the ISO. Here is where you make the changes.

Listing 2 shows a modified version of this section, in which I've commented out the binary blobs for networking, because I don't need this for my networks. I've also added AbiWord and the GROMACS package. Because these packages are fetched off your distribution's repositories, make sure you spell them as they appear there. GROMACS has several dependencies but you don't have to worry about adding them because they'll be fetched automatically.

Listing 2

Packages For Your PelicanHPC Live CD

01 ### packages to add - place names of packages you want here ###
02 cat <<PACKAGELIST > addlist
03 # basic stuff needed for cluster setup
04 ssh dhcp3-server nfs-kernel-server nfs-common atftpd ifenslave
05 # binary blobs for networking
06 # firmware-bnx2 firmware-iwlwifi firmware-ralink linux-wlan-ng-firmware
07 # resource management
08 slurm-llnl slurm-llnl-sview slurm-llnl-basic-plugins
09 # configuration and tools
10 wget bzip2 dialog less net-tools rsync fping screen
11 make htop fail2ban locales console-common
12 # mail support
13 bsd-mailx liblockfile1 mailx postfix ssl-cert
14 # MPI
15 lam-runtime lam4-dev openmpi-bin openmpi-dev
16 # Octave
17 octave3.0 octave3.0-headers gnuplot
18 # Python
19 python-scipy python-matplotlib python-numpy ipython lampython
20 # other scientific
21 gfortran libatlas-headers libatlas3gf-base
23 gromacs
24 # X stuff
25 xorg xfce4 konqueror ksysguard ksysguardd kate kpdf
26 konsole kcontrol kdenetwork kdeadmin abiword

The next bit in the make_pelican script you have to tinker with is the architecture you want to build the ISO for and whether you want the ISO or USB image.

This section also specifies the series of network addresses doled out by PelicanHPC:


The rest of the script deals with PelicanHPC internals and shouldn't be tweaked unless you know what you're doing. However, it's advisable to browse through the other sections to get a better idea about how PelicanHPC magically transforms ordinary machines into extraordinary computing clusters.

When you've tweaked the script, execute it from the console:

sh make_pelican

Now sit back and enjoy, or if you have a slow connection and are running this on a slow computer, you better do your taxes because it'll take a while to fetch all the packages and compile them into a distro image.

When it's done, you'll have a shiny new ISO named binary.iso under either the i386/ or the amd64/ directory, depending on the architecture you build for. Now transfer the USB image onto a USB stick, or test the ISO image with VirtualBox or with Qemu before burning it onto a disc. Figure 5 shows the password screen of a modified PelicanHPC Live CD.

PelicanHPC is designed with ease of use in mind for anyone who wants to use their spare computers to do some serious number crunching. Building on the experience of ParallelKnoppix, the developer has put a lot of effort behind PelicanHPC's no-fuss approach to get your cluster off the ground in a jiffy. The customization abilities are the icing on the cake and make PelicanHPC an ideal platform for building your own custom cluster environment.

The Author

Mayank Sharma has written for various Linux publications, including Linux.com, IBMdeveloperWorks, and Linux Format, and has published two books through Packt on administering Elgg and Openfire. Occasionally he teaches FLOSS technologies. You can reach him via: http://http://www.geekybodhi.net.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

comments powered by Disqus

Direct Download

Read full article as PDF:

030-035_pelicanHPC.pdf (817.68 kB)