It's quite common in University departments for roomfuls of very capable PCs to sit idle from 5pm until 8am everyday and all weekend. Thankfully, it's possible to turn such a wasted resource into a powerful cluster of Linux machines using freely available open source software. In fact, even a group of machines in an office, or at home, can be transformed into a similar state with the minimum of effort. By booting the PCs (or nodes) over the network, no changes are made to the current setup of the machines, so reverting back to their normal state is as simple as rebooting. This guide will bring you through the process of creating a High-Performance Computing (HPC) cluster using the latest version of the Debian GNU/Linux operating system, Diskless Remote Boot in Linux (DRBL), and Sun's Open Source N1 Grid Engine 6.

Requirements

Creating a Linux cluster doesn't require a huge amount of resources, either in hardware or in time, but there is a bare minimum of hardware that you need to get started:

1. A moderately powerful (at least a Pentium 4 with 1Gb RAM) machine to act as the master node. This really needs to be a dedicated machine, as this will allow users to submit jobs whether the cluster is currently running or not.
2. The master node obviously needs a network card, but if you want to give the master or the nodes internet access, or are planning to have more than about 40 nodes, you will need two or three network cards. Even high-end Gigabit ethernet cards are cheap these days, and the way your nodes will communicate with the master can make or break a successful cluster.
3. Network infrastructure to connect all of the nodes together. A 100Mbit network is the absolute minimum, particularly if you will have more than 10 nodes. Gigabit ethernet is recommended.
4. One or more node PCs which support PXE booting (network booting). If the bios doesn't have an option to boot from the network card, or you don't see any messages about netbooting when you start the PCs, then you need to install a card which supports PXE. These nodes don't need to have a hard drive, but if they do (and you have some spare space) then it can be used for a swap partition (virtual memory in Windows-speak). It helps if the nodes contain similar hardware (such as is found in most offices or University computer rooms), as it reduces the amount of configuration required (almost nobody likes to rebuild the Linux kernel 20 times).

Download the script files

You will need to download some scripts which help setup GridEngine with DRBL (Python is required to run them, but is installed by default in Debian)

Install Debian

For this project, I used the latest release of Debian, 'Etch'. You can download it here. I'm not going to go through the installation process, as you can find plenty of excellent documentation over at the Debian website. I suggest heading straight for the installation manual. Thanks to the excellent installation scripts for Diskless Remote Boot Linux (DRBL), you can choose whatever installation type you want when installing Debian. For example, if you want to use graphical configuration utilities to set up your system, the select a desktop install, which will include the X window system. If you only plan to access the master over SSH then choose a lightweight console-only install.

Install DRBL

Again, I won't duplicate the excellent and thorough installation guide available from the DRBL website over here. However, for the purposes of this guide it is important that you choose the Single System Image (SSI) mode when installing. This means that all the nodes download the same system image over the network which provides the minimum filesystem to boot Linux, and the rest of the system, including /home, /usr and /opt are NFS mounted (accessed over the network) on the master node. Therefore, any software and user files on the master node are available to all of the cluster nodes. To deploy new software across the cluster, the most dramatic step you need to take is to reboot the nodes, something that DRBL will even do for you. A final point here is to take extra care when configuring the network setup of your nodes. During installation, DRBL will assume that any network card configured with a private address (e.g. 192.168.*) will contain nodes. If you will have over 40 machines on the network, consider splitting them up into two groups, and install two network cards in the machine with different subnets, e.g. 192.168.1.1/255.255.255.0 and 192.168.2.1/255.255.255.0. Having a multi-homed master node will increase the amount of data that you can push out to nodes.

Install Grid Engine

Before you go any further, make sure tht DRBL is installed, and that you can boot your nodes over the network from your master node. All following steps assume a working DRBL Single System Image setup with one or more nodes. Now that you have a network of machines, the next step is to install software which will manage the distribution and scheduling of jobs on the cluster. There are several alternatives to choose from, including Torque, SLURM, and Condor.

Note: I strongly suggest that you do not install or use the Condor project on your system(s). Not only is the source code only available on request, and only to those that the maintainers feel are 'suitable', but by default the program will report information about your installation back to the Condor servers on a regular basis. There is no mention of this during the installation procedure. To add to this, the license requires that any publication which includes data that has been analysed or processed using Condor must include a reference to the Condor project. Given that there are plenty of open source alternatives available, I don't recommend using this.

UPDATE: As of 6.9.5, Condor is now released under the Apache Licence, v. 2.0. Therefore, some of the above concerns have been addressed. In particular, the source code is now available, but only when you agree to fill in personal information. However, I am still unsure whether usage information is reported to the maintainers by default. Caveat Emptor.

For this guide, we will be installing Sun's N1 GridEngine 6. GridEngine is simple to install on Linux, requires only a minimum of installed libraries, is relatively easy to integrate with an existing DRBL setup, and provides a wide range of powerful tools to manage and use your cluster. There is also extensive documentation available online. Installing GridEngine will involve three setups:

1. Installing the qmaster on the master node
2. Installing the execution host on the master node
3. Configuring the nodes and GridEngine to work together

Installing qmaster

This step simply involves installing qmaster on the master node as per the online documentation. However, before starting the installation script, there are a few things you will need to do:

• Make sure the hostname of the master (or any of the nodes) doesn't also point to localhost or localhost.localdomain. SGE objects very strongly if it does. So,if your /etc/hosts file looks like this: 127.0.0.1 localhost mymaster 192.168.0.1 mymaster.mydomain.com mymaster then you will want to change it to: 127.0.0.1 localhost 192.168.0.1 mymaster.mydomain.com mymaster
• Create a file containing the hostnames of your DRBL nodes. This will be used during the GridEngine installation process, when you will need to supply a textfile which contains one hostname per line. This information is available from your DRBL installation at /etc/drbl/IP_HOSTS_TABLE. You can generate it using: create_drbl_hostlist.py /path/to/output/hostlist

During the qmaster installation process you can safely accept most of the defaults, but make sure you choose to:

• Install as root user You don't have to do this, but it simplifies the process of getting SGE to run on the DRBL nodes. Please note that in recommending this, I am presuming that your cluster network is private, and the nodes won't be accessible by non-privileged users (i.e. not you).
• Do opt to verify file permissions
• Select to use BerkleyDB, but without a spool server
• Use the ID range suggested in the manual: 20000-20100
• Accept to install startup scripts
• Accept to load a file which contains the hostnames of your nodes. Here you enter the full path to file you created before running the install script.
• Use normal scheduling

Once it's installed, source the SGE settings file in your .bashrc with the command: source sge-root/cell/common/settings.sh where sge-root is the SGE installation directory.

Install the execution host on the master server

We don't actually want to make the master node an execution node (although you may want to), as it's job is to manage and schedule jobs on the other nodes. However, as the nodes rely on the master node's filesystem for their applications, we will set up the execution daemons temporarily so that we may run them remotely on the nodes. Again, installation is as per the documentation, but there are some important choices to make:

• Specify a local spool dir: /var/tmp/spool
• Accept to install startup scripts

After the install is finished:

• Move /etc/init.d/sgeexecd to somewhere safe
• Run update-rc.d -n sgeexecd remove (because we don't want to start at boot time on the master node)
• Move sgeexecd script back into /etc/init.d (which will make it available to the nodes once we rerun the drblpush script)
• Add sgeexecd to /opt/drbl/conf/client-extra-service where /opt/drbl is your DRBL installation directory. This will make sure that the execution host daemon is started automatically when the nodes boots so they will be available to run jobs.
• On the master node, save the current execution host config as a template with: qconf -sconf hostname > node.conf.template where hostname is the hostname of the master. Open the resulting file with a text editor and if the first line is 'hostname:' then remove it (thanks Josh). The config includes the custom local spool dir /var/tmp/spool which we want to be set for each of our nodes, so that the spool is storted locally in ram, and not on the master. If all of the nodes saved their spool information to the master via NFS, it would place a greater strain on the master node and the network infrastructure. Storing the information locally means that network bandwidth is saved for booting nodes and transferring user data/scripts for execution on the cluster.
• Finally, run: create_sge_nodeconfig.py /path/to/node.conf.template This script will create a new GridEngine configuration for each of the DRBL nodes (as defined in /etc/drbl/IP_HOSTS_TABLE) based on the template we saved. Note: it is not strictly necessary to have a seperate configuration for each of the nodes, as the main purpose is to set the spool location to /var/tmp/spool. This is also possible by modifying the global configuration parameters of the GridEngine. However, I personally prefer having the ability to change the configuration of individual nodes if necessary without having to create a new configuration for them each time. As we use scripts to create and update the configurations, it doesn't require any effort to adopt either setup.

Configure the nodes

At the moment, a job queue ('all.q') has been created on the master, because we ran the execution host install script. For this cluster, we can either add all our nodes to that queue or create a seperate queue for each node. For this setup we will have a seperate job queue on all execution hosts. We now want to remove the all.q queue from the master, remove it as an execute node, and add queues to all of our DRBL hosts:

• Save the queue config for the master with: qconf -sq all.q > node.queue.template
• We'll now modify the queue config template so that we can easily recreate a new config for each node with a script. Open the queue conf template in your favourite text editor and make the following changes: qname %HOST%.q hostlist %HOST% slots 1 Note: if you have more than 1 processor on all your nodes then you can change the number to reflect this shell /bin/bash Set this to whatever shell you want to use for jobs
• Run create_sge_nodequeues.py /path/to/queue.template This will create a new queue on each node with one slot per processor, enabling jobs to be submitted and executed.
• Delete the all.q queue on the master as we won't be using it: qconf -dq all.q
• And now, remove the @allhosts group because we have made individual queues for each node, so we don't need it either: qconf -dhgrp @allhosts
• And last we remove the master as an execution node. It was set as such by running the execution host installation script, but we can undo that with: qconf -de master

If you now run qhost -q | more you should see a list of all your DRBL nodes and below each should be a queue definition. Running qstat -f should now list the various queues that were created, and show how many slots each of them have and how many slots are filled at the moment. Now, rerun drblpush -i or drblpush -c /etc/drbl/drblpush and restart one of your nodes. ssh into the node when it boots and confirm that sgeexecd is running with: ps ax | grep sge. If it isn't, check for startup messages in /tmp/execd*. Back on the master, rerun qhost -q | more. The node that you have rebooted should be listed with extra information than the others, such as free memory, number of processors, and load. You can now boot the rest of your nodes to create a diskless cluster with GridEngine and DRBL!

Setting up MPI/MPICH support on GridEngine

If you want to configure MPI/MPICH support, there are two steps which need to be completed. First, you must add a new parallel environment to GridEngine which sets up the mpi/mpich startup and shutdown scripts that are provided with SGE for use with MPI/MPICH jobs. Second, you must add your new parallel environment to each of your already existing job queues. For this example we will be setting up mpich as our parallel environment. If you want non-tight MPI support, just replace mpich with mpi throughout:

• Make a copy of the mpich.template in the mpi directory
• Fill in the information that is missing (marked with <>) in the template, such as number of slots (just put 999) and the SGE root dir (for me it's /opt/n1ge6)
• Now add the parallel environment with: qconf -Ap /path/to/modified/mpich.template You can check it exists with qconf -spl and qconf -sp mpich
• Next, modify the node.queue.template that we created in a previous step, and change the 'pe_list' from 'make' to 'make mpich'. This will enable parallel jobs to be run on the queue.
• Finally, update all our queues (one for each DRBL node) with: update_sge_nodequeues.py /path/to/modified/node.queue.template
• You can then check the updated queue definitions with: qconf -sq <queuename> where queuename is one of the queues on a DRBL node.

Using QMon

QMon is the graphical interface to the Sun GridEngine system. To run it, you need some extra libraries installed. Unfortunately, they haven't (and probably won't) have a release candidate for Debian, so you need to get it manually from the Debian servers:

• Download the package for your install here.
• Install with dpkg -i package-name.deb
• Type qmon to start it.