If there is anything that the recent GMail outages taught us, it's that losing access to your email for a few hours should not mean that the end of the world has arrived. I think it also served as a reminder that if you rely 100% on the cloud for access to your data, then be prepared to be disconnected from it every once in a while. Fortunately, it's quite easy to take advantage of free email services from the likes of Google while also making sure that should the worst happen, you can still access your data. With more and more communication, both professional and personal, being conducted through email, it's no wonder that people do get nervous when they suddenly lose control over several years worth of contacts and emails. This short article will show you how easy it is to back up your GMail automatically to a remote server which you can then access in the event that GMail goes down or you lose access to your account. We'll be using nothing but free open-source software. If you already have a net-connected machine with Ubuntu installed then you're set to go. Otherwise, consider recycling an old pentium-or-better-cpu machine with a network card—the processing demands are relatively low and the most important resource is storage space for all the emails and attachments. Although these instructions are for Ubuntu, they should also work on Debian or any other Debian-based distro. To set this up you obviously need a GMail account, although any email account which can be accessed using the POP protocol can be backed up using the same method. We'll be using the excellent Fetchmail to retrieve messages from the GMail account while leaving them untouched in your GMail inbox, Procmail to deliver and filter them into a local Maildir-format mailbox, and Dovecot to access the emails remotely, all on the latest release of Ubuntu, Hardy Heron (8.04 LTS).

1. Setup GMail

• Log in to your GMail account
• Click 'Settings>Imap/Pop' and enable POP access
• You can chose whether to only download messages which arrive from now on or to download all existing messages. The latter is useful if you have already been using your GMail account for a while and would like to back all existing messages to a secure alternative location.

2. Install packages

• Fire up a terminal window and run the following command: sudo apt-get install dovecot-imapd procmail fetchmail mutt
• This should automatically download and install everything you need. The mutt package is optional and is a useful tool for checking if your IMAP server is working properly and that your emails are being downloaded (not to mention a nice command-line email client!).

3. Configure Fetchmail

• We need to define our GMail accounts in fetchmail's configuration file. Open the file /etc/fetchmailrc for editing as root. The file won't exist yet so we'll have to create it.
• Copy & paste the following into the file:
  set daemon 60 set syslog set postmaster "localuser" poll pop.gmail.com protocol pop3 username "username@gmail.com" there with password "yourpassword" is localuser here keep ssl mda "/usr/bin/procmail -d localuser"
• Change localuser to your local username on the linux box and the email and password to match your gmail account. If you want, you can add as many other accounts as you want here. For each account, just copy the text starting from 'poll...' and change the login details.
• The first line tells fetchmail to check for new emails every 60 seconds. Edit to match your preferred update frequency.
• Edit /etc/default/fetchmail as root and change the line START_DAEMON=no to START_DAEMON=yes

4. Configure Procmail (~/.procmailrc)

• We're using procmail to deliver email from fetchmail to a mailbox in Maildir format. This is necessary due to the lack of Maildir support in Fetchmail, which supports the mbox format. The main difference between the formats is that Maildir stores each email in a single file, making it very efficient when you have a lot of emails. In contrast, mbox format stores all emails in a single file.
• Create ~/.procmailrc and put the following text in it:
  MAILDIR=$HOME/Mail/ DEFAULT=$MAILDIR VERBOSE=on LOGFILE=$MAILDIR/procmail.log :0 $MAILDIR
• Set permissions on the file with: chmod 640 ~/.procmailrc
• This file simply tells procmail where to store the emails (in ~/Mail) and where to store it's logs. Procmail is capable of a lot more than this, and it supports a powerful filtering language. You can specify rules in your ~/.procmailrc file to filter emails based on sender, subject, etc.

5. Configure Dovecot

• We're almost there! We can retrieve emails from the GMail servers and deliver them locally. Now we need a way to access the email remotely. For this we're using Dovecot, and we need to tweak the configure a little.
• As root, open up /etc/dovecot/dovecot.conf and update the following lines:
  protocols = imaps mail_location = maildir:~/Mail

6. Setup maildir

• We need to create the local mailbox with: maildirmake.dovecot ~/Mail

7. Start services

Okay, now everything is set up we need to start up the services: sudo /etc/init.d/fetchmail start sudo /etc/init.d/dovecot start

8. Enjoy!

That's it! Run tail -f /var/log/syslog to check fetchmail is receiving messages and also check ~/Mail/procmail.log to see that procmail is not throwing up any errors. The final thing you need to do is make sure you can access your email server remotely. Dovecot is running over SSL on port 993, so just configure your router to enable external access to that port. Then just configure Outlook , Thunderbird, etc. to access IMAP over SSL and you have full access to all your emails on your own server. Note: if you chose to setup GMail POP access to allow all existing messages to be downloaded and you have lots of messages in your account, it will take a few runs of fetchmail to collect them all. Leave things running overnight and check back in the morning!

Optional step: Setup mutt

• Create the file ~/.muttrc and put the following in it:
  set mbox_type=Maildir set folder="~/Mail" set mask="!^\\.[^.]" set mbox="~/Mail" set record="+.Sent" set postponed="+.Drafts" set spoolfile="~/Mail"
• Now just run mutt and read your email!

Related Links

• Set Up a Debian or Ubuntu Machine as a Maildrop
• Dovecot
• Fetchmail
• Procmail

Permalink | Leave a comment  »

Anyone who needs to access a unix/linux/osx machine from windows via ssh will be familiar with PuTTY, the free ssh client. PuTTYGen, available to download here, is a handy utility for creating RSA/DSA public/private keys for authentication. If you have generated your RSA keys using PuTTYGen, and would like to use them with OpenSSH on Ubuntu, you just need to follow a few simple steps:

• Install putty with: sudo apt-get install putty
• Create the public key file:
  puttygen /path/to/puttykey.ppk -L > ~/.ssh/id_rsa.pub
• Create the private key file:
  puttygen /path/to/puttykey.ppk -O private-openssh -o ~/.ssh/id_rsa

You should now be able to log into an SSH server using your private key. To install your public key simply copy from ~/.ssh/id_rsa.pub on your local machine to ~/.ssh/authorized_keys on the remote server.

Permalink | Leave a comment  »

It's quite common in University departments for roomfuls of very capable PCs to sit idle from 5pm until 8am everyday and all weekend. Thankfully, it's possible to turn such a wasted resource into a powerful cluster of Linux machines using freely available open source software. In fact, even a group of machines in an office, or at home, can be transformed into a similar state with the minimum of effort. By booting the PCs (or nodes) over the network, no changes are made to the current setup of the machines, so reverting back to their normal state is as simple as rebooting. This guide will bring you through the process of creating a High-Performance Computing (HPC) cluster using the latest version of the Debian GNU/Linux operating system, Diskless Remote Boot in Linux (DRBL), and Sun's Open Source N1 Grid Engine 6.

Requirements

Creating a Linux cluster doesn't require a huge amount of resources, either in hardware or in time, but there is a bare minimum of hardware that you need to get started:

1. A moderately powerful (at least a Pentium 4 with 1Gb RAM) machine to act as the master node. This really needs to be a dedicated machine, as this will allow users to submit jobs whether the cluster is currently running or not.
2. The master node obviously needs a network card, but if you want to give the master or the nodes internet access, or are planning to have more than about 40 nodes, you will need two or three network cards. Even high-end Gigabit ethernet cards are cheap these days, and the way your nodes will communicate with the master can make or break a successful cluster.
3. Network infrastructure to connect all of the nodes together. A 100Mbit network is the absolute minimum, particularly if you will have more than 10 nodes. Gigabit ethernet is recommended.
4. One or more node PCs which support PXE booting (network booting). If the bios doesn't have an option to boot from the network card, or you don't see any messages about netbooting when you start the PCs, then you need to install a card which supports PXE. These nodes don't need to have a hard drive, but if they do (and you have some spare space) then it can be used for a swap partition (virtual memory in Windows-speak). It helps if the nodes contain similar hardware (such as is found in most offices or University computer rooms), as it reduces the amount of configuration required (almost nobody likes to rebuild the Linux kernel 20 times).

Download the script files

You will need to download some scripts which help setup GridEngine with DRBL (Python is required to run them, but is installed by default in Debian)

Install Debian

For this project, I used the latest release of Debian, 'Etch'. You can download it here. I'm not going to go through the installation process, as you can find plenty of excellent documentation over at the Debian website. I suggest heading straight for the installation manual. Thanks to the excellent installation scripts for Diskless Remote Boot Linux (DRBL), you can choose whatever installation type you want when installing Debian. For example, if you want to use graphical configuration utilities to set up your system, the select a desktop install, which will include the X window system. If you only plan to access the master over SSH then choose a lightweight console-only install.

Install DRBL

Again, I won't duplicate the excellent and thorough installation guide available from the DRBL website over here. However, for the purposes of this guide it is important that you choose the Single System Image (SSI) mode when installing. This means that all the nodes download the same system image over the network which provides the minimum filesystem to boot Linux, and the rest of the system, including /home, /usr and /opt are NFS mounted (accessed over the network) on the master node. Therefore, any software and user files on the master node are available to all of the cluster nodes. To deploy new software across the cluster, the most dramatic step you need to take is to reboot the nodes, something that DRBL will even do for you. A final point here is to take extra care when configuring the network setup of your nodes. During installation, DRBL will assume that any network card configured with a private address (e.g. 192.168.*) will contain nodes. If you will have over 40 machines on the network, consider splitting them up into two groups, and install two network cards in the machine with different subnets, e.g. 192.168.1.1/255.255.255.0 and 192.168.2.1/255.255.255.0. Having a multi-homed master node will increase the amount of data that you can push out to nodes.

Install Grid Engine

Before you go any further, make sure tht DRBL is installed, and that you can boot your nodes over the network from your master node. All following steps assume a working DRBL Single System Image setup with one or more nodes. Now that you have a network of machines, the next step is to install software which will manage the distribution and scheduling of jobs on the cluster. There are several alternatives to choose from, including Torque, SLURM, and Condor.

Note: I strongly suggest that you do not install or use the Condor project on your system(s). Not only is the source code only available on request, and only to those that the maintainers feel are 'suitable', but by default the program will report information about your installation back to the Condor servers on a regular basis. There is no mention of this during the installation procedure. To add to this, the license requires that any publication which includes data that has been analysed or processed using Condor must include a reference to the Condor project. Given that there are plenty of open source alternatives available, I don't recommend using this.

UPDATE: As of 6.9.5, Condor is now released under the Apache Licence, v. 2.0. Therefore, some of the above concerns have been addressed. In particular, the source code is now available, but only when you agree to fill in personal information. However, I am still unsure whether usage information is reported to the maintainers by default. Caveat Emptor.

For this guide, we will be installing Sun's N1 GridEngine 6. GridEngine is simple to install on Linux, requires only a minimum of installed libraries, is relatively easy to integrate with an existing DRBL setup, and provides a wide range of powerful tools to manage and use your cluster. There is also extensive documentation available online. Installing GridEngine will involve three setups:

1. Installing the qmaster on the master node
2. Installing the execution host on the master node
3. Configuring the nodes and GridEngine to work together

Installing qmaster

This step simply involves installing qmaster on the master node as per the online documentation. However, before starting the installation script, there are a few things you will need to do:

• Make sure the hostname of the master (or any of the nodes) doesn't also point to localhost or localhost.localdomain. SGE objects very strongly if it does. So,if your /etc/hosts file looks like this: 127.0.0.1 localhost mymaster 192.168.0.1 mymaster.mydomain.com mymaster then you will want to change it to: 127.0.0.1 localhost 192.168.0.1 mymaster.mydomain.com mymaster
• Create a file containing the hostnames of your DRBL nodes. This will be used during the GridEngine installation process, when you will need to supply a textfile which contains one hostname per line. This information is available from your DRBL installation at /etc/drbl/IP_HOSTS_TABLE. You can generate it using: create_drbl_hostlist.py /path/to/output/hostlist

During the qmaster installation process you can safely accept most of the defaults, but make sure you choose to:

• Install as root user You don't have to do this, but it simplifies the process of getting SGE to run on the DRBL nodes. Please note that in recommending this, I am presuming that your cluster network is private, and the nodes won't be accessible by non-privileged users (i.e. not you).
• Do opt to verify file permissions
• Select to use BerkleyDB, but without a spool server
• Use the ID range suggested in the manual: 20000-20100
• Accept to install startup scripts
• Accept to load a file which contains the hostnames of your nodes. Here you enter the full path to file you created before running the install script.
• Use normal scheduling

Once it's installed, source the SGE settings file in your .bashrc with the command: source sge-root/cell/common/settings.sh where sge-root is the SGE installation directory.

Install the execution host on the master server

We don't actually want to make the master node an execution node (although you may want to), as it's job is to manage and schedule jobs on the other nodes. However, as the nodes rely on the master node's filesystem for their applications, we will set up the execution daemons temporarily so that we may run them remotely on the nodes. Again, installation is as per the documentation, but there are some important choices to make:

• Specify a local spool dir: /var/tmp/spool
• Accept to install startup scripts

After the install is finished:

• Move /etc/init.d/sgeexecd to somewhere safe
• Run update-rc.d -n sgeexecd remove (because we don't want to start at boot time on the master node)
• Move sgeexecd script back into /etc/init.d (which will make it available to the nodes once we rerun the drblpush script)
• Add sgeexecd to /opt/drbl/conf/client-extra-service where /opt/drbl is your DRBL installation directory. This will make sure that the execution host daemon is started automatically when the nodes boots so they will be available to run jobs.
• On the master node, save the current execution host config as a template with: qconf -sconf hostname > node.conf.template where hostname is the hostname of the master. Open the resulting file with a text editor and if the first line is 'hostname:' then remove it (thanks Josh). The config includes the custom local spool dir /var/tmp/spool which we want to be set for each of our nodes, so that the spool is storted locally in ram, and not on the master. If all of the nodes saved their spool information to the master via NFS, it would place a greater strain on the master node and the network infrastructure. Storing the information locally means that network bandwidth is saved for booting nodes and transferring user data/scripts for execution on the cluster.
• Finally, run: create_sge_nodeconfig.py /path/to/node.conf.template This script will create a new GridEngine configuration for each of the DRBL nodes (as defined in /etc/drbl/IP_HOSTS_TABLE) based on the template we saved. Note: it is not strictly necessary to have a seperate configuration for each of the nodes, as the main purpose is to set the spool location to /var/tmp/spool. This is also possible by modifying the global configuration parameters of the GridEngine. However, I personally prefer having the ability to change the configuration of individual nodes if necessary without having to create a new configuration for them each time. As we use scripts to create and update the configurations, it doesn't require any effort to adopt either setup.

Configure the nodes

At the moment, a job queue ('all.q') has been created on the master, because we ran the execution host install script. For this cluster, we can either add all our nodes to that queue or create a seperate queue for each node. For this setup we will have a seperate job queue on all execution hosts. We now want to remove the all.q queue from the master, remove it as an execute node, and add queues to all of our DRBL hosts:

• Save the queue config for the master with: qconf -sq all.q > node.queue.template
• We'll now modify the queue config template so that we can easily recreate a new config for each node with a script. Open the queue conf template in your favourite text editor and make the following changes: qname %HOST%.q hostlist %HOST% slots 1 Note: if you have more than 1 processor on all your nodes then you can change the number to reflect this shell /bin/bash Set this to whatever shell you want to use for jobs
• Run create_sge_nodequeues.py /path/to/queue.template This will create a new queue on each node with one slot per processor, enabling jobs to be submitted and executed.
• Delete the all.q queue on the master as we won't be using it: qconf -dq all.q
• And now, remove the @allhosts group because we have made individual queues for each node, so we don't need it either: qconf -dhgrp @allhosts
• And last we remove the master as an execution node. It was set as such by running the execution host installation script, but we can undo that with: qconf -de master

If you now run qhost -q | more you should see a list of all your DRBL nodes and below each should be a queue definition. Running qstat -f should now list the various queues that were created, and show how many slots each of them have and how many slots are filled at the moment. Now, rerun drblpush -i or drblpush -c /etc/drbl/drblpush and restart one of your nodes. ssh into the node when it boots and confirm that sgeexecd is running with: ps ax | grep sge. If it isn't, check for startup messages in /tmp/execd*. Back on the master, rerun qhost -q | more. The node that you have rebooted should be listed with extra information than the others, such as free memory, number of processors, and load. You can now boot the rest of your nodes to create a diskless cluster with GridEngine and DRBL!

Setting up MPI/MPICH support on GridEngine

If you want to configure MPI/MPICH support, there are two steps which need to be completed. First, you must add a new parallel environment to GridEngine which sets up the mpi/mpich startup and shutdown scripts that are provided with SGE for use with MPI/MPICH jobs. Second, you must add your new parallel environment to each of your already existing job queues. For this example we will be setting up mpich as our parallel environment. If you want non-tight MPI support, just replace mpich with mpi throughout:

• Make a copy of the mpich.template in the mpi directory
• Fill in the information that is missing (marked with <>) in the template, such as number of slots (just put 999) and the SGE root dir (for me it's /opt/n1ge6)
• Now add the parallel environment with: qconf -Ap /path/to/modified/mpich.template You can check it exists with qconf -spl and qconf -sp mpich
• Next, modify the node.queue.template that we created in a previous step, and change the 'pe_list' from 'make' to 'make mpich'. This will enable parallel jobs to be run on the queue.
• Finally, update all our queues (one for each DRBL node) with: update_sge_nodequeues.py /path/to/modified/node.queue.template
• You can then check the updated queue definitions with: qconf -sq <queuename> where queuename is one of the queues on a DRBL node.

Using QMon

QMon is the graphical interface to the Sun GridEngine system. To run it, you need some extra libraries installed. Unfortunately, they haven't (and probably won't) have a release candidate for Debian, so you need to get it manually from the Debian servers:

• Download the package for your install here.
• Install with dpkg -i package-name.deb
• Type qmon to start it.

Permalink | Leave a comment  »