If there is anything that the recent GMail outages taught us, it’s that losing access to your email for a few hours should not mean that the end of the world has arrived. I think it also served as a reminder that if you rely 100% on the cloud for access to your data, then be prepared to be disconnected from it every once in a while. Fortunately, it’s quite easy to take advantage of free email services from the likes of Google while also making sure that should the worst happen, you can still access your data.

With more and more communication, both professional and personal, being conducted through email, it’s no wonder that people do get nervous when they suddenly lose control over several years worth of contacts and emails. This short article will show you how easy it is to back up your GMail automatically to a remote server which you can then access in the event that GMail goes down or you lose access to your account. We’ll be using nothing but free open-source software. If you already have a net-connected machine with Ubuntu installed then you’re set to go. Otherwise, consider recycling an old pentium-or-better-cpu machine with a network card—the processing demands are relatively low and the most important resource is storage space for all the emails and attachments. Although these instructions are for Ubuntu, they should also work on Debian or any other Debian-based distro.

To set this up you obviously need a GMail account, although any email account which can be accessed using the POP protocol can be backed up using the same method. We’ll be using the excellent Fetchmail to retrieve messages from the GMail account while leaving them untouched in your GMail inbox, Procmail to deliver and filter them into a local Maildir-format mailbox, and Dovecot to access the emails remotely, all on the latest release of Ubuntu, Hardy Heron (8.04 LTS).

1. Setup GMail

• Log in to your GMail account
• Click ‘Settings>Imap/Pop’ and enable POP access
• You can chose whether to only download messages which arrive from now on or to download all existing messages. The latter is useful if you have already been using your GMail account for a while and would like to back all existing messages to a secure alternative location.

2. Install packages

• Fire up a terminal window and run the following command:
sudo apt-get install dovecot-imapd procmail fetchmail mutt
• This should automatically download and install everything you need. The mutt package is optional and is a useful tool for checking if your IMAP server is working properly and that your emails are being downloaded (not to mention a nice command-line email client!).

3. Configure Fetchmail

• We need to define our GMail accounts in fetchmail’s configuration file. Open the file /etc/fetchmailrc for editing as root. The file won’t exist yet so we’ll have to create it.
• Copy & paste the following into the file:
set daemon 60
set syslog
set postmaster “localuser“
poll pop.gmail.com
protocol pop3
username “username@gmail.com” there with password “yourpassword” is localuser here
keep ssl
mda “/usr/bin/procmail -d localuser“
• Change localuser to your local username on the linux box and the email and password to match your gmail account. If you want, you can add as many other accounts as you want here. For each account, just copy the text starting from ‘poll…’ and change the login details.
• The first line tells fetchmail to check for new emails every 60 seconds. Edit to match your preferred update frequency.
• Edit /etc/default/fetchmail as root and change the line START_DAEMON=no to START_DAEMON=yes

4. Configure Procmail (~/.procmailrc)

• We’re using procmail to deliver email from fetchmail to a mailbox in Maildir format. This is necessary due to the lack of Maildir support in Fetchmail, which supports the mbox format. The main difference between the formats is that Maildir stores each email in a single file, making it very efficient when you have a lot of emails. In contrast, mbox format stores all emails in a single file.
• Create ~/.procmailrc and put the following text in it:
MAILDIR=$HOME/Mail/
DEFAULT=$MAILDIR
VERBOSE=on
LOGFILE=$MAILDIR/procmail.log
:0
$MAILDIR
• Set permissions on the file with: chmod 640 ~/.procmailrc
• This file simply tells procmail where to store the emails (in ~/Mail) and where to store it’s logs. Procmail is capable of a lot more than this, and it supports a powerful filtering language. You can specify rules in your ~/.procmailrc file to filter emails based on sender, subject, etc.

5. Configure Dovecot

• We’re almost there! We can retrieve emails from the GMail servers and deliver them locally. Now we need a way to access the email remotely. For this we’re using Dovecot, and we need to tweak the configure a little.
• As root, open up /etc/dovecot/dovecot.conf and update the following lines:
protocols = imaps
mail_location = maildir:~/Mail

6. Setup maildir

• We need to create the local mailbox with:
maildirmake.dovecot ~/Mail

7. Start services

Okay, now everything is set up we need to start up the services:

sudo /etc/init.d/fetchmail start
sudo /etc/init.d/dovecot start

8. Enjoy!

That’s it! Run tail -f /var/log/syslog to check fetchmail is receiving messages and also check ~/Mail/procmail.log to see that procmail is not throwing up any errors. The final thing you need to do is make sure you can access your email server remotely. Dovecot is running over SSL on port 993, so just configure your router to enable external access to that port. Then just configure Outlook , Thunderbird, etc. to access IMAP over SSL and you have full access to all your emails on your own server.

Note: if you chose to setup GMail POP access to allow all existing messages to be downloaded and you have lots of messages in your account, it will take a few runs of fetchmail to collect them all. Leave things running overnight and check back in the morning!

Optional step: Setup mutt

• Create the file ~/.muttrc and put the following in it:
set mbox_type=Maildir
set folder=”~/Mail”
set mask=”!^\\.[^.]“
set mbox=”~/Mail”
set record=”+.Sent”
set postponed=”+.Drafts”
set spoolfile=”~/Mail”
• Now just run mutt and read your email!

Related Links

• Set Up a Debian or Ubuntu Machine as a Maildrop
• Dovecot
• Fetchmail
• Procmail

• Deafness and the User Experience
  Interesting article about improving the experience of deaf website users in an informed way.

  My favourite quote:
  "We are not disabled and Deafness is not a disability; it’s the perception of many hearing (people) that we are disabled, and that is our disability."

• keyczar
  Easy-to-use encryption library with Java and Python bindings
• Best. Unboxing. Ever.

After playing Google App Engine for a few days, I’ve realised the power that even the basic framework provided by Google, webapp, provides. Working with Python adds a level of flexibility that I never had when working with PHP many years ago (although this was before OO had really infected PHP).

While working on my first real application for app engine I soon came across the problem of creating sensible key names for objects. If you aren’t familiar with the data object models in app engine, each object has a unique key, generated automatically when the object is created, and a key name (property: key_name) which can be specified by the user. The key name can be very useful if you need object identifiers which are human-readable, such as when you use them as part of your apps URLs.

All data objects are defined as Python classes which subclass the db.Model class, provided by Google. A simple example, from the app I’m working on, is an object which holds information about an author:

class Author(db.Model):
  first_name = db.StringProperty(required=True)
  middle_name = db.StringProperty()
  last_name = db.StringProperty(required=True)

Creating a new author is a simple matter of calling the class and passing property values either as keyword arguments or by accessing the property through the new instance:

a = Author(first_name='John',
           last_name='Smith')
a.middle_name = 'Thomas'

We can specify the optional key_name property when we create the object:

a = Author(first_name='Alan',
           middle_name='Simon',
           last_name='Jones',
           key_name='alansimonjones')

However, it’s more than likely that if we want to use the key_name property for a class of data objects, we will generate it procedurally based on the initial property values of the object. Above, we create the key name using the authors names. You could create a function which wraps the creation of the author object which does this for you:

def newAuthor(first_name, last_name, middle_name = ''):
  parts = [first_name, middle_name, last_name]

  # Define the key name from the authors names.
  # We split() and join each name to remove any spaces
  key_name = ''.join([''.join(x.split()) for x in parts]).lower()
  return Author(first_name=first_name,
             last_name=last_name,
             middle_name=middle_name,
             key_name=key_name)

but this is quite a lot of boilerplate code to solve a simple problem. Moreover, you need to make sure everyone working on the code is aware that this function must be used to create new author objects, rather than by calling the object class directly.

A more elegant solution is to encapsulate the creation of the key_name property in the class definition itself. This is bread and butter stuff for any Pythonista, and is handled by defining a constructor for our author object, using it to define our key name and then to manually call the constructor of the db.Model class. The last step is necessary as once we define our own constructor, the constructor of the super class is not called automatically. To add to the things we need to handle, it’s important to know that the db.Model class uses dynamic positional and keyword arguments. If that makes no sense to you, don’t worry, it’s simpler than you may think.

Python provides an additional way of specifying standard function arguments using the *variable syntax, and keyword arguments using the **variable syntax. If used in the definition of a function, the *variable will be a tuple of positional arguments, and **variable will be a dictionary relating keywords to variables. Therefore, we must make sure our new constructor has the same parameters as db.Model to ensure 100% compatibility.

Here is the updated author class, with the new __init__() function defined. It has been slightly altered because we access the keyword values using the kw dictionary and we need to check whether each of the properties have been specified or not by examining kw.keys():

class Author(db.Model):
  first_name = db.StringProperty(required=True)
  middle_name = db.StringProperty()
  last_name = db.StringProperty(required=True)

  def __init__(self, *args, **kw):
    parts = ['first_name', 'middle_name', 'last_name']

    # Define the key name from the authors names.
    # We split() and join each name to remove any spaces
    key_name = ''.join([''.join(kw[x].split()) for x in parts if x in kw.keys()]).lower()

    # Insert the new key_name property into the keyword arguments
    kw['key_name'] = key_name

    # Call the constructor of the base class
    db.Model.__init__(self, *args, **kw)

We can now create new author objects and the key names will be generated automatically and will follow our specified format:

a = Author(first_name='John', last_name='Smith')
# Will print 'johnsmith'
print a.key().name()

a = Author(first_name='John', middle_name='Alan Mike', last_name='Smith')
# Will print 'johnalanmikesmith'
print a.key().name()

Being able to alter the properties of an object or preconfigure core properties at the time of instantiation can be useful in all sorts of situations, but this particular example is one which I can see myself using a lot.

• JungleDisk
  Extremely cheap online storage using Amazon’s S3 service.
• Python main() functions
  Some advice from Guido about handling main() functions and input parameters.
• Jottit
  Make a website by typing into a box. No, really - that’s it.
• Things to do when you’re a bored developer
  Some ideas on how to be productive in what would otherwise be dead time.

Anyone who needs to access a unix/linux/osx machine from windows via ssh will be familiar with PuTTY, the free ssh client. PuTTYGen, available to download here, is a handy utility for creating RSA/DSA public/private keys for authentication. If you have generated your RSA keys using PuTTYGen, and would like to use them with OpenSSH on Ubuntu, you just need to follow a few simple steps:

• Install putty with: sudo apt-get install putty
• Create the public key file:
  puttygen /path/to/puttykey.ppk -L > ~/.ssh/id_rsa.pub
• Create the private key file:
  puttygen /path/to/puttykey.ppk -O private-openssh -o ~/.ssh/id_rsa

You should now be able to log into an SSH server using your private key. To install your public key simply copy from ~/.ssh/id_rsa.pub on your local machine to ~/.ssh/authorized_keys on the remote server.

If you use Matlab for audio research / development, you’ll be familiar with the pretty poor native support for audio playback. As part of my research, I wanted to play multichannel (>2) files at a variety of sampling rates and bit depths. In addition, I also wanted to use a low-latency audio layer, such as ASIO, to take direct  advantange of my audio hardware.

So, with a bit of coding I managed to put together a Matlab extension which allows for the playback of multichannel sound files using any soundcard with an ASIO driver. The plugin uses ASIO, PortAudio, and libsndfile. Samples are read from the audio file as needed, so playback start time is not dependant on the length of the file - useful in time-critical applications. Memory usage is also low because the whole file doesn’t need to be loaded into memory, only enough to fill the audio buffer.

You can download it over at the Matlab file exchange and the source is included under the GPL. Comments and suggestions welcome!

• Parallel Python
  Module for creating parallel Python scripts using either SMP or on a cluster. Much simpler than getting an MPI setup working…

It’s quite common in University departments for roomfuls of very capable PCs to sit idle from 5pm until 8am everyday and all weekend. Thankfully, it’s possible to turn such a wasted resource into a powerful cluster of Linux machines using freely available open source software. In fact, even a group of machines in an office, or at home, can be transformed into a similar state with the minimum of effort. By booting the PCs (or nodes) over the network, no changes are made to the current setup of the machines, so reverting back to their normal state is as simple as rebooting. This guide will bring you through the process of creating a High-Performance Computing (HPC) cluster using the latest version of the Debian GNU/Linux operating system, Diskless Remote Boot in Linux (DRBL), and Sun’s Open Source N1 Grid Engine 6.

Requirements

Creating a Linux cluster doesn’t require a huge amount of resources, either in hardware or in time, but there is a bare minimum of hardware that you need to get started:

1. A moderately powerful (at least a Pentium 4 with 1Gb RAM) machine to act as the master node. This really needs to be a dedicated machine, as this will allow users to submit jobs whether the cluster is currently running or not.
2. The master node obviously needs a network card, but if you want to give the master or the nodes internet access, or are planning to have more than about 40 nodes, you will need two or three network cards. Even high-end Gigabit ethernet cards are cheap these days, and the way your nodes will communicate with the master can make or break a successful cluster.
3. Network infrastructure to connect all of the nodes together. A 100Mbit network is the absolute minimum, particularly if you will have more than 10 nodes. Gigabit ethernet is recommended.
4. One or more node PCs which support PXE booting (network booting). If the bios doesn’t have an option to boot from the network card, or you don’t see any messages about netbooting when you start the PCs, then you need to install a card which supports PXE. These nodes don’t need to have a hard drive, but if they do (and you have some spare space) then it can be used for a swap partition (virtual memory in Windows-speak). It helps if the nodes contain similar hardware (such as is found in most offices or University computer rooms), as it reduces the amount of configuration required (almost nobody likes to rebuild the Linux kernel 20 times).

Download the script files

You will need to download some scripts which help setup GridEngine with DRBL (Python is required to run them, but is installed by default in Debian)

Install Debian

For this project, I used the latest release of Debian, ‘Etch’. You can download it here. I’m not going to go through the installation process, as you can find plenty of excellent documentation over at the Debian website. I suggest heading straight for the installation manual.

Thanks to the excellent installation scripts for Diskless Remote Boot Linux (DRBL), you can choose whatever installation type you want when installing Debian. For example, if you want to use graphical configuration utilities to set up your system, the select a desktop install, which will include the X window system. If you only plan to access the master over SSH then choose a lightweight console-only install.

Install DRBL

Again, I won’t duplicate the excellent and thorough installation guide available from the DRBL website over here. However, for the purposes of this guide it is important that you choose the Single System Image (SSI) mode when installing. This means that all the nodes download the same system image over the network which provides the minimum filesystem to boot Linux, and the rest of the system, including /home, /usr and /opt are NFS mounted (accessed over the network) on the master node. Therefore, any software and user files on the master node are available to all of the cluster nodes. To deploy new software across the cluster, the most dramatic step you need to take is to reboot the nodes, something that DRBL will even do for you.

A final point here is to take extra care when configuring the network setup of your nodes. During installation, DRBL will assume that any network card configured with a private address (e.g. 192.168.*) will contain nodes. If you will have over 40 machines on the network, consider splitting them up into two groups, and install two network cards in the machine with different subnets, e.g. 192.168.1.1/255.255.255.0 and 192.168.2.1/255.255.255.0. Having a multi-homed master node will increase the amount of data that you can push out to nodes.

Install Grid Engine

Before you go any further, make sure tht DRBL is installed, and that you can boot your nodes over the network from your master node. All following steps assume a working DRBL Single System Image setup with one or more nodes.

Now that you have a network of machines, the next step is to install software which will manage the distribution and scheduling of jobs on the cluster. There are several alternatives to choose from, including Torque, SLURM, and Condor.

Note: I strongly suggest that you do not install or use the Condor project on your system(s). Not only is the source code only available on request, and only to those that the maintainers feel are ’suitable’, but by default the program will report information about your installation back to the Condor servers on a regular basis. There is no mention of this during the installation procedure. To add to this, the license requires that any publication which includes data that has been analysed or processed using Condor must include a reference to the Condor project. Given that there are plenty of open source alternatives available, I don’t recommend using this.

UPDATE: As of 6.9.5, Condor is now released under the Apache Licence, v. 2.0. Therefore, some of the above concerns have been addressed. In particular, the source code is now available, but only when you agree to fill in personal information. However, I am still unsure whether usage information is reported to the maintainers by default. Caveat Emptor.

For this guide, we will be installing Sun’s N1 GridEngine 6. GridEngine is simple to install on Linux, requires only a minimum of installed libraries, is relatively easy to integrate with an existing DRBL setup, and provides a wide range of powerful tools to manage and use your cluster. There is also extensive documentation available online.

Installing GridEngine will involve three setups:

1. Installing the qmaster on the master node
2. Installing the execution host on the master node
3. Configuring the nodes and GridEngine to work together

Installing qmaster

This step simply involves installing qmaster on the master node as per the online documentation. However, before starting the installation script, there are a few things you will need to do:

• Make sure the hostname of the master (or any of the nodes) doesn’t also point to localhost or localhost.localdomain. SGE objects very strongly if it does. So,if your /etc/hosts file looks like this:
  127.0.0.1 localhost mymaster
192.168.0.1 mymaster.mydomain.com mymaster
  then you will want to change it to:
127.0.0.1 localhost
192.168.0.1 mymaster.mydomain.com mymaster
• Create a file containing the hostnames of your DRBL nodes. This will be used during the GridEngine installation process. Need to create a textfile which contains one hostname per line. This information is available from your DRBL installation at /etc/drbl/IP_HOSTS_TABLE. You can generate it using:
  create_drbl_hostlist.py /path/to/output/hostlist

During the qmaster installation process you can safely accept most of the defaults, but make sure you choose to:

• Install as root user
  You don’t have to do this, but it simplifies the process of getting SGE to run on the DRBL nodes. Please note that in recommending this, I am presuming that your cluster network is private, and the nodes won’t be accessible by non-privileged users (i.e. not you).
• Do opt to verify file permissions
• Select to use BerkleyDB, but without a spool server
• Use the ID range suggested in the manual: 20000-20100
• Accept to install startup scripts
• Accept to load a file which contains the hostnames of your nodes.
  Here you enter the full path to file you created before running the install script.
• Use normal scheduling

Once it’s installed, source the SGE settings file in your .bashrc with the command:
source sge-root/cell/common/settings.sh
where sge-root is the SGE installation directory.

Install the execution host on the master server

We don’t actually want to make the master node an execution node (although you may want to), as it’s job is to manage and schedule jobs on the other nodes. However, as the nodes rely on the master node’s filesystem for their applications, we will set up the execution daemons temporarily so that we may run them remotely on the nodes. Again, installation is as per the documentation, but there are some important choices to make:

• Specify a local spool dir: /var/tmp/spool
• Accept to install startup scripts

After the install is finished:

• Move /etc/init.d/sgeexecd to somewhere safe
• Run update-rc.d -n sgeexecd remove (because we don’t want to start at boot time on the master node)
• Move sgeexecd script back into /etc/init.d (which will make it available to the nodes once we rerun the drblpush script)
• Add sgeexecd to /opt/drbl/conf/client-extra-service
  where /opt/drbl is your DRBL installation directory. This will make sure that the execution host daemon is started automatically when the nodes boots so they will be available to run jobs.
• On the master node, save the current execution host config as a template with:
  qconf -sconf hostname > node.conf.template
  where hostname is the hostname of the master. The config includes the custom local spool dir /var/tmp/spool which we want to be set for each of our nodes, so that the spool is storted locally in ram, and not on the master. If all of the nodes saved their spool information to the master via NFS, it would place a greater strain on the master node and the network infrastructure. Storing the information locally means that network bandwidth is saved for booting nodes and transferring user data/scripts for execution on the cluster.
• Finally, run:
  create_sge_nodeconfig.py /path/to/node.conf.template
  This script will create a new GridEngine configuration for each of the DRBL nodes (as defined in /etc/drbl/IP_HOSTS_TABLE) based on the template we saved.Note: it is not strictly necessary to have a seperate configuration for each of the nodes, as the main purpose is to set the spool location to /var/tmp/spool. This is also possible by modifying the global configuration parameters of the GridEngine. However, I personally prefer having the ability to change the configuration of individual nodes if necessary without having to create a new configuration for them each time. As we use scripts to create and update the configurations, it doesn’t require any effort to adopt either setup.

Configure the nodes

At the moment, a job queue (’all.q’) has been created on the master, because we ran the execution host install script. For this cluster, we can either add all our nodes to that queue or create a seperate queue for each node. For this setup we will have a seperate job queue on all execution hosts. We now want to remove the all.q queue from the master, remove it as an execute node, and add queues to all of our DRBL hosts:

• Save the queue config for the master with:
  qconf -sq all.q > node.queue.template
• We’ll now modify the queue config template so that we can
  easily recreate a new config for each node with a script.
  Open the queue conf template in your favourite text editor
  and make the following changes:
  qname %HOST%.q
hostlist %HOST%
slots 1
  Note: if you have more than 1 processor on all your nodes
  then you can change the number to reflect this
  shell /bin/bash
  Set this to whatever shell you want to use for jobs
• Run create_sge_nodequeues.py /path/to/queue.template
  This will create a new queue on each node with one slot per
  processor, enabling jobs to be submitted and executed.
• Delete the all.q queue on the master as we won’t be
  using it:
  qconf -dq all.q
• And now, remove the @allhosts group because we have made
  individual queues for each node, so we don’t need it either:
  qconf -dhgrp @allhosts
• And last we remove the master as an execution node. It was
  set as such by running the execution host installation
  script, but we can undo that with:
  qconf -de master

If you now run qhost -q | more you should see a list of all your DRBL
nodes and below each should be a queue definition. Running `qstat -f
should now list the various queues that were created, and show how many
slots each of them have and how many slots are filled at the moment.

Now, rerun drblpush -i or drblpush -c /etc/drbl/drblpush and restart one of
your nodes. ssh into the node when it boots and confirm that sgeexecd is
running with: ps ax | grep sge. If it isn’t, check for startup messages in
/tmp/execd*.

Back on the master, rerun qhost -q | more. The node that you have
rebooted should be listed with extra information than the others,
such as free memory, number of processors, and load. You can now boot
the rest of your nodes to create a diskless cluster with GridEngine and
DRBL!

Setting up MPI/MPICH support on GridEngine

If you want to configure MPI/MPICH support, there are two steps
which need to be completed. First, you must add a new parallel
environment to GridEngine which sets up the mpi/mpich startup
and shutdown scripts that are provided with SGE for use with
MPI/MPICH jobs. Second, you must add your new parallel environment
to each of your already existing job queues. For this example we
will be setting up mpich as our parallel environment. If you want
non-tight MPI support, just replace mpich with mpi throughout:

• Make a copy of the mpich.template in the mpi directory
• Fill in the information that is missing (marked with <>) in the template, such as number of slots (just put 999) and the SGE root dir (for me it’s /opt/n1ge6)
• Now add the parallel environment with:
qconf -Ap /path/to/modified/mpich.template
  You can check it exists with qconf -spl and qconf -sp mpich
• Next, modify the node.queue.template that we created in a previous step, and change the ‘pe_list’ from ‘make’ to ‘make mpich’. This will enable parallel jobs to be run on the queue.
• Finally, update all our queues (one for each DRBL node) with:
update_sge_nodequeues.py /path/to/modified/node.queue.template
• You can then check the updated queue definitions with:
qconf -sq <queuename>
  where queuename is one of the queues on a DRBL node.

Using QMon

QMon is the graphical interface to the Sun GridEngine system. To
run it, you need some extra libraries installed. Unfortunately,
they haven’t (and probably won’t) have a release candidate for
Debian, so you need to get it manually from the Debian servers:

• Download the package for your install here.
• Install with dpkg -i package-name.deb
• Type qmon to start it.

• Mistakes in Experimental Design and Interpretation
  An excellent article discussing the common mistakes and pitfalls in the development, analysis, and interpretation of scientific experiments
• Blogging Zelda
  A blog detailing a player’s expierences as he completes each title in the Zelda series

• You Can Own an Integer Too - Get Yours Here
  Get your free 128-bit integers in the handy and easy to read HEX notation. I prefer 13,256,278,887,989,457,651,018,865,901,401,704,640 myself.
• Flying LOW!
  Amazing photographs of landings at St. Maarten airport in the Netherlands Antilles.