jcm's blog

So, I spent a while trying to figure out how I’m going to shutdown all these machines I’ve got at home. I came up with the idea of getting some more colo to handle stuff I don’t really want based at home in the first place – well, not really, it’s more hassle than it’s worth to worry about unreliable [A]DSL connections and the like.

I prefer my email to just work. So, I decided everything needed replication. Not just in the way I’ve copied things previously – now I want live copies of mail that are synchronized, right down to the meta data for which mails I’ve read/deleted or otherwise tagged/flagged. Take a look at the MX for jonmasters.org now:

jonmasters.org.         9601    IN      MX      10 mx1.jonmasters.org.
jonmasters.org.         9601    IN      MX      20 mx2.jonmasters.org.
jonmasters.org.         9601    IN      MX      30 mx3.jonmasters.org.
jonmasters.org.         9601    IN      MX      40 mx4.jonmasters.org.
jonmasters.org.         9601    IN      MX      100 mx5.jonmasters.org.

I setup a couple more colo boxen (actually low cost VM) that can see each other over my private VPN. These are mx1 (fremont.jonmasters.org) and mx2 (london.jonmasters.org). They run exim4 and both think they’re delivering locally all of my mail. Fremont delivers into /fremont/mail/jcm/Maildir, while London delivers into /london/mail/jcm/Maildir. Since fremont.jonmasters.org is the top priority MX, most of my mail will end up there, some (mostly SPAM) will go to the secondaries. Both fremont.jonmasters.org and london.jonmasters.org export /fremont and /london over NFS over the VPN so they can see each other’s mail.

I wanted a distributed filesystem (thanks to my friends at work for suggestions) but unfortunately cannot get my colo providers to do that just yet (they’re busy moving to Xen anyway – then I can just take the matter into my own hands) so I have to have this (growing) NFS hack for the moment. I can’t just deliver mail into both directories on both machines, because that won’t help when the two are out of sync. Instead, I have looked at a variety of Maildir syncing software with a view to syncing the two machines once per minute.

First off, I looked at offlineimap. Mostly because other people seem to like it. It sucks. Not only is it badly documented, but it doesn’t work in the way you would expect, errors out if it’s not happy and likes to create lots of duplicates of mail/waste space. I won’t use it again unless it’s for something that’s a really simple syncing-mail-with-laptop scenario. Oh well.

Next, I looked at maildirsync. This is a simple looking utility, but it actually works. Through a wrapper script, I now have the machines sync the maildirs of those users for which I want this replication enabled. Each user has pretty typical procmail filters and the like, thanks to some macros abstracting the difference between hosts.

The net result from this work is that I now have several places with live copies of my mail, synced together and make it available using IMAP, Squirrelmail and through other media. Eventually, I’ll increase the number of hosts involved so that I can handle multiple failures on different continents and still have mail I never read nor reply to

Jon.

This entry was posted on Wednesday, August 30th, 2006 at 10:32 pm and is filed under General. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.