Today I had some free time on my hands, so I tried something that I was planning to do for a long time – properly move away from mbox to Maildir for email storage.
It took me a bit more time than I thought it will, but I have done everything properly with migration of all messages and archives (about 50K emails) and local IMAP server. I have done it on my office workstation, where I am the only user, but the procedure is very scalable and will work fine on any number of users. I am planning to repeat it on the home server which is used by a number of people.
Introductory ‘Why?’
My email handling system was already rather complex. It took me some time to arrive to this setup, but I feel that it is the closest solution to solving all my needs. Here is an overview on how it works:
- fetchmail retrieves mail from our corporate mail server and passes it on to Exim for local delivery
- Exim performs basic SPAM filtering and delivers valid messages to procmail
- procmail performs an extensive filtering procedure with proper SPAM filtering via SpamAssassin and delivers messages to appropriate mail folders
- local mail folders are accessed by me with Mutt
- replies are composed in Vim
- outgoing messages are passed back to locally running Exim
The biggest problem that I had until now is that I didn’t have any problem. :) Being a system administrator reviewing and testing new software is a part of my job. That software often includes different email programs. The mess comes when each and every other new email client tries to build an index of all my mail folders and then leaves it there. I could, of course, test programs on a separate setup, but I think that the best way to see how something works is to actually do some work with it.
So, mbox folders had to go. But I didn’t want to substute the convinience of the local mail by clumsiness of IMAP. Maildir provides an excellent compromiss – it can be used as a local folder and IMAP server can work on top of it.
Choosing IMAP server
My office workstation runs Fedora Core 2. This distribution ships the cyrus-imapd for all your IMAP server needs. Cyrus though uses it’s own storage format incompatible with Maildir, so I had to pick something else.
A quick look at Dag’s RPMs suggested dovecot. It was the first time I’ve heard of it, so I decided to ‘apt-get install’ it. Unfortunately, it didn’t want to work for some reason. I spent few minutes investigating quiet crashes, but then decided to go for something “completely different”.
The next thing which was in mind was Courier IMAP. I’ve had some experience with it a couple of years ago, so I decided to try it out again.
Installation and configuration
None of the sites in my active memory were providing RPMs of Courier IMAP for Fedora Core 2, so I decided to build it from source. Luckily, the source comes with a ready made RPM spec in it, so the only thing I needed to do to have an RPM was:
rpmbuild -ta courier-imap-3.0.6-xxxx.gz
While it was compiling, I have found a fast and easy guide for configuration here.
After compilation and installation was done, I followed the instructions and had IMAP server up and running in no time.
The link above also mentions configuration changes you will need to make in Exim config file so that it would start delivering into those Maildir folders. Those are just a couple of lines.
Moving messages
There are a number of ways to move messages from mbox folders to Maildir ones. I’ve tried a couple of scripts until I found this perl script that actually works as expected. I used absolute paths to specify both mbox file and Maildir folder to convert though.
Fixing up all the rest
There are few minor things left to fix. All of them are easy, but without them you will not feel all the comfort. In my case I had to fix only procmail and mutt. If you have any webmail software or anything like that, you’ll have more to do. Here is a useful page for ya.
The essence is that instead of using mbox files in procmail configuration file like this:
:0: * From:.*me@here.com.* spam
you will have to use Maildir directories, which by the way start from dot (‘.’) in courier-imap case like this:
:0 * From:.*me@here.com.* .spam/
Note that the locking column is gone too. :)
Mutt configuration file also needs a couple of changes which are simple modifications of paths to the mail directories. Nothing else.
Result
Now I have “the best of both worlds” as they call it. I can use procmail and other tools processing local mail storages, and at the same time I can use any number of fully synchronized mail clients through IMAP. I have a sum of all the nice features of all mail clients. If one of them has better filtering facilities, I can use it to filter messages around. Then I can use another client which has best displaying facilities. And so on. Now I can fully interchange them without losing any comfort or messages getting saved in strange places. The trick is to set ‘drafts’ (or ‘postpone’), ‘sent’ (or ‘outbox’), and ‘trash’ (or ‘deleted’) to the IMAP folders too.
I also have to mention the speed improvements that come with this setup. The way I work is that I never keep a mail client open unless I am reading email. Opening huge mbox files takes time. Not closing a mail client leaves me with a number of unfinished messages all over the place. :) With Maildir and IMAP access mail clients construct an index file which they use for speedy access later on. With every launch of mail client I now save few seconds of my time. Good.