One of the projects that I am involved in has a requirement of importing all the historical emails from a number of Gmail accounts into another system. It’s not the most challenging of tasks, but since I spent a bit of time on it, I figured I should blog it here too, just in case a similar need will arise in the future.<\/p>\n
In my particular case, I need two different solutions. One for exporting all of the messages from all folders of all Gmail accounts in question (Gmail for Work<\/a>). And the other is for exporting only the messages from the “Sent Mail<\/em>” folder, which were sent on specific dates.<\/p>\n The solution that I derived is based on the classic tools for this purpose – fetchmail<\/a> and procmail<\/a>. Fetchmail is awesome at fetching emails using all kinds of protocols. Procmail is amazing at sorting, filtering, and otherwise processing the email messages.<\/p>\n So, here we go. First of all, we need to tell fetchmail where to get the messages from. I didn’t want to create to separate configurations for each of my tasks, so I left only the options common between them in the configuration file, and the rest I will be passing as command line arguments, depending on scenario.<\/p>\n Note that I’ve been running these tests from a dedicated environment, where I only had the root<\/em> user. You don’t have to run it as root – it’ll work as any other just fine. Also, keep in mind that I used “\/root\/fetchmail-test\/<\/em>” folder for my test runs. You might need to adjust the paths if you have it any different.<\/p>\n Here’s my fetchmail.rc<\/em> file, which I used to test a single mailbox. A new “poll” section will go into this file later, for each mailbox that I’ll need to export.<\/p>\n If you are not root, you might need to adjust the second line, replacing “root” with your username. Also, for testing purposes, you can use “fetchlimit 1<\/em>” instead of “fetchall<\/em>“.<\/p>\n Now, we need two configuration files for procmail. The first one is super simple – I’ll use this for simply pushing all downloaded messages into a single giant mbox file. Here’s the procmail-all.rc<\/em>:<\/p>\n As you can see, it only defines the verbosity level and the default mailbox. The second configuration file is a bit more complicated. I’ll use it for the sent items only. The sent items folder limit will be done with fetchmail. But I want to do further is disregard all messages, which were not sent on a specific date. Here is my procmail-sent.rc<\/em>:<\/p>\n Again, we have the verbosity level and the default mailbox to save messages to. Since I want to disregard them unless they match a certain condition, I specify \/dev\/null<\/em>. Then, I specify my condition, which is simply a bunch of regular expressions for the Date header. Usually, Date header is a not very reliable as different MUAs (Mail User Agents) use different formats, time zones, etc. In this particular case test results seemed consistent (maybe Gmail fixes the header), and I didn’t have any other more reliable criteria to use.<\/p>\n As you can see, I use a very basic condition for date matching. So, if the Date header matches either “28 Jul 2016<\/em>” or “27 Jul 2016<\/em>“, the message is saved in the mbox file, rather than being thrown into the default mailbox.<\/p>\n Now, all I need is a way to tie fetchmail and procmail together, as well as provide some additional options. For that I created the two one-liner shell scripts, just so that I won’t need to figure out the command line arguments if I look at this whole thing six month later.<\/p>\n Here is the check-all.sh script (multi-line for readability):<\/p>\n and here is the check-sent.sh script (multi-line for readability):<\/p>\n If you run either one of these scripts, you’ll see the output similar to this:<\/p>\n Here are a few resources that you might find helpful:<\/p>\n\npoll imap.gmail.com proto imap:\nusername "someuser@gmail.com" is root here\npassword "somepass"\nfetchall\nkeep\nssl\n<\/pre>\n
\nVERBOSE=0\nDEFAULT=\/root\/fetchmail-test\/fetchmail.all.mbox\n<\/pre>\n
\nVERBOSE=0\nDEFAULT=\/dev\/null\n:0\n* ^Date: .*28 Jul 2016.*|\\\n^Date: .*27 Jul 2016.*\n\/root\/fetchmail-test\/fetchmail.sent.mbox\n<\/pre>\n
\n#!\/bin\/bash\nfetchmail -f fetchmail.rc \\\n-r "[Gmail]\/All Mail" \\\n--mda "procmail \/root\/fetchmail-test\/procmail-all.rc"\n<\/pre>\n
\n#!\/bin\/bash\nfetchmail -f fetchmail.rc \\\n-r "[Gmail]\/Sent Mail" \\\n--mda "procmail \/root\/fetchmail-test\/procmail-sent.rc"\n<\/pre>\n
\n$ .\/check-all.sh\nfetchmail: WARNING: Running as root is discouraged.\n410 messages for someuser@gmail.comat imap.gmail.com (folder [Gmail]\/All Mail).\nreading message someuser@gmail.com@gmail-imap.l.google.com:1 of 410 (446 header octets) (222 body octets) not flushed\nreading message someuser@gmail.com@gmail-imap.l.google.com:2 of 410 (869 header octets) (230 body octets) not flushed\nreading message someuser@gmail.com@gmail-imap.l.google.com:3 of 410 (865 header octets) (230 body octets) not flushed\n...\n<\/pre>\n