Tim's Blog

Information, Technology, Security, and other stuff.

Email Archiving with iRedMail

Published 2015-07-22

It's been a rocky journey for me to find an easy way to archive my email in a private manner. By private, I mean on a server that I control and hosted at home. I'm not that keen on Google trawling through years and years of my email, and I can't trust that they'll maintain the integrity of it (I'm my number one customer, instead of one in a few hundred million for Google).

I started with storing everything in PST files (I'm still an avid Outlook user) with a new file each financial year and emails grouped into folders of their type or topic (e.g. Web Hosting for everything pertaining to that, PayPal receipts, etc.). This approach started to concern me that I'd be locked into only using Outlook forever, and became difficult to search on the odd occasion that I needed to.

So then I moved to extracting my email with Emailchemy (a great piece of software) into .eml files. EML files are great, because they're plaintext files (with attachments as Base64) and I was able to use Windows Explorer to search through them. But it started to suffer the exact same problems as the PST approach, especially if the directory isn't indexed.

Next I moved back to PSTs, but I merged everything back into a single PST file and just put them in folders by year (2015, 2014, etc). This became the lowest-friction solution, but I got my concerns back of the tie-in to Outlook, plus the risks of the PST getting corrupted (easy to occur if you put it on a network share), and ways to effectively backup the PSTs on a regular basis.

I would always read that the "easiest" way to store old email is in an IMAP server, and the most favoured approach for an IMAP server is a Postfix+Dovecot arrangement. I have to confess I tried, and failed, to get a dovecot server running the way I wanted. I got it up and running (after loads of modifications to config files in Dovecot and Postfix), and was able to add it as an account in Outlook, but I couldn't add IMAP folders on the client-side, only on the server-side. And it just plain confused me trying to add folders into the Maildir. Email servers really aren't my specialty.

I dug around more and found YunoHost, a really nice all-in-one package for self-hosting email and other applications. I really had trouble though installing it (perhaps due to running in Hyper-V, I didn't troubleshoot), and the instance that I did get up and running was kept throwing errors when I tried to install apps, so I decided to keep on looking for other products.

I ran into iRedMail a few times, but their site was a little off-putting to me (the sales stuff is always a bit of a deterrent to me), but I was really running out of options at this point. And there was no way in hell I was going to install Exchange at home, as I simply don't have enough RAM to spare for it. iRedMail has turned out to be amazingly easy, and exactly what I was looking for in a self-hosted Mail server. Installation effectively involves just running a bash script, and everything is taken care of. Even firewall configuration and enabling the services, which was a truly pleasant surprise. And I get IMAP folder creation straight from the Roundcube GUI.

Installing iRedMail on CentOS 7

So here's all it took to get iRedMail up and running correctly in my testbed:

Disable SELinux

This was in a test environment, so it was just simpler to disable SELinux and worry about it later.

# ssh to the server
# elevate
sudo su
 
# disable selinux
sed -i 's/enforcing/disabled/g' /etc/selinux/config
reboot

Install iRedMail

# ssh to the server
# elevate
sudo su
 
# update
yum update -y
 
# install prerequisites
yum install wget bzip2 -y
 
# download and unpack iRedMail
wget https://bitbucket.org/zhb/iredmail/downloads/iRedMail-0.9.2.tar.bz2
tar -xvf iRedMail-0.9.2.tar.bz2
cd iRedMail-0.9.2
sh iRedMail.sh

It's as simple as that. bzip2 needs to be installed, otherwise the tarball won't extract. The bash script will run you through the installation process, prompting you to provide passwords for the administrative accounts. I also picked MariaDB/MySQL as the database, as it's what I'm most comfortable with at the moment. Post installation involves authenticating to the administration site (https://sitename/iredadmin/) and creating the necessary users.

The iRedMail installation script only works well if you run it on a vanilla, minimal install server. If you've already configured other things like the firewall (perhaps by defining the default zone), then the firewall rules in the script will not be applied. It's much simpler to let the installer run on a blank machine, and then make the modifications you want.

#New Archival Strategy So now with an IMAP server, I can renew my archival processes. Instead of storing everything as PST files, I can just dump the emails into the respective IMAP folders. To backup my email, I run a daily tarball (with encryption) on the server and store it somewhere with redundancy. Should I wish to move the email to a different location in the future, it's just a matter of moving the email from one IMAP server to another, so I should be future-proof.

####Extra: Installing Postfix & Dovecot on CentOS 7

I might as well also share my testing with installing dovecot, in case you're wondering what I did. I mostly followed this guide, with some tweaks (like adding "<" to the cert path in the dovecot config). It's also written to just use echo to build the config files, so it's a little difficult to read but almost entirely scriptable (except for the mysql_secure_installation).

# elevate
sudo su
 
# disable selinux
sed -i 's/enforcing/disabled/g' /etc/selinux/config
reboot
 
# elevate
sudo su
 
# update 
yum update -y
 
# disable dovecot updates from the main repo
sed -i 's/name=CentOS-$releasever - Base/name=CentOS-#releasever - Base\nexclude=postfix/g' /etc/yum.repos.d/CentOS-Base.repo
sed -i 's/name=CentOS-$releasever - Updates/name=CentOS-#releasever - Updates\nexclude=postfix/g' /etc/yum.repos.d/CentOS-Base.repo
 
# install software
yum --enablerepo=centosplus install postfix -y
yum install dovecot mariadb-server dovecot-mysql -y
 
# enable mysql
systemctl enable mariadb.service
systemctl start mariadb.service
 
# configure mysql
mysql_secure_installation
 
# add new database for the mail server
mysql -u root -p
 
CREATE DATABASE mail;
USE mail;
GRANT SELECT, INSERT, UPDATE, DELETE ON mail.* TO 'mail_admin'@'localhost' IDENTIFIED BY 'Password';
GRANT SELECT, INSERT, UPDATE, DELETE ON mail.* TO 'mail_admin'@'localhost.localdomain' IDENTIFIED BY 'Password';
FLUSH PRIVILEGES;
CREATE TABLE domains (domain varchar(50) NOT NULL, PRIMARY KEY (domain) );
CREATE TABLE forwardings (source varchar(80) NOT NULL, destination TEXT NOT NULL, PRIMARY KEY (source) );
CREATE TABLE users (email varchar(80) NOT NULL, password varchar(20) NOT NULL, PRIMARY KEY (email) );
CREATE TABLE transport ( domain varchar(128) NOT NULL default '', transport varchar(128) NOT NULL default '', UNIQUE KEY domain (domain) );
quit
 
# bind mysql to localhost
sed -i 's/symbolic-links=0/symbolic-links=0\nbind-address=127.0.0.1/g' /etc/my.cnf
 
# restart mysql
systemctl restart mariadb.service
 
# configure postfix
touch /etc/postfix/mysql-virtual_domains.cf /etc/postfix/mysql-virtual_forwardings.cf /etc/postfix/mysql-virtual_mailboxes.cf /etc/postfix/mysql-virtual_email2email.cf
echo -e "user = mail_admin" >> /etc/postfix/mysql-virtual_domains.cf 
echo -e "user = mail_admin" >> /etc/postfix/mysql-virtual_forwardings.cf 
echo -e "user = mail_admin" >> /etc/postfix/mysql-virtual_mailboxes.cf 
echo -e "user = mail_admin" >> /etc/postfix/mysql-virtual_email2email.cf
echo -e "password = Password" >> /etc/postfix/mysql-virtual_domains.cf 
echo -e "password = Password" >> /etc/postfix/mysql-virtual_forwardings.cf 
echo -e "password = Password" >> /etc/postfix/mysql-virtual_mailboxes.cf 
echo -e "password = Password" >> /etc/postfix/mysql-virtual_email2email.cf
echo -e "dbname = mail" >> /etc/postfix/mysql-virtual_domains.cf 
echo -e "dbname = mail" >> /etc/postfix/mysql-virtual_forwardings.cf 
echo -e "dbname = mail" >> /etc/postfix/mysql-virtual_mailboxes.cf 
echo -e "dbname = mail" >> /etc/postfix/mysql-virtual_email2email.cf
echo -e "query = SELECT domain AS virtual FROM domains WHERE domain='%s'" >> /etc/postfix/mysql-virtual_domains.cf 
echo -e "query = SELECT destination FROM forwardings WHERE source='%s'" >> /etc/postfix/mysql-virtual_forwardings.cf 
echo -e "query = SELECT CONCAT(SUBSTRING_INDEX(email,'@',-1),'/',SUBSTRING_INDEX(email,'@',1),'/') FROM users WHERE email='%s'" >> /etc/postfix/mysql-virtual_mailboxes.cf 
echo -e "query = SELECT email FROM users WHERE email='%s'" >> /etc/postfix/mysql-virtual_email2email.cf
echo -e "hosts = 127.0.0.1" >> /etc/postfix/mysql-virtual_domains.cf 
echo -e "hosts = 127.0.0.1" >> /etc/postfix/mysql-virtual_forwardings.cf 
echo -e "hosts = 127.0.0.1" >> /etc/postfix/mysql-virtual_mailboxes.cf 
echo -e "hosts = 127.0.0.1" >> /etc/postfix/mysql-virtual_email2email.cf
 
# alter permissions
chmod o= /etc/postfix/mysql-virtual_*.cf
chgrp postfix /etc/postfix/mysql-virtual_*.cf
 
# add vmail user and group
groupadd -g 5000 vmail
useradd -g vmail -u 5000 vmail -d /home/vmail -m
 
# final postfix config
postconf -e 'myhostname = smtp.example.com'
postconf -e 'mydestination = $myhostname, localhost, localhost.localdomain'
postconf -e 'mynetworks = 127.0.0.0/8'
postconf -e 'inet_interfaces = all'
postconf -e 'message_size_limit = 30720000'
postconf -e 'virtual_alias_domains ='
postconf -e 'virtual_alias_maps = proxy:mysql:/etc/postfix/mysql-virtual_forwardings.cf, mysql:/etc/postfix/mysql-virtual_email2email.cf'
postconf -e 'virtual_mailbox_domains = proxy:mysql:/etc/postfix/mysql-virtual_domains.cf'
postconf -e 'virtual_mailbox_maps = proxy:mysql:/etc/postfix/mysql-virtual_mailboxes.cf'
postconf -e 'virtual_mailbox_base = /home/vmail'
postconf -e 'virtual_uid_maps = static:5000'
postconf -e 'virtual_gid_maps = static:5000'
postconf -e 'smtpd_sasl_type = dovecot'
postconf -e 'smtpd_sasl_path = private/auth'
postconf -e 'smtpd_sasl_auth_enable = yes'
postconf -e 'broken_sasl_auth_clients = yes'
postconf -e 'smtpd_sasl_authenticated_header = yes'
postconf -e 'smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination'
postconf -e 'smtpd_use_tls = yes'
postconf -e 'smtpd_tls_cert_file = /etc/pki/dovecot/certs/dovecot.pem'
postconf -e 'smtpd_tls_key_file = /etc/pki/dovecot/private/dovecot.pem'
postconf -e 'virtual_create_maildirsize = yes'
postconf -e 'virtual_maildir_extended = yes'
postconf -e 'proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps $virtual_alias_domains $virtual_mailbox_maps $virtual_mailbox_domains $relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps $recipient_canonical_maps $relocated_maps $transport_maps $mynetworks $virtual_mailbox_limit_maps'
postconf -e 'virtual_transport = dovecot'
postconf -e 'dovecot_destination_recipient_limit = 1'
 
echo -e "dovecot   unix  -       n       n       -       -       pipe" >> /etc/postfix/master.cf
echo -e "    flags=DRhu user=vmail:vmail argv=/usr/libexec/dovecot/deliver -f ${sender} -d ${recipient}" >> /etc/postfix/master.cf
 
# enable and start postfix
systemctl enable postfix.service
systemctl start postfix.service
 
# configure dovecot
mv /etc/dovecot/dovecot.conf /etc/dovecot/dovecot.conf-backup
touch /etc/dovecot/dovecot.conf
echo -e "protocols = imap" >> /etc/dovecot/dovecot.conf
echo -e "log_timestamp = \"%Y-%m-%d %H:%M:%S \"" >> /etc/dovecot/dovecot.conf
echo -e "mail_location = maildir:/home/vmail/%d/%n/Maildir" >> /etc/dovecot/dovecot.conf
echo -e "" >> /etc/dovecot/dovecot.conf
echo -e "ssl_cert = </etc/pki/dovecot/certs/dovecot.pem" >> /etc/dovecot/dovecot.conf
echo -e "ssl_key = </etc/pki/dovecot/private/dovecot.pem" >> /etc/dovecot/dovecot.conf
echo -e "" >> /etc/dovecot/dovecot.conf
echo -e "namespace {" >> /etc/dovecot/dovecot.conf
echo -e "    type = private" >> /etc/dovecot/dovecot.conf
echo -e "    separator = ." >> /etc/dovecot/dovecot.conf
echo -e "    prefix = INBOX." >> /etc/dovecot/dovecot.conf
echo -e "    inbox = yes" >> /etc/dovecot/dovecot.conf
echo -e "}" >> /etc/dovecot/dovecot.conf
echo -e "" >> /etc/dovecot/dovecot.conf
echo -e "service auth {" >> /etc/dovecot/dovecot.conf
echo -e "    unix_listener auth-master {" >> /etc/dovecot/dovecot.conf
echo -e "        mode = 0600" >> /etc/dovecot/dovecot.conf
echo -e "        user = vmail" >> /etc/dovecot/dovecot.conf
echo -e "    }" >> /etc/dovecot/dovecot.conf
echo -e "" >> /etc/dovecot/dovecot.conf
echo -e "    unix_listener /var/spool/postfix/private/auth {" >> /etc/dovecot/dovecot.conf
echo -e "        mode = 0666" >> /etc/dovecot/dovecot.conf
echo -e "        user = postfix" >> /etc/dovecot/dovecot.conf
echo -e "        group = postfix" >> /etc/dovecot/dovecot.conf
echo -e "    }" >> /etc/dovecot/dovecot.conf
echo -e "" >> /etc/dovecot/dovecot.conf
echo -e "user = root" >> /etc/dovecot/dovecot.conf
echo -e "}" >> /etc/dovecot/dovecot.conf
echo -e "" >> /etc/dovecot/dovecot.conf
echo -e "service auth-worker {" >> /etc/dovecot/dovecot.conf
echo -e "    user = root" >> /etc/dovecot/dovecot.conf
echo -e "}" >> /etc/dovecot/dovecot.conf
echo -e "" >> /etc/dovecot/dovecot.conf
echo -e "protocol lda {" >> /etc/dovecot/dovecot.conf
echo -e "    log_path = /home/vmail/dovecot-deliver.log" >> /etc/dovecot/dovecot.conf
echo -e "    auth_socket_path = /var/run/dovecot/auth-master" >> /etc/dovecot/dovecot.conf
echo -e "    postmaster_address = postmaster@example.com" >> /etc/dovecot/dovecot.conf
echo -e "}" >> /etc/dovecot/dovecot.conf
echo -e "" >> /etc/dovecot/dovecot.conf
echo -e "protocol pop3 {" >> /etc/dovecot/dovecot.conf
echo -e "    pop3_uidl_format = %08Xu%08Xv" >> /etc/dovecot/dovecot.conf
echo -e "}" >> /etc/dovecot/dovecot.conf
echo -e "" >> /etc/dovecot/dovecot.conf
echo -e "passdb {" >> /etc/dovecot/dovecot.conf
echo -e "    driver = sql" >> /etc/dovecot/dovecot.conf
echo -e "    args = /etc/dovecot/dovecot-sql.conf.ext" >> /etc/dovecot/dovecot.conf
echo -e "}" >> /etc/dovecot/dovecot.conf
echo -e "" >> /etc/dovecot/dovecot.conf
echo -e "userdb {" >> /etc/dovecot/dovecot.conf
echo -e "    driver = static" >> /etc/dovecot/dovecot.conf
echo -e "    args = uid=5000 gid=5000 home=/home/vmail/%d/%n allow_all_users=yes" >> /etc/dovecot/dovecot.conf
echo -e "}" >> /etc/dovecot/dovecot.conf
 
# configure dovecot to communicate to mysql
touch /etc/dovecot/dovecot-sql.conf.ext
echo -e "driver = mysql" >> /etc/dovecot/dovecot-sql.conf.ext
echo -e "connect = host=127.0.0.1 dbname=mail user=mail_admin password=Password" >> /etc/dovecot/dovecot-sql.conf.ext
echo -e "default_pass_scheme = CRYPT" >> /etc/dovecot/dovecot-sql.conf.ext
echo -e "password_query = SELECT email as user, password FROM users WHERE email='%u';" >> /etc/dovecot/dovecot-sql.conf.ext
 
# set file permissions
chgrp dovecot /etc/dovecot/dovecot-sql.conf.ext
chmod o= /etc/dovecot/dovecot-sql.conf.ext
 
# enable and start the service
systemctl enable dovecot.service
systemctl start dovecot.service
 
# forward mail to the postmaster account
echo "root: postmaster@example.com" >> /etc/aliases
newaliases
systemctl restart postfix.service
 
# add users to the mail server
mysql -u root -p
 
USE mail;
INSERT INTO domains (domain) VALUES ('example.com');
INSERT INTO users (email, password) VALUES ('timothy.quinn@example.com', ENCRYPT('Password'));
quit