Postfix/Zarafa mail server move

We recently needed to upgrade the storage on our email server. To keep downtime to a minimum we decided to install the server on new hardware and transition everything over. In the past when we used maildir for mail storage this was easy. Simply pause the queue, rsync the mail, point SMTP ports to new server, unpause queue, rsync again. The upgrade this time is more complicated because we are using Zarafa. Zarafa is a great email server. There are open source, free, and commercially licensed version available. It supports Microsoft Outlook clients (0 in the open source version, 3 in the free version, and as many as you pay for in the commercial version) and has an excellent webmail addon. In addition, the company behind it has released (as open source) a Microsoft Exchange ActiveSync server call Z-Push. Z-Push allows Android and iPhone users to get push email (I assume it would work for BlackBerry and Windows Phone as well, but I do not know anyone with those devices).

The main Zarafa server uses MySQL with InnoDB storage to store all email. Attachments are stored as normal files. In our installation the database file size is ~37G and the sql dump is ~22G.

Our current database is ~22G in a .sql dump file. The actual dump process takes approximately 34 minutes. Another 6 minutes to copy the file to the new host and then 116 minutes to restore into the new database server.

Because the restore takes the most time, we tried different options to see if they affected the restore time. This is what we settled on for our restore:
/usr/bin/mysqldump -Q --no-autocommit --single-transaction --skip-opt --add-drop-database --add-drop-table --add-locks --create-options --disable-keys --extended-insert --quick --set-charset -h localhost -u user -ppassword zarafa > zarafa_backup.sql

Essentially the entire process of moving the DB from the old to new server takes about 2 hours. That means that our mail queue must hold mail for 2 hours.

I search quite a bit trying to find a way to make postfix simply queue all mail. Our users are virtual (info pulled from LDAP) and everything I tried resulted in mail delivery being attempted. The setting that fixed things is virtual_transport=defer in main.cf. Set this option and then postfix reload. This makes mail start to pile up in the mail queue.

Once the incoming mail is paused, the next step is to prevent anyone from accessing their mail through the web interface. For use, the web interface is the most used access method. We simply set the Apache server to display a maintenance page for any access.

With these two things in place, it was time to start the actual copy of the DB. The MySQl dump/copy/restore can be combined into one command:
/usr/bin/mysqldump -Q --no-autocommit --single-transaction --skip-opt --add-drop-database --add-drop-table --add-locks --create-options --disable-keys --extended-insert --quick --set-charset -h localhost -u user -ppassword zarafa | mysql -h [new server] -u user -ppassword

Once the DB copy process started all that is left is waiting.

Once the DB completes, we used rsync to copy over the attachments. We had previously done this so almost all attachments were in place. This step only took about 5 minutes.

With the mail database and attachments in place on our new server, it was time to test everything. We connected directly to the IP of the neew mail host in our DMZ and ensured we could send email within the domains hosted on our server. Everything looked good so it was time to redirect the incoming mail to the new server. Simple change to the SMTP ports on the firewall and new mail was coming to the new server. Remove the maintenance page from Apache and users could access their email.

The last step is to make sure the queued mail gets delivered to the new server. Zarafa lmtp is used as the virtual_transport. The current virtual transport is defer. Edit the main.cf file and change virtual_transport=lmtp:[new server ip]:2003. postfix reload to pick up the new settings. postsuper -r ALL is the final command to attempt delivery of the queued mail.

That's it! Actually the entire planning, testing restore settings, DB copy/move testing took several days. Here is the complete list of steps:

1) Pause incoming mail queue:
~# postconf -e virtual_transport=defer; postfix reload
2) Make Apache return a maintenance page
3) Copy database to new server
~# /usr/bin/mysqldump -Q --no-autocommit --single-transaction --skip-opt --add-drop-database --add-drop-table --add-locks --create-options --disable-keys --extended-insert --quick --set-charset -h localhost -u user -ppassword zarafa | mysql -h [new server] -u user -ppassword
4) Rsync (or other file copy) attachment folder to new server
5) Start the Zarafa services on the new server. Make sure you delete the Zarafa indexer files if you copied them from the old server.
6) Test new server for sending mail within domains it is the final destination for.
7) Remove Apache maintenance page.
8) Make sure firewall SMTP ports point to new server
9) Deliver queued mail: postconf -e virtual_transport=lmtp:[new server ip]:2003; postfix reload; postsuper -r ALL

[EDIT]
We actually had a need to move our mail server twice. The second time we just copied the MySQL data files (under /var/lib/mysql) instead of doing a dump/restore on the database. Using rsync, it took less than 45 minutes to copy ~37G of data. That shortened the move time from about 2 hours to under 1 hour.