Tag/tag.png

Needs Expansion
This article is incomplete, and needs to be expanded. More info...

This page will walk you through setting up an IMAP server that will periodically download mail from any number of POP3 mail servers and store them on the IMAP server.

Purpose

The IMAP protocol allows you to access your mail from any IMAP-compatible email client. The major advantage over the POP3 protocol is the ability to view the same email from multiple computers. This process will provide that benefit for providers that do not yet support IMAP.

Requirements

This process was originally written for Ubuntu Dapper 6.06, but has been updated for Ubuntu Hardy 8.04. The IMAP server used is Dovecot, along with GetMail for retrieving the mail from the POP3 servers.

Installation

You will only need two packages (not including any dependencies) to make this work.

sudo apt-get install dovecot-imapd getmail4

Pre-Configuration / Gather Information

When installing Dovecot and GetMail, it creates the following users and groups: dovecot and mail. This is important because we will be using Dovecot's virtual users feature instead of creating a new Linux user for each account you want to gather mail for. The "mail" user was given a UID of 8 on the test machine, but yours may differ. You will need to UID number for the "mail" user. You can obtain this by running the following command:  cat /etc/passwd|grep mail  This will return a line similar to the following: mail:x:8:8:mail:/var/mail:/bin/sh The first number, which is 8 in this case, is the UID.

Configuration of Dovecot

Now that you have the UID for the "mail" user, we can edit the configuration files for Dovecot. If you are using the desktop version of Ubuntu, I recommend editing text configuration files with GEdit. You can do this opening a Terminal window and typing this:

gksu gedit /etc/dovecot/dovecot.conf

If you are using the server version of Ubuntu, open /etc/dovecot/dovecot.conf using your preferred text editor. The easiest editor to use is nano. To edit this file with nano, use this command:

sudo nano /etc/dovecot/dovecot.conf

Basic Formatting Information

This file (dovecot.conf) uses the hash(#) symbol at the beginning of lines to represent comments, which are ignored by Dovecot. The verb "uncomment" is used during these steps, which simply means removing the hash(#) symbol from the beginning of the line.

dovecot.conf

  1. Locate and uncomment the line  protocols = imap imaps 

  2. If you plan on allowing Outlook to connect to the server, you will need to uncomment the line  disable_plaintext_auth = no . You will also want to enable the Outlook workaround that prevents problems with Outlook being connected for longer than thirty minutes. You can do this by uncommenting the line  imap_client_workarounds = outlook-idle .

  3. Locate the line  first_valid_uid  and change the value to the UID of the "mail" user. So, since my "mail" user's UID is 8, the line should look like this:  first_valid_uid = 8  .

  4. Do the same for the line  last_valid_uid . This way, only our virtual users will be able to log in.

  5. Locate and uncomment the line  mail_location =  (For Dapper 6.06, look for  default_mail_env = ). Change the value to  maildir:/var/mail/%u/Maildir  This allows Dovecot to automatically create the required directories for users the first time they log in.

  6. If you want to enable users to create folders inside the Inbox folder in their mail clients, then you will need to locate the line  namespace private {  and uncomment it. You will also need to uncomment the closing brace (}) about 20 lines down.

  7. Locate and change the following the values inside the  namespace private  braces:

    1. Locate and uncomment the line  separator  and set its value to a single forward slash (/).

    2. Locate and uncomment the line  inbox = yes .

  8. Locate the line  mechanisms  inside the  auth default  braces and make sure it is uncommented.

  9. Find the following block:
    •     #passdb passwd-file {
           # Path for passwd-file
           args = 
          #} 

      And change the  args  value to  /etc/dovecot/passwd . You will also need to uncomment the line  #passdb-passwd-file  along with its closing brace (}) a couple lines down.

  10. Locate and uncomment the line  userdb static { . You will also need to uncomment its closing brace (}) about 10 lines down.

    1. You will also need to change the  args  value to  args = uid=mail gid=mail home=/var/mail/%u .

  11. Save and close the file.
  12. Enter the following commands to set the permissions of the dovecot.conf so that the server has access to read it:
    •     sudo chown dovecot /etc/dovecot/dovecot.conf
          sudo chgrp dovecot /etc/dovecot/dovecot.conf

passwd

This file will hold all the virtual users that will be allowed to log in to the Dovecot server. This example will store passwords in plain text, but will be secured so that no one but root and the Dovecot server can read it. If you need to encrypt the passwords in this file, then refer to http://wiki.dovecot.org/HowTo/SimpleVirtualInstall under the Passwords section.

  1. Create and open the file /etc/dovecot/passwd.

  2. Every line in this file will represent a new user. The syntax is rather simple:  username:{PLAIN}password , replacing "username" and "password" respectively. A quick note here is that you can also use the "@" symbol to represent domains, even though Dovecot does not care about domains.

  3. For example, if I want two virtual users, Bob and Sarah, this is what the passwd file would contain:
    •     bob:{PLAIN}mypassword
          sarah@nowhere.com:{PLAIN}thepassword
  4. Once you have finished creating your virtual users, save and close the file.
  5. In Terminal, enter the following commands:
    •     cd /etc/dovecot
          sudo chown dovecot passwd
          sudo chgrp dovecot passwd
          sudo chmod 600 passwd
      This will change the owner and group to "dovecot", and only allow root and the "dovecot" user to read the information.
  6. You can now start the Dovecot server:  sudo /etc/init.d/dovecot restart 

  7. At this point, you should test to see if you can log in to the server with an email client that supports IMAP. If you can log in without any error messages, everything is working fine. If you do get any errors from the mail client trying to log in, first, make sure the server did not give an error when running the restart command above. Second, make sure the permissions on the dovecot.conf and the passwd file are correct. The user "dovecot" should own the dovecot.conf file and the passwd file, and that the "mail" user owns the /var/mail folder including everything in it.

Final Note on Dovecot Configuration

Dovecot is pretty smart when it comes to virtual users. Any time you edit the passwd file, it updates itself, so there is no need to restart Dovecot every time you add or remove a user account.

Configuring GetMail

GetMail does have more flexibility over fetchmail, but still has its limits. It wasn't really designed for multiple accounts, so setting it up for our usage is a little more tedious. For each POP3 account it needs to connect to, it needs a separate file. Though not the most elegant way to manage multiple users and/or accounts, a bash script could manage it with little difficulty.

  1. For this part, it's easier if we make changes under the "mail" user's account, but first we need to make sure that the /var/mail has the "mail" user set as its owner. So issue this command:  sudo chown mail /var/mail .

  2. Now lets impersonate the "mail" user:  sudo su mail .

  3. Create the needed directory for getmail:  mkdir /var/mail/.getmail . The dot in front of "getmail" is important.

  4. This directory will contain our POP3 server information, so lets prevent other users from being able to read the information:  chmod 700 .getmail .

  5. Change to the newly created directory:  cd /var/mail/.getmail 

  6. Since we have to create a new file for every POP3 account, I recommend using the username for the file's name to avoid confusion. So if Sarah's username was "sarah@nowhere.com", I would create the filename "sarah@nowhere.com".

  7. Create and open a file for the first POP3 account you want to create. For our example, I'll use the filename above.
  8. This file will be referred to as an "rc file" from now on. The file should contain the following:
    •     [retriever]
          type = SimplePOP3Retriever
          server = pop3.nowhere.com
          username = sarah@nowhere.com
          password = thepassword
      
          [destination]
          type = Maildir
          path = /var/mail/sarah@nowhere.com/Maildir/
      
          [options]
          delete = false
          read_all = false
      Replace the information for "server", "username", "password", and "path" with the correct information. Also note the ending forward slash(/) at the end of the "path" property. The "delete" option is set to false for testing purposed, but you may want to set it to true later. When set to true, getmail will delete the messages from the server once they have been downloaded. The "read_all" option prevents duplicate emails from being downloaded from the server while the "delete" option is set to false.
  9. Save and close the file. Repeat this process for every POP3 account you want getmail to download from.
  10. At this point, you should test getmail to see if it is able to connect to your POP3 server and download emails to the correct location:  getmail --rcfile=sarah@nowhere.com 

  11. If didn't get any errors, then check your email client to confirm that new messages were delivered correctly. However, if you did encounter any errors, make sure that your files are owned by the "mail" user and that you have entered the information correctly.

The Cron Job

At this point, you have a working Dovecot IMAP server and a working GetMail configuration, but one problem still remains. GetMail only runs when it's told to. So in order for it to regularly download your mail from the POP3 servers, you will need to set up a cron job.

  1. You should still be impersonating the "mail" user. If not, issue this command:  sudo su mail .

  2. Since the plan is to make this process scalable, we need to create a batch file that contains the commands to run all the rc files you created. So create the file getmail.script in /var/mail and open it for editing.

  3. For each rc file, add the following line:  getmail --rcfile=sarah@nowhere.com . Obviously, you need to substitute "sarah@nowhere.com" with the name of the rc file.

  4. Save and close this file.
  5. Change the permissions so that it is executable:  chmod 755 /var/mail/getmail.script 

  6. Now we need to add the script to the crontab, so issue the following command:  crontab -e 

  7. Add the following line to the file:  */5 * * * * sh /var/mail/getmail.script 

  8. This will run the script every 5 minutes, but you can change the interval by replacing the 5 with whatever you want If you plan to set the script to run every minute, you need to remove the /5 so that star is the only thing that remains.

  9. Save and close the file.
  10. Type  exit  to stop impersonating the "mail" user.

  11. Watch as your new mail gets automatically downloaded.

Adding New Users

In order to add new users to the system and download from more POP3 accounts do the following:

  1. Add the login information to /etc/dovecot/passwd.

  2. Create a new rc file either by impersonating the "mail" user or setting the owner and group to "mail" after creation.
  3. Add a new line to the /var/mail/getmail.script file that references your new rc file.

Backup and Restore

I have found that the best way to back everything up is to simply backup the /etc/dovecot folder, and the /var/mail folder. When restoring to a new operating system installation, simply install the two packages list above, in the Installation section, then restore the /etc/dovecot and /var/mail folders. Finally, you will need to set the cron job up again for automatic mail retrieving as listed above by completing step 1 in the Cron Job section, then skipping to step 6 (since you will already have a getmail.script file).

Quick Note on Permissions

If, for some reason, you cannot restore the backed up folders with their original permissions, you will need to change the owner, group, and read/write permissions. It's very simple though, because the /etc/dovecot folder needs to be owned by the "dovecot" user and group (recursively). Likewise, the /var/mail needs to be owned by the "mail" user and group (recursively). As far as access control, setting the permissions for both folders, recursively,to 700 should be perfect, though restrictive. You could also set it to 770 to allow members of the dovecot/mail groups access to the files, while preventing other users from reading them at all.

Here are the specific commands in case you need them:

sudo chown -R dovecot /etc/dovecot
sudo chgrp -R dovecot /etc/dovecot
sudo chmod -R 700 /etc/dovecot

sudo chown -R mail /var/mail
sudo chgrp -R mail /var/mail
sudo chmod -R 700 /var/mail

Upgrading from Dapper (6.06)

If you are currently running this server on a Dapper (6.06) server and are planning to upgrade to Hardy (8.04), then it is recommended that you follow the Backup and Restore section above. Once you have completed the restore process, you will need to make a small change to the /etc/dovecot/dovecot.conf file to deal with the newer version of dovecot included with Hardy.

  1. Open the /etc/dovecot/dovecot.conf file for editing.

  2. Locate the line default_mail_env and rename it to mail_location.

  3. Save and close the file.
  4. Restart the dovecot server.

Final Notes

I am new to linux mail servers, and IMAP in general, but if you have any problems or questions, or better yet, suggestions on simplifying this process, please add a comment to the Comments section below. This is my first from-scratch wiki article, and my first linux how-to, so I hope it helps somebody.


The Dovecot Wiki is a well-organized, easily digestible reference for configuring Dovecot. Take note of the troubleshooting advice. I found this Ubuntu Community POP3-Aggregator page to be pretty much flawless, however I still found the Dovecot Wiki very helpful in both resolving problems from trivial mistakes I made, and developing a clearer understanding of what the config above is doing. (Thanks to the original author for this quite useful page. It was exactly what I was looking for! I followed the process and it worked great!)

Acknowledgments

This article was written by RossPeoples with the help of PhilBill and everyone else who was kind enough to leave their comments. Thank you!

Comments

Congratulations on a very comprehensive document, it has save me hours of digging into the dovecot docs. One comment though. In section 4 of Configuring getmail you say chmod 600 .getmail it should be chmod 700 .getmail otherwise you can't change into the directory in step 5

  • Just made the change. Thanks for the info! RossPeoples


I followed these instructions. Very useful! Two caveats though:

  1. virtual usernames can not be the same as real usernames in this setup.
  2. In the dovecot.conf that was installed on my system there are two passdb passwd-file sections. The first one is for deny, the second one can be used to specify the args=/etc/dovecot/passwd file.


I just updated the article to include the changes needed for Hardy. Also, after moving from Dapper to Hardy myself, I decided to add a Backup and Restore section on how I was able to move all my mail to the new server. It works perfectly and since I was updating from Dapper, I also added a section specific to the small change that needs to be made in the dovecot.conf file. RossPeoples


Thank you, this is just what i've been looking for. I used this config to make a home IMAP server on a G3 iMac running Ubuntu Server 6.06 (Dapper) and it worked straight away. Brilliant! No need to go through the long-winded ISP style setups and work out which bits i need and which i don't. Now i just need to add Apache/SquirrelMail and i'm done!


Thanks! Works nicely! Just a comment for people like me not figuring out where is the send message part in this HOWTO... You don't have to manage the SEND message. The mail client (Thunderbird) have to be configured to send the message by itself (easy) and it will also send a copy of the sent message to the local mail server (I absulutely wanted a local copy of the message), so the other mails clients can also see the message SENT by other peoples. I have searched two days on the web for this missing part in this HOTWO without knowing this was so simple and not needed. I hope this will save times to others. Thanks for this easy to follow Guide!


Thanks for this HowTo!

If you want to integrate Spam Assassin, have a look at its FAQ.

POP3Aggregator (last edited 2010-10-04 17:20:05 by Student)