How to Install an Email Server on Ubuntu with Postfix and Dovecot

Self-hosting email requires more than installing SMTP and IMAP. Reliable delivery depends on IP reputation, reverse DNS, TLS, SPF, DKIM, DMARC, spam filtering, open-relay prevention, queue monitoring, and recoverable backups. This guide builds a single-domain Ubuntu mail server with Postfix, Dovecot, and Rspamd.

If email is business-critical and no one can operate it continuously, use a managed provider. A correct configuration still requires patching, blacklist monitoring, abuse handling, and recovery work.

Architecture

Examples use example.com, mail.example.com, and 203.0.113.10. Replace every placeholder.

1. Verify VPS requirements

• The provider allows outbound TCP 25 and does not use a shared sending IP.
• You can set the IP's PTR to mail.example.com.
• The IP is not on major blocklists or an obviously poor-reputation range.
• The server has a static IP and roughly 2 GB RAM for a small stack.
• You control A, MX, and TXT records.

If outbound 25 is blocked, use a reputable SMTP relay. Do not bypass provider policy.

2. Create foundational DNS

Ask the VPS provider to set 203.0.113.10 PTR to mail.example.com. Forward-confirmed reverse DNS means the hostname resolves back to the same address.

Until IPv6 and its PTR are correctly configured, avoid an AAAA record for the mail host and use IPv4 in Postfix.

3. Prepare Ubuntu and the firewall

hostnamectl set-hostname mail.example.com
apt update && apt full-upgrade -y
apt install -y postfix dovecot-imapd dovecot-lmtpd \
  rspamd redis-server certbot fail2ban swaks ufw

Select Internet Site in the Postfix prompt and use example.com as the system mail name.

ufw default deny incoming
ufw default allow outgoing
ufw allow from ADMIN_IP to any port 22 proto tcp
ufw allow 25/tcp
ufw allow 80/tcp
ufw allow 587/tcp
ufw allow 993/tcp
ufw enable

Port 25 handles server SMTP, 587 authenticated submission, 993 IMAPS, and 80 certificate validation. Do not expose databases, Redis, or internal administration ports.

4. Issue a TLS certificate

certbot certonly --standalone \
  -d mail.example.com \
  --agree-tos -m admin@example.com --no-eff-email

Create a deploy hook that reloads Postfix and Dovecot after renewal, then run certbot renew --dry-run. The certificate lives under /etc/letsencrypt/live/mail.example.com/.

5. Configure Postfix

postconf -e 'myhostname = mail.example.com'
postconf -e 'mydomain = example.com'
postconf -e 'myorigin = $mydomain'
postconf -e 'mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain'
postconf -e 'inet_interfaces = all'
postconf -e 'inet_protocols = ipv4'
postconf -e 'mynetworks = 127.0.0.0/8'
postconf -e 'home_mailbox = Maildir/'
postconf -e 'smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem'
postconf -e 'smtpd_tls_key_file = /etc/letsencrypt/live/mail.example.com/privkey.pem'
postconf -e 'smtpd_tls_security_level = may'
postconf -e 'smtp_tls_security_level = may'
postconf -e 'smtpd_sasl_type = dovecot'
postconf -e 'smtpd_sasl_path = private/auth'
postconf -e 'smtpd_sasl_auth_enable = yes'
postconf -e 'smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination'

Trust only localhost in mynetworks. Broad networks can turn the host into an open relay.

Enable submission in /etc/postfix/master.cf:

submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject

6. Configure Dovecot

Set mail_location = maildir:~/Maildir in 10-mail.conf. In 10-auth.conf, use:

disable_plaintext_auth = yes
auth_mechanisms = plain login

Add this listener inside service auth in 10-master.conf:

unix_listener /var/spool/postfix/private/auth {
  mode = 0660
  user = postfix
  group = postfix
}

For Dovecot 2.3 on Ubuntu, configure 10-ssl.conf:

ssl = required
ssl_cert = </etc/letsencrypt/live/mail.example.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.example.com/privkey.pem

Dovecot syntax changes across major releases. Check dovecot --version and validate with doveconf -n.

7. Create the first mailbox

adduser alice
maildirmake.dovecot /home/alice/Maildir
chown -R alice:alice /home/alice/Maildir

The login is alice and the address is alice@example.com. System users are simple for one domain, but virtual users backed by SQL/LDAP or a packaged mail suite are better for many domains or hundreds of mailboxes.

8. Integrate Rspamd

Verify the Rspamd milter on 127.0.0.1:11332, then:

postconf -e 'milter_protocol = 6'
postconf -e 'milter_default_action = accept'
postconf -e 'smtpd_milters = inet:127.0.0.1:11332'
postconf -e 'non_smtpd_milters = inet:127.0.0.1:11332'

The accept fallback keeps SMTP available during a temporary filter failure, but monitoring must catch prolonged outages.

9. Generate DKIM

mkdir -p /var/lib/rspamd/dkim
rspamadm dkim_keygen -b 2048 \
  -s mail -d example.com \
  -k /var/lib/rspamd/dkim/example.com.mail.key \
  > /root/example.com.mail.dkim.txt
chown -R _rspamd:_rspamd /var/lib/rspamd/dkim
chmod 600 /var/lib/rspamd/dkim/example.com.mail.key

Publish the TXT public key at mail._domainkey.example.com. Configure Rspamd's dkim_signing module with selector mail, the domain, and private-key path. Confirm the package service user before assigning ownership.

postfix check
doveconf -n
rspamadm configtest
systemctl restart postfix dovecot rspamd

10. Add DMARC

_dmarc.example.com. TXT
"v=DMARC1; p=none; rua=mailto:dmarc@example.com; adkim=s; aspf=s"

Begin with monitoring. After every legitimate sender passes and aligns SPF/DKIM, move gradually to quarantine and then reject. Do not enforce rejection before inventorying CRM, invoicing, and marketing senders.

11. Test relay and delivery

postconf -n
ss -lntp | grep -E ':25|:587|:993'
swaks --server mail.example.com \
  --from outside@example.net \
  --to user@another-domain.test

Unauthenticated relay to an external domain must be rejected. Test authenticated submission with TLS, deliver to Gmail or Outlook, and inspect Authentication-Results for SPF, DKIM, and DMARC.

12. Fail2ban, logs, and queue

Enable suitable Postfix SASL and Dovecot jails, confirming Ubuntu's journal or log paths. Monitor:

journalctl -u postfix -u dovecot -u rspamd -f
postqueue -p
rspamc stat
fail2ban-client status

Alert on queue growth, disk pressure, certificate expiry, stopped filters, and delivery-error spikes. Do not retry permanently rejected mail forever.

13. Backup and recovery

Back up mailbox storage, Postfix/Dovecot/Rspamd configuration, DKIM private keys, DNS records, user data, and firewall/PTR documentation. Encrypt backups, store them outside the VPS, and test mailbox restoration in isolation. Treat DKIM keys as secrets and rotate the selector if one is exposed.

14. Operational limits

• A new IP can be throttled even with correct configuration; reputation takes time.
• Do not use this server for bulk marketing without unsubscribe, complaints, and warm-up controls.
• One VPS is a single point of failure; a secondary MX is not a mailbox backup.
• Webmail such as Roundcube is another application to secure and patch.
• Avoid catch-all mailboxes because they attract spam.

Production checklist

• Forward and reverse DNS match; SMTP HELO uses the correct FQDN.
• MX, SPF, DKIM, and DMARC resolve correctly.
• Port 25 is not an open relay; port 587 requires TLS and authentication.
• IMAP requires TLS and certificate renewal passes dry-run.
• Database and Redis are private; SSH uses keys and source restrictions.
• Queue, disk, blocklists, logs, certificates, and backups are monitored.
• An off-host backup has passed a restore test.
• postmaster@, abuse@, and DMARC reports are monitored.

Conclusion

A functional mail server can be built in hours; a trustworthy one requires continuous operations. Postfix, Dovecot, and Rspamd are a strong small-scale foundation when DNS, TLS, authentication, monitoring, and backups are complete. If PTR, outbound port 25, or on-call operations are unavailable, choose managed email or an SMTP relay.

References

• Postfix: Basic Configuration
• Dovecot: SSL/TLS Configuration
• Rspamd: MTA Integration
• Google: Email sender guidelines