Categories: Linux

(Back to the main article)

Introduction

This article describes some of the concepts and terminology used by the Postfix suite of email tools. This hopefully makes descriptions of how to install Postfix easier to understand.

The version of Postfix described here is v3.4 (as included in Ubuntu 20.04-LTS).

Origins

Postfix was intended to be (and largely is) a replacement for the sendmail program. The configuration is far saner than sendmail, and the security aspects much better.

Postfix competes with Exim (and earlier with qmail). Postfix is almost completely implemented and maintained (over decades) by Wietse Venema, an IBM research fellow.

Wikipedia has a brief overview of postfix, which links to the postfix architecture page on the postfix website; this article can be seen as an extension of the information there.

The Postfix Master Process

Postfix consists of many separate programs that work together. This design is for security; any code that parses user input is potentially vulnerable to carefully-crafted malicious input which may cause buffer-overruns or trigger other undesired behaviour. Such code can be made more secure by making it a separate program and running that program with restricted rights; the worst case is that an attacker causes the program to perform user-specified actions but when it runs with restricted permissions the damage it can cause is reduced.

There are some standard services on Unix systems that most administrators will be familiar with:

• inetd is configured with a list of (port, program) pairs; it listens on all the specified ports and starts the corresponding program when a remote system opens a connection to a listed port.
• cron is configured with a list of (schedule, program) pairs; it invokes the specified program whenever the specified schedule matches the current time

The postfix master application is somewhat like a combination of inetd and cron. The master runs continuously (started via systemd or similar) and starts other programs as needed. The programs that it may start are configured in file /etc/postfix/master.cf. In particular, on master startup it looks for all listed services with type=inet (ie which listen on a network port) and listens itself on that port, starting an instance of the application when the first client connects. On master shutdown or reload, it sends a signal to all such processes to shutdown. Entries with type=unix are instead started each time they are needed (ie are not long-running daemons); those with a non-zero “wakeup” setting are cron-like batch-mode applications which the master starts at the specified interval (seconds).

The master.cf File

Each entry starts at column 0 in the file (ie no preceding whitespace). It may be extended over many lines, as long as the following lines are indented.

The first column in an entry is the “service name”, a logical key that the master process (or other processes) use to refer to a particular postfix program. This is followed by various settings (see file header), and then the binary application to execute (eg “smtpd”) and its commandline-arguments.

Probably the most important entry, and usually the first one, is “smtp inet ….. smtpd ..” which defines that the “smtp service” as the “smtpd” MTA application listening on an inet (network) port. This is the daemon that listens on port 25 for incoming email, and places it in a queue. The master process will listen on port 25 for incoming connections and start an instance of “the smtp service” as needed (inetd-like).

Configuration Settings and the main.cf File

When the postfix master starts a process listed in master.cf, it provides it with a large set of configuration-variables that the program may read from; these:

• have default values hardwired into the Postfix configuration system
• which can be overridden in file /etc/postfix/main.cf
• which can be overridden via -o name=value options in master.cf.

Entries in master.cf may reference variables defined in main.cf, eg “-o foo=$bar” where bar is defined in main.cf.

Applications with type=unix are started repeatedly (depending on context but often once per email). Those started per-email can include email-specific variables such as $sender in their argument-list. Applications with type=inet are long-running daemons that handle many emails so such variables obviously are not available.

Each postfix program has a set of configuration-values that it looks up from the “postfix configuration” when it starts. Exactly which variables a specific program uses is documented in that program’s man-page.

Postfix has an inbuilt default value for each variable; these can be seen by running postconf -d. The defaults can be overridden via entries in main.cf and via -o commandline options in master.cf.

Postfix is intended to be scalable enough to be usable by very large companies and internet-service-providers. Sadly for those wanting to use it simply for a personal or family mailserver, this means it has a very large number of configuration-options which are mostly irrelevant to smaller configurations. However Postfix does have very good default settings.

The full set of configuration variables can be seen at:

• http://www.postfix.org/postconf.5.html

The current set of configuration values is displayed by command postconf while the default values are displayed by postconf -d.

More information about configuration can be found at:

• http://www.postfix.org/BASIC_CONFIGURATION_README.html

The Postfix default values are very carefully chosen, and for simple setups the number of variables that need to be overriden is just a dozen or two. The problem is simply to know which ones need to be overridden from the vast list in the Postfix documentation!

A long variable definition can be split over multiple lines simply by indenting the following lines.

Some variables are lists of values; the items in the list can be separated by commas or by whitespace, at your choice - unless items may contain whitespace, in which case commas are needed.

Queues and the Queue-Manager

Emails go through various “phases” as they are processed, in which they are temporarily stored on a queue then later taken from the queue and processed - possibly being written to yet another queue.

A queue is just a directory under /var/spool/postfix, eg /var/spool/postfix/incoming for emails received by smtpd or sendmail which are waiting to be delivered.

The queue-manager application periodically looks at all queue-directories, and invokes applications to process the emails it finds. Column wakeup in file master.cf defines how often the queue-manager runs, and the mapping between “service to execute” (from email metadata) and “binary to execute” is defined in master.cf.

MTAs and MDAs

A Mail Transfer Agent (MTA) is something that communicates with external systems to either accept incoming emails and store them in a local “incoming queue”, or to take emails from a local “outgoing queue” and pass them to an external system.

A Mail Delivery Agent (MDA) is something that takes email from an “incoming queue” and writes it to a recipient’s “mailbox” on the same host.

Sometimes an entire email-setup consisting of MTAs, MDAs, and their various internal components is also described as an MTA.

Mail Transfer Agents

Postfix includes the following Mail Transfer Agents:

• sendmail
• smtpd
• smtp

This sendmail is a postfix-specific application which is named sendmail for historical reasons, and provides the same command-line-interface as the original application of the same name. It is an extremely simple program which reads email from its STDIN and writes the email to a “maildrop directory”. The postfix pickup daemon process detects new emails in the maildrop-directory, applies the mail-processing rules defined in postfix configuration-files master.cf and main.cf, and moves the email to the incoming queue. Together these provide a way for applications running on the same host as postfix to generate outgoing email.

An instance of the smtpd application is started by the postfix master process when a remote client connects to port 25 (smtp) or 587 (submission). It exchanges a sequence of ESMTP commands with the remote application and if all is valid it accepts the contents of an email from the remote server and writes it to a local “incoming queue”.

The smtp application is used to send outgoing email. The postfix master process periodically executes the “queue manager” which then starts an instance of the smtp application for each email in the “outgoing queue”. The smtp app takes the domain-part of the recipient address (eg @google.com), performs a DNS lookup of the MX-record for that domain to get a host-name, opens a socket to port 25 on that remote host and exchanges a sequence of ESMTP commands with that remote email-server in order to transfer the contents of the outgoing email. Actually, the smtp application could be described as either an MTA or an MDA; it sits somewhere in-between.

Mail Delivery Agents

Postfix includes several Mail Delivery Agents:

• local
• virtual
• lmtp
• pipe

The postfix MDAs handling incoming email (sendmail + smtpd) determine which delivery-agent should be used for an email (based on the rules in main.cf and master.cf), and include this in metadata associated with the emails they write into the “incoming queue”. The postfix master-process periodically runs the queue-manager which checks each email in the “incoming queue” and executes the associated delivery-agent.

Note that each entry in master.cf starts with a postfix service name and then a later column specifies the binary application to execute (with default path of /usr/lib/postfix/sbin). These are often the same, eg service “local” is mapped to application “local” (thus /usr/lib/postfix/sbin/local). However the service-name and binary-app are separate concepts.

These specific Mail Delivery Agents are described in more detail later.

Email Classes and the SMTPD Server

Dataflow

The smtpd server listens on port 25. A remote application connects to it, and exchanges a series of ESMTP messages with it, first to identify itself and then to transfer (submit) one or more emails.

The general steps that the smtpd server performs are:

• listen on port 25 for a new TCP connection
• wait for remote app to identify itself with an EHLO $(servername) message
• performs a DNS reverse-lookup on the IP address of the remote app
  • if no PTR record is found, this is not a “serious” email server - probably a hacked desktop or similar, so drop connection
  • if PTR record does not match servername in EHLO message, this is an untrustworthy email server, so drop connection
• check whether the EHLO $(servername) is “permitted” according to the local rules
• check remote server name and IP-address against the configured “blacklists” of known spammers
• receive email; for each email:
  • categorize the email into local/virtual/relay/default (see later)
  • check whether the “recipient” address in the email-header is “permitted” according to the local configured rules
  • check whether the “sender” address in the email-header is “permitted” according to the local configured rules
  • execute any configured “rewrite” operations, filters, etc
  • determine which postfix “service” should process the email next (delivery agent aka transport)
  • store the email into an “incoming queue” (a directory) labelled with the “transport”
  • return a success-code to the remote app for this email (email accepted)

At some later time, the Postfix master-process will start the queue-manager which will deal with any queued emails.

Note that Postfix is extremely configurable; the above flow is just a typical one.

Categorization

Incoming email is categorized by the Postfix MTA into one of the following classes:

• local mail (the domain-part of the address matches an entry in list $mydestination)
• virtual mail (the domain-part of the address matches one of the configured virtual-domains; see below)
• relay mail (the domain-part of the address matches one of the configured relay-domains; see below)
• default (the email is destined for some arbitrary mailserver in the internet)

This categorization is done as the remote application submits the email to the local smtpd server, ie before an “ok, accepted” status code is returned for the email being transferred from remote system to this one.

The local category is generally intended for the use-case where each valid email-account corresponds to a native unix-user-account on the mail-server itself. The virtual category is intended for the use-case where a single server holds email for multiple users who are somehow “registered” with that mailserver but not directly as native unix users (eg being registered in a database or a simple text-file). The relay category is for large companies with several cooperating email-servers (not relevant for this article) and “default” is for outgoing email generated by local users which is intended for some email server in the internet.

The “relay” class here does not mean “relaying of email in general”; that is covered by the “default” class. The relay class is intended for cases where a large company may own multiple email-domains, and wishes to have a single front-end server for all domains which then delegates to separate back-end servers, or similar. The domains for which relaying are enabled must be explicitly configured. This is not relevant for a small “personal” email setup. Although “relay” is a class of email, any relayed email will be sent via the SMTP protocol to a remote server, so the smtp Mail Delivery Agent (“smtp”) is the standard transport for this class.

When an email is categorized as local or virtual, a check is immediately made to see if the email-recipient actually exists; if not then the email is not accepted (ie an error is returned to the SMTP client that is trying to pass the email on). Local-type emails are accepted only when local_recipient_maps points to a table that includes the email address, and by default that is set to the /etc/passwd and /etc/aliases files; therefore emails categorized as “local” are rejected unless the recipient really does have a local account, or is listed as an alias. Virtual-type emails are processed similarly, but with different tables of users. Note that aliases are not actually processed at this time, ie the incoming email is not modified. User-validation cannot easily be performed for relay-type mail, and cannot be performed at all for default-type mail.

The SMTPD server then consults a table to determine which transport (postfix service as defined in master.cf) should be used to process that email. For simple setups, this just means that one of the following variables is used to find a string which is then a key into the services-list defined in master.cf:

• local_transport (default value: “local”)
• virtual_transport (default value: “virtual”)
• relay_transport (default value: “smtp”)
• default_transport (default value: “smtp”)

Finally, the SMTPD application stores the email in the “incoming queue” (which roughly means writing it as a file in a specific directory), labelled with the transport (postfix-service-name) that should be used to deliver it. And then a success-status-code is returned to the submitting application.

Periodically the Postfix queue-manager process runs, and for each file in the “incoming queue” it:

• gets the associated transport-service name from the queue-entry
• looks up master.cf to find out which application to execute for that email (which should be under /usr/lib/postfix/sbin).
• removes the email from the queue, starts a new process as specified in master.cf, and passes the email contents to that new process.

Yes, this means the queue-manager starts a new process to deliver each email. The standard MDA applications provided by postfix are, however, fairly light-weight.

Note that Postfix is extremely configurable: each processing step listed above is typically implemented in the Postfix SMTPD server as “execute the operations specified by configuration-variable ${..}”, where that variable defines sensible default behaviour as described above, but which can be configured to perform fewer, extra, or even completely different operations at that processing stage. Some of the available options are described below in “Email Checks”.

Email Checks

The checks that the smtpd server applies to email before it “accepts” (queues) it are defined in variables which have sensible defaults but can be customized:

• smtpd_client_restrictions – a list of tests to apply after the remote application has identified itself with an EHLO message; these check whether the remote app is permitted to submit ANY email.
• smtpd_sender_restrictions – a list of tests to apply to the “sender address” of the email; normally there is nothing here.
• smtpd_recipient_restrictions – a list of tests to apply to the “recipient address” of the email
• smtpd_relay_restrictions – another list of tests to apply to the “recipient address” of the email

An email is only accepted if all tests pass. Each variable defines a list of tests which can return PERMIT, REJECT or DUNNO. As soon as a test returns PERMIT or REJECT, that is the result of the overall test. When a test returns DUNNO then the next item is tested.

There is some overlap between smtpd_recipient_restrictions and smtpd_relay_restrictions - and in fact in earlier versions of Postfix they were just one variable. However having two vars means that the relay-restrictions tests can really focus on checking outbound email while recipient-restrictions can focus on testing inbound email.

Inbound vs Outbound

Email coming from external email-servers (eg gmail) is transferred via ESMTP. Email coming from “known users” running email-client-apps on desktops is also transferred via ESMTP. It is possible to have a single smtpd server that handles both types of incoming data over port 25. However this can lead to lots of confusion, as what kinds of email are allowable are very different in the two cases. It is common practice to instead have two smtpd configurations on two different ports, with different settings in recipient-restrictions and relay-restrictions.

Early Reject

If an email is not going to be delivered, then it is far better to reject it while the remote SMTP server is in the process of sending it, than accepting and queuing it only to later reject it. In that case, it must either be silently discarded or a “bounce” mail be sent; neither is desirable.

Before-queue processing takes place while the originating SMTP server is still connected, and is submitting the email. A mail-rejection at this point is easier on all parties. If an email is accepted (ie success is returned to the originating SMTP server) and the email written to a local “incoming queue”, and then the email is analysed later, then that is “after-queue” processing. After-queue processing can be more efficient, but leads to problems if analysis comes to the conclusion that the email should not be delivered.

Mail Delivery Agents

It has already been discussed how the SMTPD server receives email from a remote system and writes it to a local “queue” (directory) together with the name of the postfix-service that should next be invoked. This service is almost always an application which is a Mail Delivery Agent (MDA) (a notable exception is spamassassin; see later).

The master application eventually starts the queue-manager which passes each queued email to an instance of the specified postfix-service.

The Local MDA

The local Mail Delivery Agent (ie the postfix program /usr/lib/postfix/sbin/local):

• extracts the recipient-name from the base part of the email recipient address (eg “foo” from “foo@example.com”)
• consults the aliases table (usually /etc/aliases), transforming recipient-address to canonical name
• consults /etc/passwd to find a user’s home-directory and UID (triggering a “bounce email” if no entry found)
• looks for user-specific .forward files
• then looks for a special mailbox_transport variable, and if defined then delegates delivery to that service
• else writes the email to a file with name/location specified via configuration (eg $HOME/.Maildir for the recipient-name)

It also has a few other features; see the man-page for the postfix local application for more information.

The local MDA only supports one domain-name, eg “foo@example.com” and thus discards the domain-part and assumes that the name (“foo”) is sufficient to identify the recipient.

In short, the local Mail Delivery Agent is intended for writing email to local mailbox-files where the recipient has a native unix login on the emailserver host.

Local delivery can be used to pass emails on to other software (such as Dovecot) by taking advantage of the mailbox_transport option to point at an LMTP-server provided by the other software. The two systems can also simply be configured to read/write mailboxes in exactly the same format and disk-locations (though this can be tricky to keep in-sync).

Note that for incoming emails of type=local, the existence of the recipient is checked twice: once by smtpd using its local_recipient_maps settings before the email is queued (in before-queue-processing), and then later by local when performing delivery (in after-queue-processing). By default these two applications use the same configuration-files to determine which users exist (/etc/passwd and /etc/aliases).

The local delivery agent also has a config-option luser_relay which states where to forward an email if no user-account can be found for it. This doesn’t normally happen, as smtpd checks for the existence of users before accepting/queuing the email. However smtpd can be configured to NOT do that, in which case luser_relay takes effect. When luser_relay is not defined, and an email cannot be delivered, then a BOUNCE email is sent back to the original sender of the email to inform them of the problem. The luser_relay option has no effect when email using mailbox_transport.

The Virtual MDA

The virtual Mail Delivery Agent uses a virtual users file rather than hard-wiring /etc/passwd as the source of information about valid email recipients. And rather than defaulting to writing emails into a mailbox relative to a user $HOME directory, mailboxes are stored into mailboxes under a “virtual base directory”.

Unlike the local MDA, the virtual MDA supports multiple domain-names, eg “me@example1.com” and “me@example2.com”. All configuration therefore identifies users with the “full” address, not just the name-part, and the target mailbox into which emails are written is also domain-specific (so that the two addresses above are different mailboxes).

In most other aspects, the virtual MDA is similar to the local MDA. This application also has a man-page with further information.

In short, the virtual Mail Delivery Agent is intended for writing email to local mailbox-files where the recipient does not have a native unix login on the emailserver host.

The SMTP MDA

Although the smtp application was categorised as a “mail transfer agent” above, it is also a kind of mail-delivery-agent. When invoked, it performs some DNS lookups to find the host-server for the recipient’s email address (eg for ‘recipient=you@example.com’ looks up the MX record for example.com) then opens a socket to port 25 on that server and passes the email contents over.

In practice, the Postgres “smtp” program is probably just a trivial app which passes the email on to a longer-running local postfix process so that outgoing emails to the same target server can be batched and sent over one network-socket for efficiency. It is likely that the communications-channel from the trivial “smtp” app and its corresponding daemon is the socket at /var/spool/postfix/private/smtp. However that is an internal Postfix implementation detail that is not relevant for configuring Postfix.

The smtp delivery agent is used to handle emails with class=default, ie those which are outgoing from the local server to somewhere on the internet. It is also used for emails with class=relay, ie outgoing from this email server to another tightly-coupled “partner server”, but relaying is a “big company” feature not relevant for this article about personal email servers.

The LMTP MDA

The lmtp Mail Delivery Agent is much simpler than local or even virtual: it just opens a specified socket (usually a filesystem-socket) on which an LMTP-daemon process is presumably listening, and exchanges a series of LMTP-protocol messages with that server to transfer the contents of the email to the LMTP server. This sounds a lot like SMTP, and in fact it is: LMTP is a subset of SMTP, with many of the security-related features taken out as an LMTP server expects its clients to be “trustworthy”. LMTP is therefore suitable only for delivery within an email-server or set of cooperating email servers. And LMTP servers are not expected to write their incoming email into an inbound queue, but instead into a user mailbox. Postfix provides an LMTP client application which acts as a “gateway” to any other applications that provide an LMTP server - such as the Dovecot IMAP server.

And as it happens, the Postfix “smtp” program can be used as a client to talk to an lmtp server, so in /etc/postfix/master.cf the entry for servie “lmtp” also points at the smtp application.

There are two ways to integrate Postfix with Dovecot: either use local or virtual transport, and very carefully ensure that the way that emails are written by postfix is exactly what/where dovecot expects - or use lmtp transport and let Dovecot handle writing of the emails to disk (much better).

The Pipe MDA

The pipe application is a simple gateway to any arbitrary email-processing application. When the queue-manager finds an email waiting in a queue which is labelled with delivery-service “pipe” then it executes the pipe application with whichever arguments are defined in file master.cf; the pipe application then forks a new process as specified with its commandline arguments and writes the email contents to the STDIN of that child process.

Debugging A Postfix Installation

All postfix applications support a “-v” commandline option which can simply be added in master.cf to obtain more information in /var/log/mail.log.

See also /var/log/mail.err.

The current postfix configuration can be seen with:

• postconf

The default configuration can be seen with:

• postconf -d

Postfix and Spamassassin

Spamassassin is a completely different project than Postfix, but is very often used together with it. It is a very widely-used application and can be installed via apt (just like postfix and dovecot). It needs almost no configuration itself, but configuring Postfix to use it is not entirely trivial.

Spamassassin evaluates the “spam factor” of an email so that other applications can then choose whether to discard it, move it to a spam-folder, or take other action. In a home-email-server scenario, it is probably best to just pass emails through Postfix even when they have a high spam-factor, and use dovecot-configuration or even email-client-configuration to push these to a spam-folder so they can be easily ignored or deleted by the end-user. Spam evaluation is purely heuristic in nature, and can sometimes go wrong so automatic deletion of emails is probably best avoided.

Spamassassin comes as two applications: a server (spamd) and a client (spamc). The server runs continuously, while the client is started for each email to be evaluated. As a simple “email recipient” it is possible to install and use spamassassin to evaluate your emails after they have been delivered, ie to use spamassassin even when your email-provider does not. However it is more common for spamassassin to be integrated into the email-server so that all incoming (and possibly outgoing) emails are checked for spam.

Spamassassin accepts a valid Email as input (via a pipe, smtp, lmtp, amavis, or other protocol), and returns the same email with some additional mime-headers which indicate the “spam score” for the email. The score is computed by running a set of configurable rules against the mail; there are official sources of rules which are regularly updated (daily or weekly).

The email-server can potentially take action depending upon the email spam-score (eg just delete it); however it is more usual to deliver the email to the end user anyway, and for the email client application to place the email in a “spam folder” if the spamassassin spam-score is above a certain threshold and the sender is not in the user’s address-book.

There are several ways to integrate spamassassin with Postfix, but the easiest way is to tell the smtpd server to use a specific service (defined in master.cf) as a “content filter”; content-filters are applications that accept an email and return a modified version of the input - which is exactly what spamassassin does. The service definition then just specifies that “spamc” should be started and the email written to it via STDIN (“pipe”).

spamassassin unix  -     n       n       -       -       pipe
  user=debian-spamd
  argv=/usr/bin/spamc -f
  -e /usr/sbin/sendmail -oi -f ${sender} ${recipient}

smtp      inet  n       -       y       -       5       smtpd
  -o content_filter=spamassassin

Param user indicates which user-account to start the app as.

Param argv = arguments to use when execing the service:

• /usr/bin/spamc = binary to execute (spamc)
• -e … = don’t write results to STDOUT but instead execute the specified app; everything after a “-e” belongs to the other program

Note that because the service-entry in master.mf is executed by queue-manager once for every queued email, per-email variables like $sender and $recipient can be used in the commandline of the executed application.

See the spamassassin spamc documentation for more information.

The spamassassin spamd application happens to be implemented in Perl; however that isn’t really relevant. The spamc application is native (fortunately, as it is started frequently on servers handling large volumes of email).