Mail Alias/List Management
Author: Ed Korthof
1. Introduction
1.1 Scope
This document outlines a system to manage mailing aliases and
lists for Tigris. This is to be a separate project which is used within
Helm (and which can be used through other contexts).
The system should satisfy the following goals:
- Mail alias/list management should be centralized within a
single module.
- The system used to update the file system should not increase
the overall complexity of tigris any more than is necessary.
- The system must work with the current MTA (qmail) and MLM (ezmlm),
but as much as is reasonable, it should be possible to modify so
that other MTAs/MLMs are supported through use of the adapter pattern.
- Multiple domains must be supported.
- The system must scale reasonably.
1.2 Overview
1.2.1 Current System
To manage mailing lists, there is a body of Java code in
org.tigris.helm.admin.ProjMail* which relates to the user interface. This runs
within the servlet engine (JServ) and provides access to a class
(org.tigris.helm.tool.MailingList) which handles the core logic. It stores the
resulting information in an SQL database (MySQL) and then calls a Perl
CGI script, update.cgi, which makes updates to data files stored on
the file system.
User management is done via org.tigris.helm.admin.User* classes; these
store the users' e-mail addresses (for which we create aliases) in
MySQL. When update.cgi is called, it also recreates the aliases for
those users.
Domain management is not integrated into Tigris currently.
1.2.2 Proposed System
The Perl scripts responsible for updating the file system will be
replaced with a Java process on the mail server, accessed via RMI. This
server--using file system data--will return information about and make
changes to the current state. However, it will not contain core
business logic in this iteration.
The Java server accessible via RMI on the mail server (which may
be the same as the webserver) will use interfaces which are not specific
to ezmlm (thus allowing for easy swapping of the MLM). The interfaces
are defined in section 3.
The initial layer of Java code living within JServ/Helm will be
retained. However, the MailingList class will be changed
to MailingListSpec, with a number of differences behind
the API. It will use RMI to talk with a Java process on the mail server
(rather than talking with the MySQL database). In this implementation,
the communication may be "inefficient," in that many RMI calls will be
used -- the assumption is that the net speed will still be acceptable,
and the operation of creating or editing a mailing list will be fairly
infrequent.
The handling of user mail aliases will be explicitly broken
out. A new class will be created,
org.tigris.helm.tool.MailingAliasSpec,
which will deal with the mail server in a similar way to
MailingListSpec.
1.3 Terminology
The following is a list of terminology used in this document.
- anzu server
- the JVM process which lives on the mail server.
- anzu client
- any access of the anzu server -- either from a command line client or
from code living withing the webserver.
- mailing list
- an address which will send mail to zero or more addresses, where
significant customization of behavior is possible on a per-list basis,
and the addresses to which e-mail is sent are treated as separate
entities.
- mailing alias
- an address which forwards e-mail received to zero or more addresses.
No customization is possible on a per-alias basis, and if multiple
addresses recieve mail to this alias, they are not treated as separate
entities.
- base domain
- a domain which contains a set of projects, each with its own domain
name. for example, tigris.org is a base domain.
- project domain
- a domain which corresponds to a single project, and which exists
within the context of a base domain. for example, anzu.tigris.org is a
project domain.
- mailing address
- This includes both mailing lists and mailing aliases.
2. System Outline
2.1 User Interface/Helm serlvets
The user interface is part of Helm; it uses servlets in the
org.tigris.helm.admin package. It runs built within the Joist framework,
using Velocity templates. These servlets allow autorized users to
create/configure/destroy mailing lists, and to edit the list of
subscribers to each list. Security is implemented through the Joist
framework.
There is no a-priori reason that anzu needs to live within the Helm
or Joist frameworks -- it has been designed so that it should be possible
to set up mailing address management with other forms of authentication
and operation. anzu is primarily intended as a programmatic interface
for managing mailing address, but is designed so that
2.2 Servlet API for talking with the Mail Server
This will consist of classes within the org.tigris.helm.tool package,
which will communicate via RMI to a java process on the mail server
(possibly the same machine). These classes will be responsible for any
business logic which does not relate to MTA/MLM internals.
2.3 Mail Server
The code running in the mail server Java process will handle dealing
directly with the mail transfer agent and mailing list manager. It will
contain any business logic related to MTAs and MLMs -- both those which
are specific to qmail/ezmlm, and those which are general for tigris.
2.4 Security
The initial implementation will use a security mechanism where the
client and the server share a secret file. In order to do any operation
either reading from or writing to the disk, the client will request a
challenge. The server will generate it randomly and store it in memory;
the client must generated the MD5 signature of the challenge pluse the
secret; this response is valid for a single operation. Subsequent
operations will require additional challenge.
An alternate approach would be to use an SSL socket for the JNI
connection -- however, doing this requires a certain amount of certificate/
key management, so it is being left for a later date.
3. Detailed Design for Java Code
3.1 Interfaces
The following interfaces are defined:
MailingAddressManager, MailingListManager,
MailingList, MailingAlias. These extend
Remote and are used on the mail server.
3.1.1 AddressManager
Methods:
- public MailingListManager getMailingListManager()
- public String[] getListNames(String domain, String initialSubstring)
- public MailingList getList(String domain, String list)
- public MailingAlias getAlias(String domain, String alias)
- public void addDomain(String domain)
- public Logger getLogger()
- public File getMailDirectory(String domain)
3.1.2 MailingListManager
This interface is intended to allow specification of defaults which
will be handled in an MLM-specific way.
Methods:
- public void setDefaultOptions(String domain, String listType,
Hashtable options)
- public Hashtable getDefaultOptions(String domain, String listType)
- public void addDomain(String domain)
- public String[] getListNames(String domain,
String initialSubstring)
3.1.3 MailingList
This interface extends remote. Classes implementing it are used on the
mail server to represent actual lists.
Methods:
- public void update()
- public void create()
- public void delete()
- public String getMLType()
- public Hashtable getOptions()
- public void setOptions(Hashtable options)
- public void subscribe(String[] addresses, String group)
- public void unsubscribe(String[] addresses, String group)
- public String[] listMembers(String group)
It is possible to get a reference to a mailing list whether or not it
already exists. However, update will only work if it already exists, and
create will only work if it doesn't.
3.1.4 MailingAlias
This interface extends remote. Classes implementing it are used on
the mail server to represent mailing aliases which are not lists.
Methods:
- public String getDestination()
- public String setDestination(String)
- public void update()
- public void create()
- public void delete() throws RemoteException
Like a mailing list, getting a reference to a MailingAlias can succede
whether or the alias exists currently; the same comments wrt
update/create apply as well.
3.2 Classes
3.2.1 org.tigris.helm.mail.QmailEzmlmManager
This class implements MailingAddressManager.
3.2.1.1 Additional Methods
- public MailingAddressManager(String confFile)
- public File getEzmlmBinariesDirectory()
3.2.1.2 Implementation Details
When addDomain is called, this class will modify the appropriate qmail
control files. In the interests of scalability and managability, different
domains will be put into different directories -- so it's also necessary
to create a new directory.
The name of this directory is like the domain hierarchy,
except that the last two parts are kept together. for example, for
scarab.tigris.org, the directory would be: (mailDir)/tigris.org/scarab;
and for ui.scarab.tigris.org, the directory would be:
(mailDir)/tigris.org/scarab/ui.
Several files in the /var/qmail/control directory must also be
touched -- the following table shows those changes (note that
domain_subdir refers to the directory described above --
eg. (mailDir)/tigris.org/scarab/ui):
| file | line(s) added |
|---|
| /var/qmail/control/virtualdoamins |
(domain):tigris-(domain) |
| /var/qmail/users/assign |
+tigris-(domain)-:tigrisq:(tigrisq_uid):(tigrisq_gid):(domain_subir):-::
=tigris-(domain):tigrisq:(tigrisq_uid):(tigrisq_gid):(domain_subdir)::: |
3.2.2 org.tigris.helm.mail.EzmlmManager
This class implements MailingListManager. It provides a way to set
defaults for ezmlm lists.
3.2.2.1 Additional Methods
- public EzmlmManager(QmailEzmlmManager manager)
- public String getEzmlmBinariesDirectory()
- public Hashtable getOptions(EzmlmList list)
- public void setOptions(EzmlmList list, Hashtable options)
- public String[] associatedAliases()
- public boolean isAllowedList(String domain, String list)
- public boolean isAllowedAlias(String domain, String alias)
- public void setBaseDefaults(String domain)
3.2.2.2 Implementation Details
When addDomain is called, this class will create a directory under
the mail directory for that domain (as determined by the QmailEzmlmManager)
-- this directory will be called default-options, and it'll hold the
defaults for this domain.
Default values for a domain will be stored in
(mailDir)/(domain)/default-options. Note that options for a
given domain will be merged -- that is, the options for
scarab.tigris.org will be based on the information in
(mailDir)/default, overriden by the values in
(mailDir)/tigris.org/default, overriden by the values in
(mailDir)/tigris.org/scarab/default.
As a consequence of the override behavior, calling setDefaultOptions
with the Hashtable returned by getDefaultOptions will probably result in
changes to the file system -- the defaults from
(mailDir)/default and (mailDir)/tigris.org/default
will be saved in (mailDir)/tigris.org/scarab/default, which
means that changes to the defaults in one of the parent directories will
not change the defaults in the child directory, or for mailing
lists created in the child domain.
The setOptions/getOptions functions are provided so that
EzmlmLists can read and store their options using the same code as that
used for the default option lists.
default-options will be specifically excluded as a mailing
list name.
associatedAliases will return the aliases which may
be associated with a mailing list. That will include the following,
and any additional aliases which come up:
-accept-default
-reject-default
-return-default
-owner
-digest
-digest-owner
-digest-return-default
-default
isAllowedAlias will disallow any alias which is of
the form of an alias which could be associated with a mailing list,
or which matches an existing mailing list.
3.2.3 org.tigris.helm.mail.EzmlmList
This class implements MailingList. The logic it contains is for
calling the appropriate ezmlm programs and changing the behavior of
ezmlm lists (via the control files and ezmlm-make.
3.2.3.1 Additional Methods
- public EzmlmList (String domain, String list AddressManager manager)
throws AddressInUseException
3.2.3.2 Implementation Details
That this class will generate the literal command line arguments to
ezmlm-make. It relies on EzmlmManager to store and retrieve the
options.
To implement delete for ezmlm lists, we will delete the associated
directory, the (list) alias itself, and any aliases
generated from (list) plus one of the suffixes returned by
EzmlmManager.associatedAliases().
isList distinguishes ezmlm lists by the fact that the
list will have a corresponding directory.
For ezmlm, group must be one of "Subscribers", "Digest Subscribers", and
"Moderators".
3.2.4 org.tigris.helm.mail.QmailAlias
This class implements MailingAlias.
3.2.4.1 Methods
- public MailingAlias(File mailDirectory, String domain, String alias,
String destination) throws AddressInUseException, IOException
- public MailingAlias(File mailDirectory, String domain, String alias)
throws AddressNotFoundException, IOException
3.2.4.2 Implementation Details
This class looks at the .qmail-* files in (mailDir) (as
determined by the MailingAddressManager. Note that it will
not properly recognize all addresses associated with an ezmlm mailing
list: EzmlmMailingList should be used for that.
3.2.5 org.tigris.helm.mail.Logger
This class is a simple logger, designed so that better logging can
be added at a later date.
3.2.5.1 Methods
- public Logger()
- public void log(String message)
- public void log(Exception e)
- public void log(String message, Exception e)
- public void debug(String message, int level)
- public void setDebugFlag(int level)
3.2.5.2 Variables
- public static final DEBUG_LEVEL_NONE
- public static final DEBUG_LEVEL_WARN
- public static final DEBUG_LEVEL_INFO
3.2.6 org.tigris.helm.mail.ListOption
This abstract class represents a generic mailing list option.
currently, that means either a Boolean or a String option.
3.2.6.1 Methods
- public static ListOption parseOption(String option)
- abstract public Object getValue()
- abstract public int optionType()
3.2.6.2 Class variables
- public static final int STRING
- public static final int BOOLEAN
3.2.6.3 Implementation Details
The parseOption function is designed to take a String of the form:
{option_type}:{name}:{value}. The {option_type} portion is used to choose a
concrete class (either String or Boolean); and the {value} portion is
based along to that class' constructor.
3.2.7 org.tigris.helm.mail.BooleanOption
3.2.7.1 Methods
- public Boolean getValue()
- public BooleanOption(String name, String value)
- public BooleanOption(String name, boolean value)
- public int optionType()
3.2.8 org.tigris.helm.mail.StringOption
3.2.8.1 Methods
- public String getValue()
- public StringOption(String name, String value)
- public int optionType()
- public void setValue(String value)
3.2.8.2 Implementation Details
In some cases, special logic must be used to as to properly handle
the idea of removing an option. (For example, ezmlm will sometimes
operate differently if a file exists, even if the contents are empty.)
For this reason, an empty or null string will generally indicate that
this option is disabled. StringOption must not be used for options
for which an empty string is a meaningfull value (since it will
deactivate the option).
3.2.9 org.tigris.helm.tool.MailingAliasSpec
This class will be within the serlvet engine (not the mail server's
Java process). It represents the information required to create a mail
alias.
3.2.9.1 Methods:
- public MailingAliasSpec(String domain, String alias,
String destination) throws HandlerException
- public static MailingAliasSpec lookup(String domain, String alias)
throws AddressNotFound, HandlerException
- public void create () throws HandlerException, AddressInUse
- public void update (String destination) throws HandlerException
- public void delete () throws HandlerException
3.2.9.2 Implementation Details
The static lookup function will get a
reference to a MailingAddressManager (via
java.rmi.Naming.Lookup, using
a system property for the location of the resource to lookup). This
class will then use that to create or find a MailingAlias, and operate
on it. Note that the constructor does not actually create the alias --
create must be called explicitly. (This is for symetry
with MailingListSpec.)
Because mailing aliases are very similar from one MTA to another,
there is no need to make this class specific to qmail.
3.2.10 org.tigris.helm.tool.MailingListSpec
This class also will be used by the servlet engine (not the mail
server's Java process), and will be used in a similar manner.
3.2.10.1 Methods
- public MailingListSpec(String domain, String list)
- public static MailingListSpec lookup (String domain, String list)
throws AddressNotFound, HandlerException
- public void setOption(String option, Boolean value)
throws InvalidOption
- public Hashtable getOptions() throws InvalidOption
- public void setOptions(Hashtable options) throws InvalidOption
- public String[] getGroups() throws HandlerException
- public void create() throws HandlerException
- public void update() throws HandlerException
- public void delete() throws HandlerException
3.2.10.2 Implementation Details
As with MailingAliasSpec, the constructor will not actually create
the list: create() must be called.
The commands for setting options will (in this iteration) be
proxied out to the MailingList on the mail server. The options are
intended to be reasonably MLM-independent values; a value which
is not supported by the MLM in use will result in an exception.
3.3 Exceptions
Generally, classes will throw RemoteException with an appropriate
error message. Where it is possible to have more intelligent behavior
based on the nature of the exception, a different class will be used.
At the least, that will include the following classes:
public class AddressInUseException extends RemoteException
public class AddressNotFoundException extends RemoteException
public class DomainInUseException extends RemoteException
public class InvalidDomainException extends RemoteException
public class InvalidGroupException extends RemoteException
public class InvalidFileException extends RemoteException
public class InvalidOptionException extends RemoteException
4 anzu Server Detailed Design
This section outlines several things:
- the structure for the data files
- the locations and directory structure for data files
- the locations and directory structure for control and configuration files
- the process whereby we will get qmail to deliver mail to the appropraite
mailing address
Although a database would be an effective means for storing much of
this information, it would add an additional requirement for the mail
server. Since a database is not a requirement (merely a convience), this
software will be built so that a database isn't currently used. At some
point, it would be useful to abstract the data handling layer so that it
would be possible to use a database at this point.
4.1 Data File Structure
The following types of data files are used.
- Options Definitions
- Options Files
- Mailing Addresses
4.1.1 Options Definitions
The options definition is a properties file. It has one special
value: options.list; the contents of that value are a space separated
list of the option names. For each option name, the following
properties should exist:
| property name | effect |
|---|
| name | matches with the names used in the options
files |
| type | either string or boolean -- determines the type
of this option |
| displayType | this suggests when to show this option in
the user interface (may be ignored) |
| renderType | one of the following: option,
file, custom, this determines how this option should
affect the list. MLM specific interpritation is expected. |
| renderName | used to determine how this option should be
used in creating or modifying the list. MLM specific interpritation
is expected; for qmail, the values are either the name of the
option, or the name of the file, or (FIXME: TBD) |
| defaultValue | the value to use if no option is defined
for this list or list type. |
| description | a suggested description to present in the
user interface. |
4.1.2 Options File
These files contain option values -- either the defaults for a given
domain or the values for a particular list. These are Java properties
files, and there is one such for each mailing list or type of default
mailing list.
The structure of these files is simple -- the property names are the
same as the option names (from the option types file). The values must
be valid for the options which the represent.
It would be possible to combine these all into a few files, but there
are functional advantages to using a separate file for each one: it
means that the files can be saved to disk after each modification
without worrying about synchronization. In a perfect world, the JVM
would never die unexpectedly and could always save state to disk as part
of the shutdown; but that's not always possible in practice.
4.1.3 Mailing Addresses
The format for mailing addresses will be the standard which qmail uses;
this is the same standard used by sendmail for .forward files.
4.2 Data File Locations
The following classes of data files need to be handled:
- anzu server wide data: the list of valid domains, secret file
- domain wide defaults for mailing list options (editable)
- individual mailing list options
- aliases and mailing list information
The default base directory for data files is /var/anzu, but
it is configurable. It will be refered to below as
{data_dir}.
4.2.1 anzu Server Wide Defaults
The directory containing these files is
{data_dir}/anzu_server/. domains.txt contains the list
of domains, and secret_key contains the secret file.
4.2.2 Domain Wide Defaults
The following are the domain wide defaults:
- mailing list types and corresponding options
- generic e-mail aliases (root, webmaster, etc.)
The list types and default options will go in
{data_dir}/list_types/{domain}/ --
each list type will have a corresponding file, and the set of all
files in this directory will be the set of list types available.
For the structure of these files, see
The generic aliases will go in
{data_dir}/generic_aliases/{domain}/ --
each file will be treated as an alias file, and the set of all files
in thisdirectory will be the set of all generic aliases.
4.2.3 Individual mailing list options
The structure of these files will be the same as for the domain wide
defaults. The directories used will be of the form
{data_dir}/lists/{domain}/ -- the file names in
those directories will correspond to the mailing list names.
4.3 Control File Locations
The default base directory for control files is /etc/anzu,
but it is configurable. It will be refered to below as
{control_dir}. Currently, the only control file is the options
type file; it will be: {control_dir}/option-types.
TODO: add configuration file?
4.4 Mail Delivery Walkthrough
This design is intended with several goals. First, it should
minimize the security risks and impact to a qmail installation. Second,
it should provide sufficient flexibility that we could change to another
MTA without undue difficulty.
The implementation choice is a perl script which recieves all e-mail
sent to any of the domains
5 Future Additions
- automatic subscription to lists according to some criteria
(cvs@*.domain, eg)
- support list merging and renaming (at least subscribers, hopefully
archives or as much as is possible).
