There are a number of mailing lists served off of the mailing list server Eqinfo.
Eqinfo is a set of three machines. One is a K6-2 PC, another is a Pentium 2, and the third is a Pentium 100. The name eqinfo.wr.usgs.gov is actually a set of MX records that resolve to three different IP addresses. One machine in Menlo Park and one in Pasadena are listed at equal priority, while the old Pentium here in Pasadena is listed as a backup, so mail will only go through it if both the higher-priority machines are unreachable. All are running FreeBSD. I have a set of notes about setting up these machines.
The mailing lists are run with ezmlm and Qmail. Qmail is an open-source replacement for Sendmail, optimized for speed and security. Ezmlm is a mailing list manager that is optimized for handling large lists, and also for ease of use. The lists were originally set up using Majordomo, but we switched to ezmlm in order to get its nice subscription mechanism and bounce handling.
The files for Qmail are stored under /var/qmail. The alias directory, which contains system-wide aliases is stored in /home/alias. Aliases are handled with .qmail files, which are analagous to Sendmail's .forward files. System-wide aliases are kept in this directory.
One important feature of Qmail is parallel delivery. Where Sendmail
delivers mail one piece at a time, Qmail will fork off many delivery
agents to work in parallel, which makes for faster deliver times. The
maximum number of concurrent processes is set in the file
# cd /usr/ports/mail/qmail # setenv WITH_BIG_CONCURRENCY_PATCH "" # makeWhenever this value is changed, qmail must be restarted:
eqinfo# ps -auwwx | grep qmail
qmails 79160 0.0 0.6 840 380 p0- I Tue07AM 0:43.04 qmail-send
qmaill 79161 0.0 0.4 780 272 p0- I Tue07AM 0:15.47 multilog t s200000 n15 /var/log/qmail
root 79162 0.0 0.4 784 252 p0- I Tue07AM 0:01.09 qmail-lspawn |preline -f /usr/libexec/mail.local -r "${SENDER:-MAILER-DAEMON}" -d "$USER"
qmailr 79163 0.0 1.7 1552 1084 p0- I Tue07AM 0:09.12 qmail-rspawn
qmailq 79164 0.0 0.5 772 276 p0- I Tue07AM 0:05.98 qmail-clean
eqinfo# kill -TERM 79160
eqinfo# /usr/local/etc/rc.d/qmail.sh start
The mailing lists are all owned by the listmgr account, and are stored in directories under /home/listmgr. The mailing list addresses are mapped to the listmgr account in the file /var/qmail/users/assign. The format of this file is:
+quake-all-:listmgr:5000:10:/home/listmgr/quake-all:-:: =quake-all:listmgr:5000:10:/home/listmgr/quake-all:::This assigns the alias quake-all to the listmgr account's quake-all directory, where there is a .qmail file to handle it. All other addresses associated with this list, like quake-all-subscribe and others, are assigned to other .qmail files in the quake-all directory. The assign file is built into the cdb database with the command:
/var/qmail/bin/qmail-newu
There are a set of shell scripts under this account to aid in managing the mailing list. Typing 'help' in the listmgr account will give a list of the available commands.
This is the listmgr account on eqinfo. There are a number of shell script
available to help you with administering the mailing lists:
activity: Shows subscribe/unsubscribe activity for the current day,
or last 24 hours.
listsubs: Lists the subscribers to a list.
adduser: Adds a subscriber to a list.
removeuser: Removes an address from a mailing list.
listposters: Lists the addresses allowed to post to a list.
addposter: Adds an address to the list of addresses allowed to post to a list.
countlist: Counts the subscribers to a list.
makelist: Creates a new mailing list.
Each of these scripts will give a usage message if it is invoked without
arguments.
Each machine maintains its own independent copy of each mailing list. Since subscription requests can end up on any one of the three, there is a mechanism for maintaining the lists. There is a cron job on each machine that runs from the listmgr account:
*/15 * * * * /home/listmgr/bin/listsync.pl > /dev/nullThis checks the log file for each mailing list and propagates any changes to the other servers. This is done on each server.
Additionally, there is a job that checks the lists on each server against each other to make sure that they are all the same. This runs once an hour:
4 * * * * /home/listmgr/bin/integrity3.pl
The handling of incoming mail for each list is specified by the .qmail file in the list directory. There were a few changes to this file as compared to a standard mailing list. The first is the addition of the line:
|/usr/local/bin/ezmlm-issubn /home/listmgr/quake-all/list/allow || bouncesaying 'Not allowed'This only allows posting of mail from the addresses listed in the list/allow directory. Anyone else who tries to send mail to the list will get a bounce with a 'not allowed' message. To add an address to the allow directory, use the addposter command.
The second change is the addition of "-T" to the second line:
|/usr/local/bin/ezmlm-reject -T '/home/listmgr/quake-all/list'This allows messages posted to list to be sent to eqinfo.wr.usgs.gov or to eqinfo.gps.caltech.edu.
The last line in editor is a call to ezmlm-warn. This handles bounces. The timeout value by default is about 11 days. Since earthquake messages can be weeks or months apart, this timeout has been set higher for our mailing lists. For quake-all it is set to 30 days, and for quake-large and shakemail it is 180 days.
|/usr/local/bin/ezmlm-warn -t 180 '/home/listmgr/shakemail/list' || exit 0
Subscriptions are open to all lists except for office and felt. Subscriptions to these lists must be approved by the list moderator. This is set up with the ezmlm-sub program:
ezmlm-sub /home/listmgr/felt/list/mod stan@iron.gps.caltech.eduThis creates the moderator database in the felt/list/mod directory. Creating the felt/list/modsub directory enables subscription moderation. Anyone who tries to subscribe to the list will get the standard confirmation message, but after they confirm, a second confirmation message is sent to the moderator for approval.
There is a message trailer file in list/text/trailer. This is appended to all outgoing messages:
-- You are receiving this message through the <#l#> mailing list. To subscribe, go to http://pasadena.wr.usgs.gov/mailing-lists.html To unsubscribe, send mail to <#l#>-unsubscribe@<#h#>Note that <#l#> and <#h#> are replaced be ezmlm with the mailing list name and host name. The trailer file is the same for all the mailing lists.
The text that is sent to users by ezmlm is all stored in files in the list/text directory. In particular, the subscription confirmation text is stored in sub-confirm. Any of these files can be modified by the listmgr account.
The listmgr account runs the activity command every night at midnight to generate a report on subscription and unsubscription activity for the previous day. Here is a sample report.
There are also graphs of server activity available at http://bort.gps.caltech.edu/mrtg/eqinfo-rrdtest.html. There is small shell script that gets the mail statistics to report back to MRTG. This is /home/mrtg/qmail-mrtg/qmail-mrtg-mess, and is run by cron at five-minute intervals. The results are mailed to an alias on Bort that collects the numbers and feeds them to MRTG.
In the interests of tracking how much work the server has to do, it reports the number of list subscriptions back to MRTG. This is done with a shell script running as a cron job on Eqinfo. This is the crontab:
0 * * * * /home/listmgr/bin/countit.sh cube 1 * * * * /home/listmgr/bin/countit.sh quake-all 2 * * * * /home/listmgr/bin/countit.sh quake-large 3 * * * * /home/listmgr/bin/countit.sh shakemailThis gets the count for each list and mails it back to Bort. There, there are aliases to receive the counts:
mrtg-quake-all: "| /home/mrtg/quake-all.sh" mrtg-quake-large: "| /home/mrtg/quake-large.sh" mrtg-shakemail: "| /home/mrtg/shakemail.sh" mrtg-cube: "| /home/mrtg/cube.sh"These shell scripts look like:
bort# cat quake-all.sh
#!/bin/sh
# Take the message and strip off the headers.
sed '1,/^$/d' > /home/mrtg/quake-all-count.txt
The final result is a text file with a single number in it. MRTG runs
every hour, picks up these files, and regenerates the graphs. The
crontab line for this run is:
5 * * * * /usr/local/bin/mrtg /usr/local/etc/mrtg/mrtg3.cfgThe graphs can be seen at http://bort.gps.caltech.edu/cgi-bin/eqinfo-lists.pl.
Maintained by Stan Schwarz
Honeywell Technical Services
Pasadena, California
Last updated 08/21/2002
Back to System Administration
Back to Main