home : internet : mail : toaster : custom programs : maillogs multilog postprocessing

maillogs - A log file processer for mail server logs

Mail::Toaster is fairly complex, bundling together a wide variety of software programs, each that log in their own fashion. In order to get useful statistics out of all those logs, something was needed to process those logs. Maillogs is that something. Maillogs is designed to be used in conjunction wiith graphing utilities like MRTG or RRDtool. A fully functional working example is RRDutil. In order to work well with graphing apps, maillogs maintains ever increasing counters for email logs. It also as a postprocessor for multilog.

It handles log processing of the following types:

  • qmail mail delivery (send)
  • qmail-pop3d (pop3) (edit file to set pop3 daemon)
  • courier pop3d, pop3dssl (pop3)
  • courier imapd, imapdssl (imap)
  • smtp-auth
  • sqwebmail (webmail)
  • squirrelmail (webmail)
  • rbl blocks
  • spamasssassin
  • qmailscanner

The script and Mail::Toaster::Logs perl module is easily extendible as we dream up new things about our email system that we want to count and graph.

Maillogs does several things. First, it acts as a simple log postprocessor. By default it takes files that qmail outputs (/var/log/mail/[send|smtp|pop3]/current) and collects a bunch of counters about the logs and stores them in counter files.

  • For smtp logs we count denied RBL connections (rblsmtpd).
  • For qmail-send we log a slew of message send statistics.
  • For pop3, IMAP, webmail, and SMTP-AUTH, we count the number of connections and the number of successful logins.

Counters are written to files in /var/log/mail/counters/ by default. You can alter the locations of the logs files by editing the toaster.conf file included with this script. That file should be installed in /usr/local/etc.

what_am_i?

maillogs behaves differently depending on how it's called. It has (currently) four distinct personalities. Allow me to introduce you to multilog.

The multilog program (http://cr.yp.to/daemontools/multilog.html) has a poorly documented ability to run a postprocessor on logs after they reach a predetermined size (or are otherwise triggered).

This script is that postprocessor. If you rename this file to be named one of "smtplog", "pop3log", or "sendlog" it will become a log file processor for that qmail server protocol. See the maillogs FAQ for more details.

If called as maillogs, it will return ever incrementing counters for the protocol you entered. Run maillogs without any parameters to see the supported options. It's expected that you'll be calling maillogs from an SNMP agent allowing this solution to scale from single to clustered systems.

Design considerations

  • Counters will be polled via SNMP. Script must be able to return counts instantly, even when dealing with HUGE logs.
  • Must work with multilog and syslog logging formats
  • Outputs data in a format suitable for polling via SNMP
  • Simple configuration
  • Fail safe, errors must be noticed and reported but not fatal

sub-sections
multilog postprocessing
maillogs-FAQ
maillogs-changes

Last modified on 4/28/05.