multilog postprocessing home : internet : mail : toaster : custom programs : maillogs : maillogs-FAQ maillogs-changes

What can maillogs do?

It's primary purpose is to Maintain and report counters for email related log files.

It can also maintains log archives (organized by date) by piping the logs through cronolog. They are stored in $maillogs/yyyy/mm/dd/????log. This is the default because it's very easy for programs and persons to work with a days worth of log files.

Maillogs has other fun reporting features as well. See "maillogs yesterday" for an example.

When you install Mail::Toaster, its installed to /usr/local/sbin/maillogs. That's where RRDUtil (via net-snmp) expects to find it when it wants to poll for the current mail counters.

To manually view the counters, simply run "maillogs protocol". Run maillogs without a protocol to see of list of options.

Log files more than 24 hours old are automatically compressed.

What does the output look like?

You'll get back output that looks like this:

  • # maillogs smtp
  • connect:7:connect_last:7:smtp_block_count:3514773:
  • smtp_block_dns:21386:smtp_block_dsbl:11626:
  • smtp_block_maps:11:smtp_block_ordb:3064:
  • smtp_block_other:3256547:smtp_block_spamcop:32772:
  • smtp_block_spamhaus:189367:
    • maillogs webmail
  • success:7:success_last:7
    • maillogs send
  • concurrencyremote:65.2743691384389:delivery:715748:
  • delivery_deferral:222113:delivery_failure:7332:
  • delivery_success:486303:message_bounce:2477:
  • message_bytes:546858953:message_end:45442:
  • message_info:45720:message_new:45720:other:298:
  • start_delivery:715771:start_delivery_local:42054:
  • start_delivery_remote:673717:status:1431575:
  • status_localp:14273.8000000053:status_remotep:934451
    • maillogs pop3
  • pop3_connect:8951:pop3_ssl_connect:0:
  • pop3_success:8951:pop3_ssl_success:0
    • maillogs imap
  • connect_imap:2488:connect_imap_ssl:2488:
  • imap_connect_success:817:imap_ssl_success:845

This format is very nice for feeding into SNMP stats collectors like MRTG and my own RRDutil.

multilog postprocessor usage

To use as a postprocessor, you need to install maillogs in your mail log directory as directed in the multilog man page (http://cr.yp.to/daemontools/multilog.html). If you are using it with my mail toaster, this will do:

  • toaster_setup.pl -s maillogs

In order to actually use the script now, you must be logging to multilog, and your log/run files must have the post-processor statement in them. My smtp log/run looks like this:

  • #!/bin/sh
  • PATH=/var/qmail/bin:/usr/local/bin:/usr/bin:/bin
  • export PATH
  • exec setuidgid qmaill multilog !./smtplog s100000 /var/log/mail/smtp

The maillogs script alters it's behavior based on how it's called. When it's called as smtplog, it expects qmail's SMTP log format (technically, rblsmtpd), and produces counters based on that. The counter file it produces is (by default) /var/log/mail/counters/smtp_rbl.txt

Why are counters ever increasing?

Well, they aren't, but they are. :) RRDtool, MRTG, and apps like it expect that counters will increase like an odometer on your car, constantly growing unti they reach a specific threshhold such as 999,999 on your car. With RRDtool, numbers are supposed to increase until they reach a 32 or 64 bit number.

Your log files (think of /var/log/maillog) are counters and behave normally until syslog decides to rotate them. How to deal with a counter that shrinks before reaching such a threshhold is not something those applications can properly take into account and the assumptions they make aren't correct for this case. To work around that (and prevent a HUGE spike in the graph) I maintain a last count variable for syslog results and do some math on the new count so that the result is ever increasing counters.

Do you support isoqlog?

Yes. One caveat, the HTML output directory (as defined in isoqlog.conf must be owned and writable by the user which is set up in your supervise/send/log/run file. On most FreeBSD qmail systems, that will be user qmaill and group qnofiles. Otherwise isoqlog refuses to write to it. You can set it up like this (adjust paths as necessary):

  • chown -R qmaill:qnofiles /usr/local/www/data/isoqlog
  • chmod -R 755 /usr/local/www/data/isoqlog

Maillogs will detect if isoqlog is installed and every time maillogs rotates your qmail-send logs, it'll trigger isoqlog to process your log files. I've noticed that isoqlog assumes that the contents of your log directory are the entire days logs. Since my logs roll every 5 minutes (because I collect stats for RRDUtil) I have set multilog to save 288 files (the number of 5 minute periods in a day). That has worked quite well.

What assumptions do you make?

That you have installed the Mail::Toaster perl modules and configured toaster.conf. See the documentation for toaster.conf for details of how those settings affect your system.

I get errors when I run it. Is something wrong?

It's normal to get a few errors about files missing or not readable the first time you run maillogs. That's because the counter files it's looking for don't exist yet. Maillogs will create them for you (assuming it has permission to do so).

If you get errors after running it a couple times, then pay attention to the errors because they are surely telling you something.

Last modified on 4/28/05.