Logmonster

Web Server Log Processing Utility Author: Matt Simerson

Frequently Asked Questions

Why did you write Logmonster?

Typical Scenario: You have a web server that serves your domain. You write a simple script to restart apache each night and pipe the logs off to your analyzer. It works.

ISP/Hosting Scenario: Each server hosts many domains. You may also have load balanced servers (multiple machines) serving each domain. A tool like this is necessary to:

collect all the log files from each server
split the logs based on the virtual host(s)
discard invalid entries (phishing, referrer spam, etc)
sort them into chronological order
feed logs into analyzer
do something with the raw logs (compress, save to vhost dir, etc)

Why use cronolog?

Read the Apache docs and all the caveats required to rotate logs, including restarting the server at the right time. Factor that into using servers in different time zones and you will find it much more reliable to use cronolog. I have used cronolog for years and never had an issue.

Why not use one file per vhost so you don't have to split them?

A problem is that you end up with lots of open file descriptors (one per vhost). That only scales so far. There is still the problem of collecting log files from multiple servers and sorting them before feeding them into the log processor. The logs are easier to manage if they all start in one place.

What is the recommended way to implement logmonster?

Adjust your web server log settings as shown in Logmonster Install
If you aren't already using cronolog, start. Wait a day.
Test by running "logmonster -i day -n -v"

It will tell you what it is doing and everything should look reasonable. Correct anything you do not like (creating $statsdir for domains that should have it, etc) and then create a cron entry or periodic script. Read the output from logmonster in your mailbox. When you're confident everything is tuned, adjust crontab and add a "-q" to it so it only emails if there are errors.

How do I enable log processing for a virtual domain?

Create the directory ("stats" by default) within the vhost's DocumentRoot.

For example, the docroot for example.com is /home/example.com/html. To enable virtual host processing, create the directory /home/example.com/html/stats. Their statistics will be processed.

How do I process my logs hourly?

Set cronolog to "%Y/%m/%d/%H"
run logmonster with -i hour
adjust the cron entry to run every hour.

If you use webalizer, get acquainted with webalizer -p and its limits.

How do I use the back feature?

Imagine you shut your server down at 0:55 last night to do some system maintenance. You brought it back up at at 1:05 (10 minutes later) but your cron job that runs logmonster at 1:00am did not run. The solution is to run logmonster manually.

Now, suppose you made an err that caused logmonster to not run for the last week. You return from vacation and notice the errors in your mailbox, because that B<is> where you configured cron stuff to go, right? Now you set about to fix the problem.

The way to process old logs with logmonster is to use the -b option. In our example, we would run "logmonster -i day -b7". Logmonster will confirm the date with you and then dutifully process the logs from 7 days ago. Then run again with "-d -b6", etc until you are current.

What assumptions do logmonster.pl make?

You use cronolog
You have enough memory to fit your largest zones log file into RAM
You have the following Perl modules installed

Most systems have all but Compress::Zlib and Params::Validate installed

FileHandle
POSIX
Date::Format
File::Copy
File::Path
Date::Parse
Compress::Zlib
Params::Validate

Your logs are set up properly. See "Apache Logs"
The time on your web servers is synchronized (think NTP)
You use webalizer, http-analyze, or AWstats for log processing

What should I set vhost to?

vhost should be either a file with all your directives listed (ie, httpd.conf) or a directory (my favorite way) that contains files, each containing the VirtualHost and related directives for that Apache vhost. This is from the configuration file:

##
# vhost -  This is where Logmonster learns about your Apache vhosts. If
#          you list them in your httpd.conf, then this should be set to
#          the full path of your httpd.conf file. 
#
#              vhost = /usr/local/etc/apache/httpd.conf
#
#          If you use a include directory for your vhosts, then this
#          should be the full path to that directory.
#
#              vhost = /usr/local/etc/apache/vhosts
#
#vhost     = /etc/httpd/vhosts               # darwin
#vhost     = /var/www/vhosts                 # linux
#vhost     = /usr/local/etc/apache2/Includes  # freebsd
#
vhost     = /usr/local/etc/apache/Includes

Can I use this with web servers other than Apache?

Absolutely. Set up a configuration file with your vhost information in it and point logmonster at it. The format for each vhost is as follows:

   <VirtualHost>
     ServerName www.tnpi.net
     ServerAlias www.thenetworkpeople.net *.tnpi.net
     DocumentRoot /home/tnpi.net/html
   </VirtualHost>

Create as many vhost directives as you need and logmonster will parse them all. When you make changes to your web server, update this file as well.

All the other rules apply equally. You will want to use Apache's ELF (Extended Log Format) with the virtual hostname appended to the logs and pipe the logs to cronolog for reasons mentioned elsewhere.

Cronolog and selinux are not playing nicely

I just finished installing cronolog on a selinux system (CentOS) with the sestatus set to enforcing. There were problems getting the permissions correct so that cronolog would be allowed to create files and dirs. I added this to solve the problem.

CentOS and RHEL4 and other RH clones use the file: /etc/selinux/targeted/contexts/files/file_contexts to store context info for files and dirs. In the above file add the location that you want logs to be written in if different than the standard /var/log/httpd like so:

/var/log/apache(/.*)?   system_u:object_r:httpd_log_t

This line would allow cronolog to create and write files in the new location. Hope this saves someone else the trouble. Another way to do this would be to use the command line chcon facility like so:

chcon -R -h -t httpd_log_t /var/log/apache

I have not rebooted my server or tested to see that on a system reset the chcon settings survive but I doubt it.

I hope this info saves someone else the trouble of looking it up and diagnosing a problem. -- Lewis Bergman

How do my logs need to be set up?

The default version of Apache's ELF format is quite good. However, on a system with many virtual hosts, determining which vhost a particular entry is for can be difficult. I wrote a parser that was about 98% effective. However, there is a better way.

Many web servers support adding the vhost name to each log entry. This is 100% effective and makes it quite easy to determine which vhost a log entry was served for. See the Logmonster Install page for how to configure your web servers logs.

Logmonster FAQ

Contents