/ 
Apache-Logmonster-2.75
River stage zero No dependents
/ Apache::Logmonster

NAME

Apache::Logmonster

SYNOPSIS

Processor for Apache logs

DESCRIPTION

A tool to collect log files from multiple Apache web servers, split them based on the virtual host, sort the logs into cronological order, and then pipe them into a log file analyzer of your choice (webalizer, http-analyze, AWstats, etc).

FEATURES

Log Retrieval from one or multiple hosts
Ouputs to webalizer, http-analyze, and AWstats.
Automatic configuration by reading Apache config files.
Outputs stats into each virtual domains stats dir, if that directory exists. (HINT: Easy way to enable or disable stats for a virtual host).
Efficient: uses Compress::Zlib to read directly from .gz files to minimize disk use. Skips processing logs for vhosts with no $statsdir. Doesn't sort if you only have logs from one host.
Flexible: you can run it monthly, daily, or hourly
Reporting: saves an on disk activity report and an email friendly report.
Reliable: lots of error checking so if something goes wrong, it'll give you a useful error message.
Understands and correctly deals with server aliases

INSTALLATION

Step 1 - Download and install (it's FREE!)

http://www.tnpi.biz/store/product_info.php?cPath=2&products_id=40

Install like every other perl module:

perl Makefile.PL
make test
make install

To install the config file use "make conf" or "make newconf". newconf will overwrite any existing config file, so use it only for new installs.

Step 2 - Edit logmonster.conf
vi /usr/local/etc/logmonster.conf
Step 3 - Edit httpd.conf

Adjust the CustomLog and ErrorLog definitions. We make two changes, adding %v (the vhost name) to the CustomLog and adding cronolog to automatically rotate the log files.

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %v" combined
CustomLog "| /usr/local/sbin/cronolog /var/log/apache/%Y/%m/%d/access.log" combined
ErrorLog "| /usr/local/sbin/cronolog /var/log/apache/%Y/%m/%d/error.log"
Step 4 - Test manually, then add to cron.
crontab -u root -e
5 1 * * * /usr/local/sbin/logmonster -d
Step 5 - Read the FAQ

http://www.tnpi.biz/internet/www/logmonster/faq.shtml

Step 6 - Enjoy

Allow Logmonster to make your life easier by handling your log processing. Enjoy the daily summary emails, and then express your gratitude by making a small donation to support future development efforts.

DEPENDENCIES

Compress::Zlib
Date::Parse (TimeDate)

report_hits

report_hits reads a days log file and reports the results back to standard out. The logfile contains key/value pairs like so:

matt.simerson:4054
www.tnpi.biz:15381
www.nictool.com:895

This file is read by logmonster when called in -r (report) mode and is expected to be called via an SNMP agent.

report_open

In addition to emailing you a copy of the report, Logmonster leaves behind a copy in the log directory.

check_stats_dir

Each domain on your web server is expected to have a "stats" dir. I name mine "stats" and locate in their DocumentRoot, owned by root so that the user doesn't delete it. This sub first goes through the list of files in (by default) /var/log/apache/tmp/doms, which is a file for each vhost. The file name matches the vhost name the contents are the log entries that correspond to that vhost.

If the file is zero bytes, it deletes it as there is nothing to do.

Otherwise, it gathers the vhost name from the file and checks the %domains hash to see if a directory path exists for that vhost. If no hash entry is found or the entry is not a directory, then we declare the hits unmatched and discard them.

For log files with entries, we check inside the docroot for a stats directory. If no stats directory exists, then we discard those entries as well.

feed_the_machine

feed_the_machine takes the sorted vhost logs and feeds them into the stats processor that you chose.

sort_vhost_logs

At this point, we'll have collected the Apache logs from each web server and split them up based on which vhost they were served for. However, our stats processors (most of them) require the logs to be sorted in cronological date order. So, we open up each vhosts logs for the day, read them into a hash, sort them based on their log entry date, and then write them back out.

split_logs_to_vhosts

After collecting the log files from each server in the cluster, we need to split them up based upon the vhost they were intended for. This sub does that.

AUTHOR

Matt Simerson <matt@tnpi.biz>

BUGS

None known. Report any to author.

TODO

Support for analog.

Support for individual webalizer.conf file for each domain

Delete log files older than X days/month

Do something with error logs (other than just compress)

If files to process are larger than 10MB, find a nicer way to sort them rather than reading them all into a hash. Now I create two hashes, one with data and one with dates. I sort the date hash, and using those sorted hash keys, output the data hash to a sorted file. This is necessary as wusage and http-analyze require logs to be fed in chronological order. Take a look at awstats logresolvemerge as a possibility.

SEE ALSO

http://www.tnpi.biz/internet/www/logmonster

COPYRIGHT

Copyright (c) 2003-2004, The Network People, Inc. All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

Neither the name of the The Network People, Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DIS CLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.