NAME
Log::Unrotate - Incremental log reader with a transparent rotation handling
VERSION
version 1.30
SYNOPSIS
use Log::Unrotate;
my $reader = Log::Unrotate->new({
log => 'xxx.log',
pos => 'xxx.pos',
});
my $line = $reader->read();
my $another_line = $reader->read();
$reader->commit(); # serialize the position on disk into 'pos' file
my $position = $reader->position();
$reader->read();
$reader->read();
$reader->commit($position); # rollback the last 2 reads
my $lag = $reader->lag();
DESCRIPTION
Log::Unrotate
allows you to read any log file incrementally and transparently.
Incrementally means that you can store store the reading position to the special file ("pos-file") using commit()
, restart the process, and then continue from where you left.
Transparently means that Log::Unrotate
automatically jumps from one log to the next. For example, if you were reading foo.log, then stored the position and left for a day, and then while you were away, foo.log got renamed to foo.log.1, while the new foo.log got some new content in it, Log::Unrotate
will find the right log and give you the remaining lines from foo.log.1 before moving to foo.log.
Even better, it will do the right thing even if the log rotation happens while you were reading the log.
Log::Unrotate
tries really hard to never skip any data from logs. If it's not sure about what to do, it throws an exception. This is an extremely rare situation, and it is a good default for building a simple and robust message queue on top of this class, but if you prefer a quick-and-dirty recovering, you can enable autofix_cursor option.
METHODS
- new($params)
-
Creates the new unrotate object.
- pos
-
Name of a file to store log reading position. Will be created automatically if missing.
Value '-' means not to use a position file. I.e., pretend it doesn't exist at the start and ignore commit calls.
- cursor
-
Instead of
pos
file, you can specify any custom cursor. SeeLog::Unrotate::Cursor
for the cursor API details. - autofix_cursor
-
Recreate a cursor if it's broken.
Warning will be printed on recovery.
- rollback_period
-
Time period in seconds. If rollback_period is greater than 0,
commit
method will save some positions history (at least one previous position older then rollback_period would be preserved) to allow recovery when the last position is broken for some reason. (Position may sometimes become invalid because of the host's hard reboot.)The feature is enabled by default (with value 300), set to 0 to disable, or set to some greater value if your heavily-loaded host is not flushing its filesystem buffers on disk this often.
- log
-
Name of a log file. Value
-
means standard input stream. - start
-
Describes the initialization behavior of new cursors. Allowed values:
begin
(default),end
,first
.When start is
begin
, we'll read current log from the beginning.When start is
end
, we'll put current position inlog
at the end (useful for big files when some new script don't need to read everything).When start is
first
,Log::Unrotate
will start from the oldest log file available.
I.e., if there are foo.log, foo.log.1, and foo.log.2,
begin
will start from the top of foo.log,end
will skip to the bottom of foo.log, whilefirst
will start from the top of foo.log.2. - end
-
Describes the reading behavior when we reach the end of a log. Allowed values:
fixed
(default),future
.When end is
fixed
, the log is read up to the position where it ended when the reader object was created. This is the default, so you don't wait in a reading loop indefinitely because somebody keeps adding new lines to the log.When end is
future
, it allows the reading of the part of the log that was appended after the reader was created (useful for reading from stdin).
- lock
-
Describes the locking behaviour. Allowed values:
none
(default),blocking
,nonblocking
.When lock is
blocking
, lock named pos.lock will be acquired in the blocking mode.When lock is
nonblocking
, lock named pos.lock will be acquired in the nonblocking mode; if lock file is already locked, exception will be raised.
- check_lastline
-
This flag is set by default. It enables content checks when detecting log rotations. There is actually no reason to disable this option.
- check_inode
-
Enable inode checks when detecting log rotations. This option should not be enabled when retrieving logs via rsync or some other way which modifies inodes.
This flag is disabled by default, because check_lastline is superior and should be enough for finding the right file.
- read()
-
Read a line from the log file.
- position()
-
Get your current position in log as an object passible to
commit()
. - commit()
- commit($position)
-
Save the current position to the pos-file. You can also save some other position, previosly obtained with
position()
.Pos-file gets commited using a temporary file, so it won't be lost if disk space is depleted.
- lag()
-
Get the size of data remaining to be read, in bytes.
It takes all log files into account, so if you're in the middle of foo.log.1, it will return the size of remaining data in it, plus the size of foo.log (if it exists).
- log_number()
-
Get the current log's number.
- log_name()
-
Get the log's name. Doesn't contain
.N
postfix even if cursor points to old log file.
BUGS & CAVEATS
To find and open correct log is a race-condition-prone task.
This module was used in production environment for many years, and many bugs were found and fixed. The only known case when position file can become broken is when logrotate is invoked twice in *very* short amount of time, which should never be a case.
Don't set the check_inode option on virtual hosts, especially on openvz-based ones. If you move your data, inodes of files will change and your position file will become broken. In fact, don't set check_inode at all, it's deprecated.
The logrotate config should not use the compress
option to make that module function properly. If you need to compress logs, set delaycompress
option too.
This module expects the logs to be named foo.log, foo.log.1, foo.log.2, etc. Skipping some numbers in the sequence is ok, but postfixes should be *positive integers* to be properly sorted. If you use some other naming scheme, for example, foo.log.20130604-140500, you're out of luck. Patches welcome!
AUTHORS
Andrei Mishchenko druxa@yandex-team.ru
, Vyacheslav Matjukhin me@berekuk.ru
.
SEE ALSO
File::LogReader - another implementation of the same idea.
unrotate - console script for reading logs.
COPYRIGHT
Copyright (c) 2006-2013 Yandex LTD. All rights reserved.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
See <http://www.perl.com/perl/misc/Artistic.html>