/ 
Schedule-Easing-0.1.3
/ Schedule::Easing

NAME

Schedule::Easing - Stateless, stable filtering of events with ramp-up activation on a schedule

VERSION

Version 0.1.3

SYNOPSIS

use Schedule::Easing;

my $easing=Schedule::Easing->new(
  schedule=>[
    {
      type  =>'md5',
      name  =>'Sample one',
      match =>qr/^prefix (?<digest>\w+ \d+) at time \d+$/,
      tsA   =>123_000,
      tsB   =>456_000,
      begin =>0,
      final =>1,
      ...
    },
    {
      type  =>'linear',
      name  =>'Sample two',
      match =>qr/^prefix \w+ (?<value>\d+) at time \d+$/,
      tsA   =>123_000,
      tsB   =>456_000,
      ymin  =>100, # the minimum <value>
      ymax  =>999, # the maximum <value>
      begin =>0.05,
      final =>0.95,
    },
  ],
);

my @matches=$easing->matches(ts=>time(), events=>\@events);

DESCRIPTION

Easing provides stateless, stable selection of point-in-time events that need to be exposed with increasing frequency over a period of time. Events may be infrequent or real-time, low or high volume, but must contain some manner of identification for categorization, such as a reported line number or non-random content that can be used to compute a message digest. As time increases, the percentage of events emitted will be monotonically increasing.

Contrasted with throttling, which suppresses any incoming events in real-time once a threshold is exceeded, easing ensures that new events are uniformly distributed over the configured period of time. Whereas throttling requires cached statistics, easing can be performed without resident processes or data stores.

As an example, easing permits an alerting system to be configured based on a large number of reported failures that already exist, without a sudden shift from "no alerts" to a large number of unmanageable alerts. Easing can also be leveraged in A-to-B activation scenarios, supporting staged deployments or similar.

EXAMPLE

Suppose the following failures and warnings are logged

[1755487809] WARNING stock exceeded, requested 24 apples
[1755487826] ERROR invalid type in request, handler.pm line 270
[1755487863] ERROR invalid type in request, handler.pm line 133
[1755487887] WARNING stock exceeded, requested 36 plums
[1755487903] ERROR logging failure for order 8052060, will retry from logger.pm line 323
[1755487925] ERROR invalid type in request, handler.pm line 485
[1755487944] WARNING stock exceeded, requested 144 cherries
[1755487947] ERROR logging failure for order 7463359, will retry from logger.pm line 323
[1755487969] ERROR logging failure for order 8405888, will retry from logger.pm line 323
[1755487990] ERROR logging failure for order 5695806, will retry from logger.pm line 323
[1755488009] INFO stock increased, 144 apples
[1755488012] WARNING stock exceeded, requested 60 grapes
[1755488026] ERROR logging failure for order 4762096, will retry from logger.pm line 323
[1755488049] INFO stock increased, 144 cherries
[1755488059] ERROR invalid type in request, handler.pm line 19
[1755488060] ERROR logging failure for order 9096813, will retry from logger.pm line 323
[1755488187] INFO stock increased, 144 plums
[1755488245] INFO stock increased, 144 grapes
[1755488259] ERROR invalid type in request, handler.pm line 299

The first category of errors, "invalid type", could be transmitted based on the line number of the error. The start and end timestamps have been chosen for this example (roughly 22 minutes apart). At the beginning of the window, no matching lines will emit. At the middle of the window (11min), lines with numbers 0 through 250 will be emitted. At the end and after, all matching lines will be emitted.

{
  name=>'Invalid type errors', type=>'numeric',
  match=>qr/ERROR invalid type.*line (?<value>\d+)/,
  ymin=>0, ymax=>500,
  begin=>0.00, final=>1.00,
  tsA=>1755487800, tsB=>1755489160,
}

The logging failures contain order numbers that can be used to compute a message digest. Over the configured window of time, messages will appear only if their digest exceeds the percent offset within the window. At the halfway point (11min), roughly 50% of all messages will be transmitted.

{
  name=>'Logging failure', type=>'md5',
  match=>qr/ERROR logging failure for order (?<digest>\d+)/,
  begin=>0.00, final=>1.00,
  tsA=>1755487800, tsB=>1755489160,
},

Finally, all stock messages can be blocked.

{
  name=>'Ignore stock messages', type=>'block',
  match=>qr/(.*(?:INFO|WARNING) stock.*)/,
},

With the above configurations, if the sample lines arrive at the given timestamps, the following will be included in the output.

[1755488026] ERROR logging failure for order 4762096, will retry from logger.pm line 323
[1755488059] ERROR invalid type in request, handler.pm line 19

If the logging failures occur again for the same order numbers, on retries, they may be included in the output at a later time. The line-numbered failures will also eventually be included later. Eventually, both types of errors will always be output, because the final value is set to 1/one. The stock messages will always be blocked.

As a simple example, this shows the basic values. In practical use, the configured starting and ending times are likely to span days or weeks, if the errors and warnings will be of a type that requires manual review and correction by individuals. Presumably errors and warnings of these types are common in the system, don't require immediate action, already have other alarms if they reach a critical state, but nevertheless represent ongoing inefficiencies or technical debt that should be addressed.

Configuration

The configuration is an array of line matchers and associated easing configuration. For each input line, configured patterns will be checked in order. The first matching pattern will determine if the message is to be emitted or omitted by the easing configuration. When the line is matched, no additional line matchers will be checked. If a line matches no configured pattern, it is always emitted.

In general, each easing configuration requires a type, match pattern, two timestamps to configure the window, and a begin and final message rate threshold.

Window Specification

Most easing types require a start and end, tsA and tsB, to specify the activation window. Values are Unix epoch seconds. By default, tsA represents the start of the easing behavior, before which all matching messages will be blocked. By default, tsB represents the conclusion of the easing behavior, after which all matching messages will be included. The common use case is that, between the two times, the rate of included messages will increase linearly from 0% to 100%.

Message Thresholds

The default thresholds are begin=0 and final=1, so message inclusion only begins after tsA, and 100% of messages are included after tsB.

Adjusting the values controls the initial and final message rates. At begin=0.1 and final=0.85, for example, initially 10% of matching messages will be included, and by tsB 85% will be included. The remaining 15% will never be emitted, unless the configuration is subsequently updated, but note that the content of the 15% omitted will depend on the easing type selected.

For all easing types, if final=1, matching messages are always included after tsB. When final<1, messages that weren't included before tsB will never be included after tsB.

Easing Types

The value given to each message when comparing against the currently-computed threshold is determined by the easing type.

Numeric

The numeric type requires that the match configuration include a capture group named value that contains a Perl number:

match=>qr/ERROR invalid type.*line (?<value>\d+)/,

The captured value is converted to a percentage by configuring the expected range of values:

ymin=>0
ymax=>500,

Values outside the range are not an error. For values below ymin, they will be included starting at tsA. If a matched value exceeds the configured ymax, it will only be emitted when final=1 starting at tsB; if final<1, it will never be emitted.

MD5

The MD5 type requires that the match configuration include one capture group named digest, or multiple groups following the pattern digest.*:

match=>qr/ERROR logging failure for order (?<digest>\d+)/
match=>qr/^hello (?<digest0>\w+) (?<digest1>\w+)/

All matching groups are ordered by name, concatenated, and used to compute the message digest. The digest value is used modulo the time range, tsB-tsA, to determine if the message crosses the current threshold for inclusion.

Block

For configuration convenience, messages that match match can always be blocked by setting type=block.

Unmatched Lines

All lines are included by default, so setting schedule=[] is equivalent to cat(1). Lines are handled by the first easing configuration that matches the line, which will either include or dispense with the line. If a line matches no easing configuration, it will be included.

Easing Functions

The easing function computes the threshold, p, for the current timestamp, ts, during the configured window, tsA to tsB, given the begin and final rates. Roughly speaking, p is the percentage of matching messages that will be included at that time.

The default function is linear, but others can be selected within the easing configuration by indicating a shape and, if applicable, a shapeopt array.

Linear

The threshold is a linear ramp from begin to final. This is the default and is equivalent to:

shape   =>'linear',
shapeopt=>[],

Step

The thresholds increase from begin to final in a uniform configured number of steps:

shape   =>'step',
shapeopt=>[steps],

When steps=1, no messages will be included until tsB.

When steps=2, there will be a step to the halfway threshold at 0.5(tsA+tsB), and a final step at tsB.

Polynomial

The thresholds follow the curve with the configured exponent:

shape   =>'power',
shapeopt=>[exponent],

Exponents greater than one delay the starting rate at which messages appear. Fractional exponents less than one will cause an increase in initial message rates.

For example, if ts is in the exact middle of the window, supposing begin=0 and final=1, then p=0.5 for exponent=1. At exponent=2, however, p=0.25, so fewer messages are included. At exponent=0.5, p=0.707, so more messages are included.

Event Objects

All above examples call $easing->matches(events=[event, ...])> using linewise matches. That is, event="..." is a message string.

An array of event objects can be passed instead of simple strings. An individual event can be a hash of the form {message="...",...}> or an array of the form [message,...], and the message will be used for matching purposes. The returned list will be the entire matching event objects.

Validation and Debugging

When a non-fatal error is encountered, $$easing{_err}=1 will be set. At this time no additional information is provided, except as emitted by carp.

Invalid easing configurations

Some parameters to an easing entry are mandatory and will fail initialization via confess: Improper schedule structure, entry structure, or invalid types for begin, final, tsA, tsB, match.

Expired easing configurations

Initializing the object with Schedule::Easing->new(warnExpired=1)> will report if any entries in the schedule have tsB in the past, and final=1 or final=0. This permits discovery and removal of unneeded pattern matchers, which otherwise slow down processing. When final<1, no warnings will be reported since filtering is still active.

SEE ALSO

Algorithm::Easing

Throttling solutions...

LICENSE

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Library General Public License Version 3 as published by the Free Software Foundation.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Library General Public License for more details.