NAME
Punk::Observe::Alert - rule evaluation, and the two states everybody forgets
SYNOPSIS
use Punk::Observe::Alert;
my $ticks = Punk::Observe::Alert::run(
{ op => '>', threshold => 100, for => 60e9, every => 30e9 },
[ { at => $t0, rows => [ 'api' => 150 ] },
{ at => $t0 + 60e9, rows => [ 'api' => 150 ] } ]);
for my $n (@{ $ticks->[-1]{notes} }) {
printf "%s went %s -> %s\n", $n->{series}, $n->{from}, $n->{to};
}DESCRIPTION
A rule is an OQL string executed on the same path the dashboard uses, so an alert cannot fire on a different answer than the graph shows. This module is what happens to the answer.
State is per series
A rule grouped by service produces many series and each carries its own state. One state per rule is the bug that makes an alert resolve because a different service recovered, and it is easy to write because a rule feels like one thing.
The states are ok, pending, firing, stale and error.
The transition that sends nothing
pending back to ok notifies nobody. That is the entire purpose of for: an implementation that notifies there produces exactly the flapping the setting exists to prevent.
for is measured on the condition holding continuously. A breach that clears inside the window resets it, so a rule that spikes for one second a minute does not fire after for of wall time.
A vanished series must not stay firing
A pod is deleted, its series stops being reported, and a naive implementation leaves it red for ever. A permanently red dashboard is how alerting loses its audience, and once it has, the real alert is not read either.
A series absent from a result goes stale, and leaves firing for ok after two evaluation intervals. If it was firing, that emits a resolution saying the series stopped existing - a different event from the condition clearing, and one worth telling apart.
An evaluation error is not "ok"
A rule whose query fails - a store error, a budget refusal, a bad threshold - goes to error and notifies. It does not report healthy.
This is the failure mode most likely to be written by accident, because the natural code path treats "no rows" and "no answer" identically, and the result is a system that reports green because it could not look. So the evaluation carries a status alongside its rows and the two are never collapsed.
An error notifies once, not every tick: a rule that pages every thirty seconds while a store is down is a rule that gets silenced, which is the same as not having it. A successful evaluation re-arms it.
FUNCTIONS
run
my $ticks = Punk::Observe::Alert::run(\%rule, \@ticks);Drives a rule through a sequence of evaluations. The rule takes op (one of >, >=, <, <=, ==, !=), threshold, for and every, the last two in nanoseconds.
Each tick takes at (a nanosecond instant, which becomes the clock for that evaluation), either rows as a flat key-value list or fail for an evaluation that did not run, and returns:
{
notes => [ { series, kind, from, to, at, fired_at }, ... ],
states => [ { series, state, since, fired_at }, ... ],
}kind is 1 firing, 2 resolved, 3 resolved because the series vanished, 4 error.
fired_at is part of the outbox dedupe key. It is stable while a series stays firing and new after a resolve and re-fire, or the second page would be deduplicated away and never delivered.
The clock is injected for the duration of the call and restored afterwards. Nothing sleeps: every transition is a step of at.