NAME
Punk::Observe::Evaluate - the alert evaluation pass, and its delivery
SYNOPSIS
# what the plugin registers on the host's queue - nothing to write:
# task observe.evaluate the pass
# task observe.notify one delivery, retried by the queue
# cron observe-evaluate @every 30s
# the pass, directly, for a host or a test that has no queue
my $out = Punk::Observe::Evaluate::run(db => $backend, store => $store);
# { evaluated => 3, transitions => 1, enqueued => 1, groups => [...] }
# one delivery
Punk::Observe::Evaluate::notify(
db => $backend, group => $id, token => $t,
on_alert => sub { my ($event) = @_; ... });DESCRIPTION
The loop every host used to write, in the distribution.
Rules live in the configuration store and are edited on the alerts screen; this module reads them, queries the telemetry store, drives Punk::Observe::Alert's state machine, records state and transitions, and hands deliveries to the queue. The host's whole contribution is the on_alert callback - delivery is the one thing the core cannot know.
One pass
Each rule keeps its own cadence inside the shared pass: the cron interval is the resolution, the rule's every is honoured on top of it. The query window is derived from the rule - twice its for, four of its every, never less than fifteen minutes - rather than fixed, so a five-minute rule does not read an hour of data and a forty-minute one is not starved.
A query that fails or answers with the wrong shape becomes a fail tick, so the state machine's own error latch does the work: the rule goes to error, notifies once, and re-arms on the next successful evaluation. Hand-recording error states beside the machine is how the demo got that wrong. The failure's reason - the store's error, or "add a bucket stage" for the unbucketed query somebody writes by accident - is written onto the state row and shown on the alerts screen, and cleared when the rule recovers, so it can never be stale.
An empty successful answer is not a failure: a query that answered with no series in the window is every series vanishing, which is the machine's stale semantics rather than its error latch. And a successful evaluation reconciles the rule's state rows it did not mention: an error row - including the synthetic all a fail tick records - resolves to ok, and anything else goes stale, exactly as it would had the series vanished within one run. Without this, a rule whose query was fixed to group by a label kept its old error row for ever, because no evaluation mentioned that series again. A failed pass reconciles nothing - a query that could not run says nothing about which series exist.
State, transition and outbox row are written in one transaction with a rollback guard. The transaction is the idempotency - a crashed pass leaves the old state and the re-run re-derives the same transition - and the guard is a lesson paid for: a die between begin_work and commit with nothing to roll it back left an evaluator holding the write lock while it slept, and every other writer in the application saw "database is locked".
Grouping, and the outbox
One bad deploy is one message listing forty series, not forty messages. Notifying transitions join their rule's open group; the group holds for group_wait and is then claimed and delivered whole. At most one group per rule is open at a time - enforced by a partial unique index, not by code remembering to - so a series arriving after its group flushed opens a new one: the forty-first service is a second notification, not a lost one.
The outbox key is the transition itself: (rule, series, fired_at, at). A concurrent leader inserting the same transition is refused, and a resolve-and-refire carries new instants and is a new notification. Keying on fired_at alone would have silently dropped every error episode after the first, because a rule that cannot evaluate has no fired_at.
repeat_interval (off unless configured) re-delivers a group whose rule is still firing; one whose rule went quiet retires instead of re-paging.
Silences
Checked at delivery, not at insert - a silence created after a member was enqueued but before its group came due must still suppress it. Suppression is recorded on the row, so the screen can say why nobody was paged; the state machine never hears of it, which is what "a silence suppresses notification, not state" means mechanically.
Delivery
observe.notify makes one callback invocation per group and lets the queue own the failure story: a die retries with full jitter, five attempts, and the terminal failure is a dead letter - recorded on the group and its members, visible, never dropped. The next fire opens a fresh group, so one unreachable webhook does not stop alerting.
Delivery is at least once: a crash between the callback returning and the sent-mark re-delivers. The alternative - marking first - is at most once, which is a lost page. delivery_key in the event is the idempotency token for a host that wants exactly-once on its own side.
FUNCTIONS
run
my $out = Punk::Observe::Evaluate::run(
db => $backend, store => $store,
now => $ns, tenant => $t, force => 1,
group_wait_ns => 30e9, repeat_ns => 0);The pass. Returns evaluated, transitions, enqueued and the claimed groups - handed back rather than delivered, because delivery belongs on the queue with its retries, and a test wants to see what would be sent without sending it. now injects the clock; force ignores each rule's own cadence.
notify
my $r = Punk::Observe::Evaluate::notify(
db => $backend, group => $id, token => $t,
on_alert => sub { ... }, prefix => '/observe', app => $class);One delivery. A stale token - the group was reaped and re-claimed while the job sat in the queue - is a loud no-op, not a die: dying would spend five retries delivering nothing. The event handed to on_alert carries the app class, tenant, the rule row, kind, one entry per series with raw values, the path to the rule's screen, and delivery_key.
evaluate_job / notify_job
The Punk::Queue task bodies. Each receives ($job, $app_class, ...) and recovers the plugin's state - store, database, callback - from the class, because a task body receives its job and its arguments and nothing else, and the worker compiled the same application class the server did.
cron_task
my $code = Punk::Observe::Evaluate::cron_task(db => ..., store => ...,
on_alert => ...);
$code->($queue);The Health-shaped closure, for a host that declares its own cron rather than letting the plugin register one. Takes the queue, takes the leader lock, runs the pass, delivers, returns the transition count.
SEE ALSO
Punk::Observe::Alert - the state machine this drives. Punk::Observe::Route - the in-memory grouping primitives this persists. Punk::Plugin::Observe - where the tasks and the cron are registered.
AUTHOR
LNATION, <email at lnation.org>
LICENSE AND COPYRIGHT
This software is Copyright (c) 2026 by LNATION.
This is free software, licensed under the Artistic License 2.0.