/ 
Punk-Observe-0.05
/ Punk::Observe::Health

NAME

Punk::Observe::Health - poll the services being observed

SYNOPSIS

my ($recs, $state) = Punk::Observe::Health::points(
    'shop', 503, '{"status":"unready","checks":{"db":{"ok":false}}}');

DESCRIPTION

A client for a protocol that already exists. Every Punk service with Punk::Plugin::Health enabled has a /readyz that runs its registered checks and answers with named checks, a boolean, a duration and a reason. Nothing had been reading it.

/readyz, not /healthz

/healthz runs zero checks by construction: the liveness callback is never given the check list. It answers "this process is up and serving", which is worth knowing and is not health. Polling it and calling the answer a health check would be a filter that is never applied wearing a different hat.

The result is a metric, which is why there is no new alerting code

Each poll writes metric points. Because they are ordinary metrics, everything downstream already works: the history is queryable in OQL, a dashboard panel charts it with no new panel type, and the alert evaluator fires on it with no new rule type. punk.health.ok below 1 for five minutes is the alert, expressible in the language today.

That is the argument for storing rather than showing live. A status page that polls on request can say what is broken now; it cannot say "this has been flapping since 03:00", which is the question during an incident.

Two metrics, and a label for the third answer

punk.health.ok    value 1|0   by target, check
punk.health.ms    value ms    by target, check

The target-level answer is the same metric with no check label, which is how this language already spells an ungrouped series - so | by check puts it in the group with the empty key rather than needing a check name invented for it to hang on. A reader that wants only the per-check series asks for the ones that have a check:

metric punk.health.ok | where check != ""

A target can be ready, unready or unreachable, and collapsing the last two is the tempting simplification: "the database check failed" and "we could not ask" lead to different actions. All three are ok = 0 or 1 and carry a state label saying which - so an alert can ignore the distinction (punk.health.ok below 1 for five minutes is the rule either way) and an operator looking at the page can see it.

A fourth, unknown, is a target that answered something that is not JSON - a proxy error page, most often. It is not the service reporting itself unhealthy, and it is not a pass.

FUNCTIONS

points

my ($recs, $state) = points($target, $http_status, $body, $now_ns);

An HTTP answer as metric records. Pure: no I/O, so every failure mode is a fixture rather than a network condition to reproduce. A false or zero $http_status means the request never completed.

poll

my $recs = poll(\%target, ua => $ua, allow => \@allowlist, now => $ns);

One poll. The SSRF policy is re-checked here rather than trusted from save time, because a row can be edited by anything holding the database and an allowlist can be narrowed after a target was stored.

Pass ua, or loop. Awaiting a Fetch::Future runs the loop the agent was built on. Omitting both gives Fetch::Loop::Standalone, and awaiting that inside a Hyperman worker drives a loop which is not the worker's - so the request being served waits on a health poll of an unrelated service. The cron task builds one agent on the worker's loop and passes it down; a caller with no loop at all still works, which is what the hand-runnable binary and the tests rely on.

ua also takes anything with a get returning a future, which is how the tests avoid a network.

run / run_and_store

my $recs = run(db => $db, tenant => 'default', loop => $loop);
my $n    = run_and_store(db => $db, store => $store, loop => $loop);

Every enabled target, once. run returns the records so a caller decides where they go; run_and_store appends them and seals if the log is full.

Never on the request that draws the screen. A poll is outbound network I/O with a timeout, so putting it in the request makes the status page as slow as the slowest target and hangs it when one blackholes packets. It belongs in a cron task under the leader lease - and the lease matters for the same reason it matters for compaction: four workers must not all poll the same target, or the history is four times the traffic and the ms series is meaningless.

One agent is built for the whole pass rather than one per target, and a target that throws does not take the pass down: the next one still needs polling.

uptime_events

my $events = Punk::Observe::Health::uptime_events(
    store => $store, from => $ns, to => $ns);

The up-and-down bands the health chart draws, one series per target and per check, in the shape "alert_timeline" in Punk::Observe::Plot takes.

Derived from the polls, not from a transitions table - health has none, and adding one would put the same truth in two places that can disagree. The window is bucketed into about a hundred and twenty bands, never finer than thirty seconds, and a band is emitted only where the state changed: a target that never failed is one band rather than one per poll.

min over the bucket, deliberately. A bucket in which any poll failed is a bucket the service was down in; an average would render a minute with one failure in five as mostly up, which is the reading that loses the incident. A bucket holding no poll at all is not drawn as down - nobody looked - so the bands either side of it simply meet.

What this replaced was a bar per target of its most recent poll latency: a number the table already prints, beside which a service that had been down for forty minutes drew the same bar as a healthy one.

cron_task

my $code = Punk::Observe::Health::cron_task(
    db => $db, store => $store, loop => $loop, owner => $$);
# register with whatever schedules work in your application:
$q->enqueue('observe.health' => ...);   # or a cron entry calling $code

Returns a coderef that takes a Punk::Queue and runs one pass under the leader lease. owner must be an integer - the queue's own is an IV, and a string numifies to zero, which would give every worker in the pool the same owner and let one renew another's lease. It defaults to the pid. The lease is the point rather than the schedule: four workers must not all poll the same target, or the history is four times the traffic and the ms series is meaningless.

Losing the race is the normal case on a pool and returns 0 without polling - another worker is doing the pass. That is not an error.

This is the shape the rest of the periodic work takes: compaction, retention and alert evaluation are cron tasks under the same lease, so all four inherit election, restart, backoff and the admin UI rather than reinventing them four times.

Register it with the cron keyword Punk::Plugin::Queue installs:

plugin 'Queue' => { dsn => ... };
cron '*/1 * * * *' => Punk::Observe::Health::cron_task(
    db => $db, store => $store), { name => 'observe-health' };

The lease is still taken inside the task rather than assumed from the scheduler: the scheduler runs in the pool, so without it four workers can each fire the same minute. bin/punk-observe-health runs the same pass by hand for a host that would rather drive it from system cron, and for the question this is usually run to settle at 3am.

SEE ALSO

Punk::Plugin::Health, Punk::Observe::Target, Punk::Observe::Config