NAME
Punk::Observe::Retain - compaction, rollups and deletion
SYNOPSIS
use Punk::Observe::Retain;
my $s = Punk::Observe::Retain::sweep(\@segment_paths, $cutoff_ns);
printf "freed %s bytes from %d segments\n",
$s->{bytes_freed}, $s->{unlinked};
my $r = Punk::Observe::Retain::rollup(
[ map { { t => $_ * 1_000_000_000, v => $_ + 0.0 } } 1 .. 3600 ], 3);
printf "avg over the hour: %s\n", $r->{value_1h};DESCRIPTION
Data is deleted by whole block - two hours - and never by record. Deleting one line would mean rewriting a compressed block, which would make a segment mutable and remove the property every reader depends on.
So "delete this one log line" is not supported, and the granularity is the answer to why.
Deletion cannot break a reader
The deletion primitive is unlink(2), never ftruncate(2).
A reader holding a memory map of an unlinked file keeps reading it correctly: the name is gone, the data lives until the last mapping drops. A reader holding a map of a truncated file takes SIGBUS on the next touch - not an error return, a signal, killing the worker mid-request for every connection it was holding.
"sweep" reports truncate_calls, and it must be zero.
A deleted file that is still mapped is still on the disk
A large segment unlinked an hour ago occupies its space until the last worker drops its mapping, so the space a retention policy promises and the space the filesystem reports can differ with no visible explanation. That is what the generation table is for: a segment is removable when no reader holds its generation.
Downsampling refuses percentiles
Two tiers, at five minutes and one hour, each point carrying {count, sum, min, max, last}. That set is closed under merging, so an hourly point is built from twelve five-minute points without returning to the raw data. It answers count, sum, avg, min, max and rate exactly.
It cannot answer a percentile, and it refuses to. There is no function of those five numbers that yields a p95, and every approximation that looks close is wrong in the tail - which is the only part of a latency chart anybody reads. "rollup" returns a refusal naming the alternative rather than a plausible wrong number.
Where the series is a histogram the percentile merges exactly from the bucket counts, and that is the supported path for a long-range percentile.
Counter resets are carried into the rollup, because the raw points that would reveal one are dropped afterwards.
AGGREGATE CODES
Passed as an integer to "rollup":
1 count 4 min 7 p90
2 sum 5 max 8 p95
3 avg 6 p50 9 p99
10 distinctCodes 6 to 9 are refused over a downsampled range.
FUNCTIONS
pass
my $out = Punk::Observe::Retain::pass(
store => $store, keep_ns => $ns, dry_run => 0);One retention pass: every sealed segment whose newest record is older than now - keep_ns is unlinked, its index sidecar with it, and any sidecar left orphaned by an earlier crash is cleaned up. Returns considered, marked, unlinked, kept, bytes_freed, orphan_idx_removed, unknown_kept and the cutoff it used.
Expiry is decided from the sidecar summaries - the same t_min/t_max a query prunes on - and keyed on t_max: keying on t_min would delete a segment still holding data inside the window. A segment whose sidecar cannot be read is kept and counted in unknown_kept, because deleting on an unknown age is deletion.
There is no default window and no code path that shortens a file. A truncated segment is SIGBUS for every reader mapping it; an unlinked one keeps reading to the end for anyone already holding it, which "read_through_unlink" proves rather than assumes.
bytes adds a budget over and above the window: after the time sweep, if the sealed segments still total more than that many bytes, the oldest are deleted - sidecars in pairs - until the store fits. The window answers "how far back must I be able to look"; the budget answers "how much disk may that cost", and when they disagree the budget wins, because a full disk loses everything rather than the oldest hour. The result gains budget_deleted, budget_freed and bytes (the sealed total after). Under dry_run the budget stage is skipped entirely and says so in budget_skipped - it has no rehearsal mode, and a dry run must not delete.
dry_run takes every decision and skips only the unlink, so the numbers it reports are the numbers the real run would act on.
parse_keep
my $ns = Punk::Observe::Retain::parse_keep('30d'); # undef if not a windowAn operator-written window into nanoseconds, using the query language's own unit table so the two cannot disagree about what a week is. There is no month, for the reason the lexer gives: 1m meaning a month somewhere would be a trap nobody recovers from.
Years are units here - 7y is the window a production store keeps - and y is exactly 365 days, which "Durations" in Punk::Observe::Query explains.
A window too large to represent is refused, and this is the refusal that matters: the cutoff is now - keep in unsigned nanoseconds, so a keep past the last representable instant comes back round as a cutoff of now, which marks every segment in the store for deletion. Under that ceiling nothing needs clamping - the subtraction floors at zero, so a century-long window on a store a week old keeps all of it.
cron_task
my $code = Punk::Observe::Retain::cron_task(
store => $store, keep_ns => $ns, owner => $$);
# cron '17 * * * *' => sub { $code->($queue) };The scheduled shape, identical to "cron_task" in Punk::Observe::Health so that wiring the second one feels like the first: a coderef taking a Punk::Queue, running one pass under the leader lease. Losing the lease race is the normal case on a worker pool, not an error - another worker is doing the pass. owner must be an integer and defaults to the pid.
retain_job
The cron target Punk::Plugin::Observe registers when it is given retain => { keep => '7d' }. Runs one "pass" under the observe.retain lease and reports what it removed. Not called by hand; punk-queue calls it.
merge
my $out = Punk::Observe::Retain::merge([ \@run_a, \@run_b ]);Merges sorted runs into one ordered stream and collapses duplicates. Each record is a hashref taking t, series and kind.
{ records => [ { t, series }, ... ], emitted => 240, duplicates => 12 }The merge is deterministic: the same runs in the same order always produce the same output, which is what makes re-compaction idempotent rather than a source of duplicate points.
duplicates counts records collapsed because an earlier compaction had already emitted them.
rollup
my $out = Punk::Observe::Retain::rollup(\@points, $agg);Folds raw points into the five-minute tier, promotes that to the hourly tier, and answers $agg from each. Each point is a hashref taking t, v and reset.
{
buckets_5m => [ { t, count, sum, min, max, last, resets }, ... ],
n_5m => 12, n_1h => 1,
ok_5m => 1, ok_1h => 1,
value_5m => ..., value_1h => ...,
resets_5m => 0, resets_1h => 0,
}value_5m and value_1h are present only when the corresponding ok is true. When ok_5m is false, refusal carries the message explaining what to do instead - the aggregate was one the tier cannot answer.
The hourly value must equal the five-minute value for every aggregate the tier supports. That is the closure property the whole design rests on.
sweep
my $out = Punk::Observe::Retain::sweep(\@paths, $cutoff_ns);Marks every segment whose data ends before $cutoff_ns and unlinks it.
{
considered => 40, marked => 12,
unlinked => 12, kept => 28,
bytes_freed => 41943040,
truncate_calls => 0,
}truncate_calls must be zero. A non-zero value means something took ftruncate to a segment, and a reader mapping it will take SIGBUS.
read_through_unlink
my $out = Punk::Observe::Retain::read_through_unlink($path);Opens a segment, keeps the mapping, unlinks the path underneath it, and reads every record again through the same mapping.
{
opened => 1, unlinked => 1, records => 600,
sum_before => ..., sum_after => ..., same => 1,
}same is the property: the data read identically after its name was removed. This is why deletion is unlink and not ftruncate.
generations
my $busy = Punk::Observe::Retain::generations(
[ acquire => 1, busy => 1, release => 1, busy => 1 ]);Drives the generation table with a flat list of (operation, generation) pairs and returns an arrayref holding the result of each busy query, in order. Operations are acquire, release and busy.
A generation is busy while any reader holds it. Segments belonging to a busy generation are not removable, which is what stops a sweep pulling the ground from under a query already running.
block_removable
my $bool = Punk::Observe::Retain::block_removable($segments, $expired_all);Whether a block can be removed: it must have no segments still referencing it, and every segment that did must have expired. Both conditions, because removing a block that one segment still points at turns a query into a hole.
SEE ALSO
Punk::Observe, Punk::Observe::Segment, Punk::Observe::Metric