/ 
Punk-Observe-0.01
/ Punk::Observe::Retain

NAME

Punk::Observe::Retain - compaction, rollups and deletion

SYNOPSIS

use Punk::Observe::Retain;

my $s = Punk::Observe::Retain::sweep(\@segment_paths, $cutoff_ns);
printf "freed %s bytes from %d segments\n",
    $s->{bytes_freed}, $s->{unlinked};

my $r = Punk::Observe::Retain::rollup(
    [ map { { t => $_ * 1_000_000_000, v => $_ + 0.0 } } 1 .. 3600 ], 3);
printf "avg over the hour: %s\n", $r->{value_1h};

DESCRIPTION

Data is deleted by whole block - two hours - and never by record. Deleting one line would mean rewriting a compressed block, which would make a segment mutable and remove the property every reader depends on.

So "delete this one log line" is not supported, and the granularity is the answer to why.

Deletion cannot break a reader

The deletion primitive is unlink(2), never ftruncate(2).

A reader holding a memory map of an unlinked file keeps reading it correctly: the name is gone, the data lives until the last mapping drops. A reader holding a map of a truncated file takes SIGBUS on the next touch - not an error return, a signal, killing the worker mid-request for every connection it was holding.

"sweep" reports truncate_calls, and it must be zero.

A deleted file that is still mapped is still on the disk

A large segment unlinked an hour ago occupies its space until the last worker drops its mapping, so the space a retention policy promises and the space the filesystem reports can differ with no visible explanation. That is what the generation table is for: a segment is removable when no reader holds its generation.

Downsampling refuses percentiles

Two tiers, at five minutes and one hour, each point carrying {count, sum, min, max, last}. That set is closed under merging, so an hourly point is built from twelve five-minute points without returning to the raw data. It answers count, sum, avg, min, max and rate exactly.

It cannot answer a percentile, and it refuses to. There is no function of those five numbers that yields a p95, and every approximation that looks close is wrong in the tail - which is the only part of a latency chart anybody reads. "rollup" returns a refusal naming the alternative rather than a plausible wrong number.

Where the series is a histogram the percentile merges exactly from the bucket counts, and that is the supported path for a long-range percentile.

Counter resets are carried into the rollup, because the raw points that would reveal one are dropped afterwards.

AGGREGATE CODES

Passed as an integer to "rollup":

1  count      4  min       7  p90
2  sum        5  max       8  p95
3  avg        6  p50       9  p99
                            10 distinct

Codes 6 to 9 are refused over a downsampled range.

FUNCTIONS

merge

my $out = Punk::Observe::Retain::merge([ \@run_a, \@run_b ]);

Merges sorted runs into one ordered stream and collapses duplicates. Each record is a hashref taking t, series and kind.

{ records => [ { t, series }, ... ], emitted => 240, duplicates => 12 }

The merge is deterministic: the same runs in the same order always produce the same output, which is what makes re-compaction idempotent rather than a source of duplicate points.

duplicates counts records collapsed because an earlier compaction had already emitted them.

rollup

my $out = Punk::Observe::Retain::rollup(\@points, $agg);

Folds raw points into the five-minute tier, promotes that to the hourly tier, and answers $agg from each. Each point is a hashref taking t, v and reset.

{
  buckets_5m => [ { t, count, sum, min, max, last, resets }, ... ],
  n_5m => 12,  n_1h => 1,
  ok_5m => 1,  ok_1h => 1,
  value_5m => ..., value_1h => ...,
  resets_5m => 0,  resets_1h => 0,
}

value_5m and value_1h are present only when the corresponding ok is true. When ok_5m is false, refusal carries the message explaining what to do instead - the aggregate was one the tier cannot answer.

The hourly value must equal the five-minute value for every aggregate the tier supports. That is the closure property the whole design rests on.

sweep

my $out = Punk::Observe::Retain::sweep(\@paths, $cutoff_ns);

Marks every segment whose data ends before $cutoff_ns and unlinks it.

{
  considered => 40,  marked  => 12,
  unlinked   => 12,  kept    => 28,
  bytes_freed => 41943040,
  truncate_calls => 0,
}

truncate_calls must be zero. A non-zero value means something took ftruncate to a segment, and a reader mapping it will take SIGBUS.

my $out = Punk::Observe::Retain::read_through_unlink($path);

Opens a segment, keeps the mapping, unlinks the path underneath it, and reads every record again through the same mapping.

{
  opened => 1,  unlinked => 1,  records => 600,
  sum_before => ..., sum_after => ..., same => 1,
}

same is the property: the data read identically after its name was removed. This is why deletion is unlink and not ftruncate.

generations

my $busy = Punk::Observe::Retain::generations(
    [ acquire => 1, busy => 1, release => 1, busy => 1 ]);

Drives the generation table with a flat list of (operation, generation) pairs and returns an arrayref holding the result of each busy query, in order. Operations are acquire, release and busy.

A generation is busy while any reader holds it. Segments belonging to a busy generation are not removable, which is what stops a sweep pulling the ground from under a query already running.

block_removable

my $bool = Punk::Observe::Retain::block_removable($segments, $expired_all);

Whether a block can be removed: it must have no segments still referencing it, and every segment that did must have expired. Both conditions, because removing a block that one segment still points at turns a query into a hole.

SEE ALSO

Punk::Observe, Punk::Observe::Segment, Punk::Observe::Metric