NAME
Data::Histogram::Shared - shared-memory HdrHistogram for Linux
SYNOPSIS
use Data::Histogram::Shared;
# track values in [1, 3_600_000_000] with 3 significant figures, anonymous
my $h = Data::Histogram::Shared->new(undef, 1, 3_600_000_000, 3);
$h->record(120); # record the value 120 once
$h->record(2500, 4); # record the value 2500 four times
$h->value_at_percentile(50); # median (50th percentile)
$h->percentile(99); # 99th percentile (alias)
$h->min; $h->max; $h->mean; # min / max / arithmetic mean
$h->total_count; # number of recorded values
# bulk record in a single lock acquisition
$h->record_many([ 100, 250, 250, 900, 1500 ]);
# merge another histogram of identical geometry (cellwise add)
my $other = Data::Histogram::Shared->new(undef, 1, 3_600_000_000, 3);
$other->record_many([ 50, 75, 80 ]);
$h->merge($other);
# share across processes via a backing file
my $shared = Data::Histogram::Shared->new("/tmp/latency.hdr", 1, 3_600_000_000, 3);DESCRIPTION
A High Dynamic Range histogram (HdrHistogram) in shared memory: a compact, fixed-size structure that records integer values across a very wide range and answers percentile, minimum, maximum and mean queries within a fixed, configurable relative error. It is the standard tool for latency and response- time measurement, where the values of interest span many orders of magnitude (microseconds to seconds) yet a constant number of significant figures must be preserved across the whole range.
You construct the histogram with the lowest and highest values it must track and the number of sig_figs (significant figures) of precision you require. Values are bucketed logarithmically -- one bucket per power of two of magnitude -- and linearly within each bucket, so that any two values that agree to sig_figs significant figures fall into the same sub-bucket and are treated as equivalent. A percentile query is therefore accurate to within 1 / 10**sig_figs relative error: with the default 3 significant figures, a reported percentile is within 0.1% of the true value. Memory is proportional to the dynamic range and the precision, not to the number of values recorded: you can record billions of samples into a few kilobytes of counters.
The histogram stores integer values only. To record floating-point quantities, scale them to integers yourself before recording -- for example, record a latency in microseconds (or nanoseconds) rather than fractional seconds, and divide the reported percentiles back down.
Because the counts array lives in a shared mapping, several processes share one histogram: any process that opens the same backing file, inherits the anonymous mapping across fork, or reopens a passed memfd, sees the others' recordings and contributes its own. A write-preferring futex rwlock with dead-process recovery guards mutation, so many processes may record and query concurrently. Two histograms of identical geometry (same lowest, highest and sig_figs) can be combined with merge (cellwise add of their counts arrays), which yields a histogram whose distribution is the union of the two input streams. Linux-only. Requires 64-bit Perl.
METHODS
Constructors
my $h = Data::Histogram::Shared->new($path, $lowest, $highest, $sig_figs);
my $h = Data::Histogram::Shared->new(undef, 1, 3_600_000_000, 3); # defaults
my $h = Data::Histogram::Shared->new_memfd($name, $lowest, $highest, $sig_figs);
my $h = Data::Histogram::Shared->new_from_fd($fd);$path is the backing file (undef or omitted for an anonymous mapping). $lowest is the lowest value that can be distinguished from 0 and must be >= 1 (default 1). $highest is the highest value that can be tracked and must be >= 2 * $lowest (default 3_600_000_000, i.e. one hour in microseconds). $sig_figs is the number of significant figures of precision and must be in the range 1..5 (default 3). new and new_memfd croak if any argument is out of range.
$lowest additionally must satisfy floor(log2($lowest)) + ceil(log2(2 * 10**$sig_figs)) - 1 <= 61 (a histogram-geometry limit); in practice this only rejects a very large $lowest -- above roughly 2**51 at the default sig_figs = 3. Such a histogram croaks at construction.
From these the histogram derives its bucket geometry: a unit magnitude of floor(log2($lowest)), 2 * 10**sig_figs sub-buckets per power of two (rounded up to a power of two), and as many buckets as are needed to cover $highest. When reopening an existing file or memfd, the stored geometry wins and the caller's $lowest/$highest/$sig_figs arguments are ignored. new_memfd creates a Linux memfd (transferable via its memfd descriptor); new_from_fd reopens one in another process.
Recording values
my $count = $h->record($value); # record once; returns new total_count
my $count = $h->record($value, $n); # record $n times; returns new total_count
my $added = $h->record_many(\@values); # record each once; returns how many
$h->reset; # clear every count back to emptyrecord adds $n (default 1) occurrences of the integer $value to the histogram and returns the new total_count. $value must be in the range 0 .. $highest: a negative value croaks, and a value above $highest croaks ("value ... exceeds highest_trackable_value"). Values strictly below $lowest are recorded but collapse into the lowest sub-bucket. $n is an unsigned integer.
record_many takes an array reference and records each element once under a single write lock, returning the number recorded. Every element is range-checked before the lock is taken, so an out-of-range element croaks without recording any of the batch (and without holding the lock).
The precision contract: any two values that agree to sig_figs significant figures are equivalent -- they fall into the same sub-bucket and are indistinguishable once recorded. A value read back out of the histogram (for example via a percentile query) is the highest value equivalent to the bucket it landed in, so it is always >= the recorded value and within 1 / 10**sig_figs relative error of it.
Querying
my $v = $h->value_at_percentile($p); # value at the $p-th percentile (0..100)
my $v = $h->percentile($p); # alias for value_at_percentile
my $c = $h->count_at_value($value); # count recorded in $value's bucket
my $lo = $h->min; # lowest recorded value (0 if empty)
my $hi = $h->max; # highest recorded value (0 if empty)
my $mu = $h->mean; # arithmetic mean (0 if empty)
my $n = $h->total_count; # number of recorded values
my $n = $h->count; # alias for total_countvalue_at_percentile returns the value below which $p percent of the recorded values lie: the highest equivalent value of the bucket at that percentile, so it never underestimates. $p is a percentile in the range 0..100 (not a 0..1 quantile). percentile is a short alias. An empty histogram returns 0 for any percentile.
count_at_value returns the number of values recorded in the bucket that $value falls into (so values equivalent to $value are counted together). $value is range-checked like record: a negative value, or a value above $highest, croaks. min and max are the exact lowest and highest values ever recorded (each 0 when nothing has been recorded). mean is the arithmetic mean, computed from each bucket's median-equivalent value. total_count (aliased count) is the number of recorded values -- the sum of all the per-bucket counts.
Merging
$h->merge($other);Folds $other's counts array into $h by cellwise addition, so $h then represents the combined distribution of both histograms; $h's total_count, min and max are updated to span both. Both histograms must have identical geometry -- the same lowest, highest and sig_figs (merge croaks on a mismatch). $other is read under its own lock into a private snapshot first, so merging is deadlock-free even if two processes merge each other concurrently; $other is not modified. Counts that would overflow a 64-bit cell saturate at the maximum value.
Introspection and lifecycle
$h->lowest; $h->highest; $h->sig_figs; $h->counts_len; $h->stats;
$h->path; $h->memfd; $h->sync; $h->unlink; # or Class->unlink($path)lowest, highest and sig_figs return the configured geometry; counts_len is the number of 64-bit counter cells. sync flushes the mapping to its backing store (a no-op for anonymous and memfd histograms, which have none); unlink removes the backing file (also callable as Class->unlink($path)); path returns the backing path (undef for anonymous, memfd, or fd-reopened histograms) and memfd the backing descriptor -- the memfd of a new_memfd histogram or the dup'd fd of a new_from_fd histogram, and -1 for file-backed or anonymous histograms.
STATS
stats() returns a hashref describing the histogram:
lowest-- the lowest trackable value.highest-- the highest trackable value.sig_figs-- the significant figures of precision.count-- the number of recorded values (total_count).min-- the lowest recorded value (0 if empty).max-- the highest recorded value (0 if empty).mean-- the arithmetic mean of the recorded values (0.0 if empty).counts_len-- the number of 64-bit counter cells.bucket_count-- the number of logarithmic buckets.sub_bucket_count-- the number of linear sub-buckets per bucket.ops-- running count of mutating operations (record,record_many,merge,reset).mmap_size-- bytes of the shared mapping.
SHARING ACROSS PROCESSES
The histogram lives in a shared mapping, shared the same three ways as the rest of the family: a backing file (every process calls new($path, ...) on the same path with matching geometry), an anonymous mapping inherited across fork, or a memfd whose descriptor is passed to an unrelated process (over a UNIX socket via SCM_RIGHTS, or via /proc/$pid/fd/$n) and reopened with new_from_fd($fd). Because the mapping is shared, every process records into and queries the same counts array, so the distribution reflects the combined stream all of them have recorded.
# producer and consumer share one histogram with no coordination
my $h = Data::Histogram::Shared->new(undef, 1, 1_000_000, 3); # before fork
unless (fork) { $h->record(500) for 1 .. 10; exit }
wait;
print $h->value_at_percentile(50), "\n"; # the child's recordingsSECURITY
The mmap region is writable by all processes that open it. Do not share backing files with untrusted processes.
CRASH SAFETY
Mutation is guarded by a futex-based write-preferring rwlock with PID-encoded ownership; if a holder dies, the next contender detects the dead owner and recovers. Each count increment is a single word store, so a crash leaves the histogram consistent up to the last completed record. Limitation: PID reuse is not detected (very unlikely in practice).
SEE ALSO
Data::CountMinSketch::Shared, Data::HyperLogLog::Shared, Data::BloomFilter::Shared, Data::Intern::Shared, Data::SortedSet::Shared, Data::SpatialHash::Shared, and the rest of the Data::*::Shared family.
AUTHOR
vividsnow
LICENSE
This is free software; you can redistribute it and/or modify it under the same terms as Perl itself.