NAME
Data::HashMap::Shared - Type-specialized shared-memory hash maps for multiprocess access
SYNOPSIS
use Data::HashMap::Shared::II;
# Create or open a shared map (file-backed mmap)
my $map = Data::HashMap::Shared::II->new('/tmp/mymap.shm', 100000);
# Keyword API (fastest)
shm_ii_put $map, 42, 100;
my $val = shm_ii_get $map, 42;
# Method API
$map->put(42, 100);
my $v = $map->get(42);
# Atomic counters (lock-free fast path)
shm_ii_incr $map, 1;
shm_ii_incr_by $map, 1, 10;
# LRU cache (evicts least-recently-used when full)
my $cache = Data::HashMap::Shared::II->new('/tmp/cache.shm', 100000, 1000);
shm_ii_put $cache, 42, 100; # auto-evicts LRU entry if size > 1000
# TTL (entries expire after N seconds)
my $ttl_map = Data::HashMap::Shared::II->new('/tmp/ttl.shm', 100000, 0, 60);
shm_ii_put $ttl_map, 1, 10; # expires in 60s
shm_ii_put_ttl $ttl_map, 2, 20, 5; # per-key: expires in 5s
# Multiprocess
if (fork() == 0) {
my $child = Data::HashMap::Shared::II->new('/tmp/mymap.shm', 100000);
shm_ii_incr $child, 1; # atomic increment visible to parent
exit;
}
wait;DESCRIPTION
Data::HashMap::Shared provides type-specialized hash maps stored in file-backed shared memory (mmap(MAP_SHARED)), enabling efficient multiprocess data sharing on Linux.
Linux-only. Requires 64-bit Perl.
Features
File-backed mmap for cross-process sharing
Futex-based read-write lock (fast userspace path)
Lock-free atomic counters (incr/decr under read lock)
Elastic capacity (starts small, grows/shrinks automatically)
Arena allocator for string storage in shared memory
Keyword API via XS::Parse::Keyword for maximum speed
Opt-in LRU eviction and per-key TTL (lock-free reads via clock eviction)
Stale lock recovery (automatic detection of dead lock holders via PID tracking)
Variants
Constructor
my $map = Data::HashMap::Shared::II->new($path, $max_entries);
my $map = Data::HashMap::Shared::II->new(undef, $max_entries); # anonymous
my $map = Data::HashMap::Shared::II->new($path, $max_entries, $max_size);
my $map = Data::HashMap::Shared::II->new($path, $max_entries, $max_size, $ttl);
my $map = Data::HashMap::Shared::II->new($path, $max_entries, $max_size, $ttl, $lru_skip);
my $map = Data::HashMap::Shared::II->new_memfd($name, $max_entries, ...); # memfd-backed
my $map = Data::HashMap::Shared::II->new_from_fd($fd); # reopen memfd
my $fd = $map->memfd; # -1 if not memfdCreates or opens a shared hash map backed by file $path. Passing undef as the path creates an anonymous MAP_SHARED|MAP_ANONYMOUS mapping that is inherited across fork but has no filesystem presence.
new_memfd creates an unlinked memfd-backed map whose file descriptor can be passed to another process (via SCM_RIGHTS, fork+exec, or duped+open). new_from_fd reopens such a descriptor. Both require a 64-bit Perl on Linux (memfd_create(2)). $max_entries, $max_size, $ttl, and $lru_skip are used only when creating a new file; when opening an existing one, all parameters are read from the stored header and the constructor arguments are ignored. Multiple processes can open the same file simultaneously. Dies if the file exists but was created by a different variant or is corrupt.
Optional $max_size enables LRU eviction: when the map reaches $max_size entries, the least-recently-used entry is evicted on insert. Set to 0 (default) to disable. LRU uses a clock/second-chance algorithm: get sets an accessed bit (lock-free, no write lock), and eviction gives a second chance to recently accessed entries before evicting.
Optional $ttl sets a default time-to-live in seconds for all entries. Expired entries are lazily removed on access. Set to 0 (default) to disable. When TTL is active, get and exists check expiry.
Optional $lru_skip (0-99, default 0) sets the probability (as a percentage) of skipping LRU promotion on get. This reduces write-lock contention for Zipfian (power-law) access patterns where a small set of hot keys dominates reads. The LRU tail (eviction victim) is never skipped, preserving eviction correctness. Set to 0 for strict LRU ordering.
Zero-cost when disabled: with both $max_size=0 and $ttl=0, the fast lock-free read path is used. The only overhead is a branch (predicted away).
String Keys and UTF-8
String-key variants (SS, SI, IS, I16S, I32S, SI16, SI32) compare keys as raw bytes: two keys are the same entry if and only if they contain the same byte sequence. The SV UTF-8 flag is stored alongside the key so retrieval round-trips it to the returned SV, but it is not part of key identity. Consequences:
ASCII keys with a toggled UTF-8 flag hash and match the same entry (
use utf8,utf8::upgrade, andutf8::downgradeon ASCII are all equivalent from the map's point of view).Non-ASCII keys with different byte encodings are distinct.
"caf\xe9"(latin-1, 4 bytes) and"café"withuse utf8(5 UTF-8 bytes) are two different keys. If your input comes in mixed encodings, normalize withEncode::encode_utf8before use.
Sharding
my $map = Data::HashMap::Shared::II->new_sharded($path_prefix, $shards, $max_entries, ...);Creates $shards independent maps (files $path_prefix.0, $path_prefix.1, ...) behind a single handle, each with up to $max_entries entries (total capacity is $shards * $max_entries). Per-key operations automatically route to the correct shard via hash dispatch. Writes to different shards proceed in parallel with independent locks.
All operations work transparently on sharded maps: put, get, remove, exists, add, update, swap, take, incr, cas, get_or_set, put_ttl, touch, persist, set_ttl, keys, values, items, to_hash, set_multi (method only), get_multi (method only), each, pop, shift, drain, clear, flush_expired, flush_expired_partial, size, stats (method only), reserve, and all diagnostic keywords.
Cursors chain across shards automatically. cursor_seek routes to the correct shard based on key hash. $shards is rounded up to the next power of 2.
API
Replace xx with variant prefix: i16, i32, ii, i16s, i32s, is, si16, si32, si, ss.
my $ok = shm_xx_put $map, $key, $value; # insert or overwrite
my $ok = shm_xx_add $map, $key, $value; # insert only if key absent
my $ok = shm_xx_update $map, $key, $value; # overwrite only if key exists
my $old = shm_xx_swap $map, $key, $value; # put + return old value (undef if new)
my $n = $map->set_multi($k, $v, ...); # batch put under single lock, returns count
my @v = $map->get_multi($k1, $k2, ...); # batch get under single lock with prefetch pipeline
my $v = shm_xx_get $map, $key; # returns undef if not found
my $ok = shm_xx_remove $map, $key; # returns false if not found
my $ok = shm_xx_exists $map, $key; # returns boolean
my $s = shm_xx_size $map;
my $m = shm_xx_max_entries $map;
my @k = shm_xx_keys $map;
my @v = shm_xx_values $map;
my @items = shm_xx_items $map; # flat (k, v, k, v, ...)
while (my ($k, $v) = shm_xx_each $map) { ... } # auto-resets at end
shm_xx_iter_reset $map;
shm_xx_clear $map;
my $href = shm_xx_to_hash $map;
my $v = shm_xx_get_or_set $map, $key, $default; # returns valueInteger-value variants also have:
my $n = shm_xx_incr $map, $key; # returns new value
my $n = shm_xx_decr $map, $key; # returns new value
my $ok = shm_xx_cas $map, $key, $expected, $desired; # compare-and-swap
my $n = shm_xx_incr_by $map, $key, $delta;LRU/TTL operations (require TTL-enabled map for put_ttl):
my $ok = shm_xx_put_ttl $map, $key, $value, $ttl_sec; # per-key TTL (0 = permanent); requires TTL-enabled map
my $ms = shm_xx_max_size $map; # LRU capacity (0 = disabled)
my $t = shm_xx_ttl $map; # default TTL in seconds
my $r = shm_xx_ttl_remaining $map, $key; # seconds left (0 = permanent, undef if missing/expired/no TTL)
my $ok = shm_xx_touch $map, $key; # reset TTL to default; promotes in LRU; false if no TTL/LRU
my $ok = shm_xx_persist $map, $key; # remove TTL, make key permanent; false on non-TTL maps
my $ok = shm_xx_set_ttl $map, $key, $sec; # change TTL without changing value (0 = permanent); false on non-TTL maps
my $n = shm_xx_flush_expired $map; # proactively expire all stale entries, returns count
my ($n, $done) = shm_xx_flush_expired_partial $map, $limit; # gradual: scan $limit slotsAtomic remove-and-return:
my $v = shm_xx_take $map, $key; # remove key and return value (undef if missing)
my ($k, $v) = shm_xx_pop $map; # remove+return from LRU tail / scan forward
my ($k, $v) = shm_xx_shift $map; # remove+return from LRU head / scan backward
my @kv = shm_xx_drain $map, $n; # remove+return up to N entries as flat (k,v,...) listpop and shift remove from opposite ends: pop takes the LRU tail (oldest / least recently used) while shift takes the LRU head (newest / most recently used). On non-LRU maps, pop scans forward and shift scans backward. drain removes in pop order (tail-first). Useful for work-queue patterns and batch processing.
Cursors (independent iterators, allow nesting and removal during iteration):
my $cur = shm_xx_cursor $map; # create cursor
while (my ($k, $v) = shm_xx_cursor_next $cur) { ... }
shm_xx_cursor_reset $cur; # restart from beginning
shm_xx_cursor_seek $cur, $key; # position at specific key (best-effort across resize)
# cursor auto-destroyed when out of scopeshm_xx_each is also safe to use with remove during iteration. Resize/compaction is deferred until iteration ends.
Diagnostics:
my $cap = shm_xx_capacity $map; # current table capacity (slots)
my $tb = shm_xx_tombstones $map; # tombstone count
my $au = shm_xx_arena_used $map; # arena bytes used (0 for int-only)
my $ac = shm_xx_arena_cap $map; # arena total capacity (0 for int-only)
my $sz = shm_xx_mmap_size $map; # backing file size in bytes
my $ok = shm_xx_reserve $map, $n; # pre-grow (false if exceeds max)
my $ev = shm_xx_stat_evictions $map; # cumulative LRU eviction count
my $ex = shm_xx_stat_expired $map; # cumulative TTL expiration count
my $rc = shm_xx_stat_recoveries $map; # cumulative stale lock recovery count
my $p = $map->path; # backing file path (method only)
my $s = $map->stats; # hashref with all diagnostics in one call
# stats keys: size, capacity, max_entries, tombstones, mmap_size,
# arena_used, arena_cap, evictions, expired, recoveries, max_size, ttlset_multi, stats, path, and unlink are method-only (no keyword form).
File management:
$map->unlink; # remove backing file (mmap stays valid)
Data::HashMap::Shared::II->unlink($path); # class method formCrash Safety
If a process dies (e.g., SIGKILL, OOM kill) while holding the write lock, other processes will detect the stale lock within 2 seconds via PID tracking and automatically recover. The writer's PID is encoded in the rwlock word itself (single atomic CAS, no crash window), so recovery is reliable even if the process is killed mid-acquisition. On timeout, waiters check kill($pid, 0) and CAS-release the lock if the holder is dead.
Limitation: PID-based recovery assumes all processes share the same PID namespace. Cross-container sharing (different PID namespaces) is not supported.
After recovery from a mid-mutation crash, the map data may be inconsistent. Calling clear after detecting a stale lock recovery is recommended for safety-critical applications.
BENCHMARKS
Throughput versus other shared-memory / on-disk solutions, 25K entries, single process, Linux x86_64. All values in M ops/s (higher is better). Run perl -Mblib bench/vs.pl 25000 to reproduce.
Integer key → integer value (Shared::II):
BerkeleyDB LMDB Shared::II
INSERT 31 46 184
LOOKUP 35 40 383
INCREMENT 16 18 165String key → string value, short (inline ≤ 7B, Shared::SS):
FastMmap BerkeleyDB LMDB SharedMem Shared::SS
INSERT 11 26 40 62 130
LOOKUP 10 32 34 146 213
DELETE 14 18 -- 32 68String key → string value, long (~50-100B, Shared::SS):
BerkeleyDB LMDB SharedMem Shared::SS
INSERT 25 37 61 133
LOOKUP 30 33 125 229LRU cache lookup (25K entries, lock-free clock eviction):
plain LRU
II 350 373 (lock-free, ~6% faster via clock)
SS 159 159Cross-process (25K SS entries, 2 processes, ops/s):
Shared::SS SharedMem LMDB
READS 3,250,000 1,986,000 728,000
WRITES 2,801,000 826,000 95,000
MIXED 50/50 3,691,000 1,963,000 211,000LMDB benchmarked with MDB_WRITEMAP|MDB_NOSYNC|MDB_NOMETASYNC|MDB_NORDAHEAD. BerkeleyDB with DB_PRIVATE|128MB cache.
Key takeaways:
10x faster lookups than LMDB for integer keys (lock-free seqlock path)
1.5x faster than Hash::SharedMem for short string lookups (inline strings, no arena overhead)
1.8x faster than Hash::SharedMem for long string lookups
4.5x faster cross-process reads than LMDB; 3.4x faster writes than SharedMem
LRU reads are lock-free (clock eviction) — no overhead vs plain maps
Atomic
incris 9x faster than get+put on competitorsStrings ≤ 7 bytes stored inline in node (zero arena overhead)
SEE ALSO
Data::Buffer::Shared - typed shared array
Data::Queue::Shared - FIFO queue
Data::PubSub::Shared - publish-subscribe ring
Data::ReqRep::Shared - request-reply
Data::Sync::Shared - synchronization primitives
Data::Pool::Shared - fixed-size object pool
Data::Stack::Shared - LIFO stack
Data::Deque::Shared - double-ended queue
Data::Log::Shared - append-only log (WAL)
Data::Heap::Shared - priority queue
Data::Graph::Shared - directed weighted graph
Data::BitSet::Shared - shared bitset (lock-free per-bit ops)
Data::RingBuffer::Shared - fixed-size overwriting ring buffer
AUTHOR
vividsnow
LICENSE
This is free software; you can redistribute it and/or modify it under the same terms as Perl itself.