NAME
Data::Buffer::Shared - Type-specialized shared-memory buffers for multiprocess access
SYNOPSIS
use Data::Buffer::Shared::I64;
# Create or open a shared buffer (file-backed mmap)
my $buf = Data::Buffer::Shared::I64->new('/tmp/mybuf.shm', 1024);
# Keyword API (fastest)
buf_i64_set $buf, 0, 42;
my $val = buf_i64_get $buf, 0;
# Method API
$buf->set(0, 42);
my $v = $buf->get(0);
# Lock-free atomic operations (integer types)
buf_i64_incr $buf, 0;
buf_i64_add $buf, 0, 10;
buf_i64_cas $buf, 0, 52, 100;
# Multiprocess
if (fork() == 0) {
my $child = Data::Buffer::Shared::I64->new('/tmp/mybuf.shm', 1024);
buf_i64_incr $child, 0; # atomic, visible to parent
exit;
}
wait;DESCRIPTION
Data::Buffer::Shared provides type-specialized fixed-capacity buffers stored in file-backed shared memory (mmap(MAP_SHARED)), enabling efficient multiprocess data sharing on Linux.
Linux-only. Requires 64-bit Perl.
Features
File-backed mmap for cross-process sharing
Lock-free atomic get/set for numeric types (single-element)
Lock-free atomic counters: incr/decr/add/cas (integer types)
Seqlock-guarded bulk operations (slice, fill)
Write-preferring futex read-write lock with dead-process recovery
eventfd-based cross-process notification (optional)
Keyword API via XS::Parse::Keyword
Presized: fixed capacity, no growing
Variants
The Str variant stores each value in a fixed $max_len-byte slot, NUL-padded, and trims trailing NULs on the way out. Values longer than $max_len are truncated to it. Embedded and leading NULs are preserved, so "a\0b" round-trips exactly -- but trailing NULs do not: "abc\0" reads back as "abc", and a value that is all NULs reads back as the empty string. For binary payloads that may end in NUL, store an explicit length alongside the value or use a numeric variant.
Constructors
my $buf = Data::Buffer::Shared::I64->new($path, $capacity); # file-backed
my $buf = Data::Buffer::Shared::I64->new_anon($capacity); # anonymous
my $buf = Data::Buffer::Shared::I64->new_memfd($name, $capacity); # memfd
my $buf = Data::Buffer::Shared::I64->new_from_fd($fd); # reopen memfdThe Str variant takes an additional $max_len argument giving the per-element fixed byte width:
my $buf = Data::Buffer::Shared::Str->new($path, $capacity, $max_len);
my $buf = Data::Buffer::Shared::Str->new_anon($capacity, $max_len);
my $buf = Data::Buffer::Shared::Str->new_memfd($name, $capacity, $max_len);
my $buf = Data::Buffer::Shared::Str->new_from_fd($fd, $max_len);new_from_fd duplicates the caller's fd internally; the caller keeps ownership of the passed fd. The Str variant requires the same $max_len the original was created with: it is recorded in the header as the element size and checked on attach, so passing a different $max_len dies with an "elem_size mismatch" error. The descriptor you pass is duplicated (F_DUPFD_CLOEXEC), so it stays yours to close and closing it does not disturb the handle.
Lifecycle
my $p = $buf->path; # backing file path, or undef for anon/memfd
my $fd = $buf->fd; # memfd fd, or undef for anon/file-backed
my $fd = $buf->memfd; # alias of fd()
$buf->clear; # zero all elements (write-locked)
$buf->sync; # msync(MS_SYNC) mmap to backing store
$buf->unlink; # remove backing file (dies for anonymous buffers)
my $h = $buf->stats; # diagnostic hashrefunlink also works as a class method: Data::Buffer::Shared::I64->unlink($path). It croaks if the removal fails -- except when the file is already gone, which is what you asked for, so a cleanup path may safely run twice.
memfd is an alias of fd (present on every variant): both return the backing file descriptor for a memfd-backed buffer (created with new_memfd), or undef for anonymous and file-backed buffers.
API
Replace xx with variant prefix: i8, u8, i16, u16, i32, u32, i64, u64, f32, f64, str.
buf_xx_set $buf, $idx, $value; # set element (lock-free atomic for numeric)
my $v = buf_xx_get $buf, $idx; # get element (lock-free atomic for numeric)
my @v = buf_xx_slice $buf, $from, $count; # bulk read (seqlock)
buf_xx_fill $buf, $value; # fill all elements (write-locked)
buf_xx_clear $buf; # zero all elements (write-locked)set_slice is a method only (its variadic argument list has no keyword form); it writes a run of elements starting at $from and returns true on success:
$buf->set_slice($from, @values); # bulk write (write-locked)Integer variants also have:
my $n = buf_xx_incr $buf, $idx; # atomic increment, returns new value
my $n = buf_xx_decr $buf, $idx; # atomic decrement
my $n = buf_xx_add $buf, $idx, $delta; # atomic add
my $ok = buf_xx_cas $buf, $idx, $old, $new; # compare-and-swap
my $p = buf_xx_cmpxchg $buf, $idx, $old, $new; # CAS, returns prior value
my $n = buf_xx_atomic_and $buf, $idx, $mask; # atomic AND (integer variants)
my $n = buf_xx_atomic_or $buf, $idx, $mask; # atomic OR
my $n = buf_xx_atomic_xor $buf, $idx, $mask; # atomic XORRaw / bulk:
my $raw = buf_xx_get_raw $buf, $byte_off, $nbytes; # raw bytes, seqlock-guarded
buf_xx_set_raw $buf, $byte_off, $raw; # raw bytes, write-locked
$buf->add_slice($from, @deltas); # batch atomic add (integer variants; flat list)
my $ptr = buf_xx_ptr $buf; # raw pointer to data, for FFI use
my $ptr = buf_xx_ptr_at $buf, $idx; # pointer to element at indexget_raw and set_raw address the data area in bytes, not element indices -- unlike every other accessor here. On an I64 buffer $buf->get_raw(4, 4) returns bytes 4..7, which is the upper half of element 0 and the lower half of element 1, not elements 4..7. Multiply by the element size to address elements. Both are bounds-checked against the data area and croak rather than run past it.
Zero-copy:
my $sv = $buf->as_scalar; # mmap-aliased read-only scalar refThe returned scalar aliases the mapped bytes directly (no copy) and holds a reference to the buffer so the mapping stays alive while it is in use.
Cross-process notification (all variants):
my $efd = $buf->create_eventfd; # create + attach an eventfd, returns the fd
$buf->attach_eventfd($fd); # attach an already-open eventfd
my $efd = $buf->eventfd; # current eventfd, or undef if none
$buf->notify; # signal (eventfd write)
my $n = $buf->wait_notify; # drain the counter, non-blocking (undef if 0)These are a thin wrapper over an eventfd(2) descriptor stored in the handle, letting one process signal another that the buffer changed. The eventfd is created non-blocking, so wait_notify does not block: it reads and clears the counter, returning the accumulated notify count, or undef when the counter is zero (nothing pending) or no eventfd is attached. Nothing else in the API depends on them; watch the descriptor for readability in an event loop rather than expecting a blocking wakeup.
Diagnostics:
my $c = buf_xx_capacity $buf;
my $s = buf_xx_mmap_size $buf;
my $e = buf_xx_elem_size $buf;
my $h = $buf->stats; # hashref: capacity/elem_size/mmap_size/variant_id/recoveriesPersistence:
$buf->sync; # msync(MS_SYNC) mmap to backing storeExplicit locking (for batch operations):
buf_xx_lock_wr $buf; # write lock + seqlock begin
buf_xx_unlock_wr $buf; # seqlock end + write unlock
buf_xx_lock_rd $buf; # read lock
buf_xx_unlock_rd $buf; # read unlockThe explicit locks are non-recursive and non-upgradable: calling lock_wr while holding lock_rd on the same handle, or calling lock_wr twice without an intervening unlock_wr, self-deadlocks. Dropping the last reference to a handle while holding one of its locks leaks that handle's reader slot (and any held lock contribution) until the process exits.
CONCURRENCY AND CRASH SAFETY
Single-element numeric get/set and the atomic counter operations (incr/decr/add/cas/cmpxchg/atomic_and/atomic_or/atomic_xor) are lock-free and safe to call concurrently from any number of processes. Bulk reads (slice, get_raw) are guarded by a seqlock and retry if a writer intervenes. Bulk writes (set_slice, fill, clear, set_raw) and the explicit lock_wr/unlock_wr region take a write lock.
The write/read lock is a write-preferring futex read-write lock with dead-process recovery: if a process crashes while holding the lock, another process detects the dead holder and reclaims its contribution so the mapping does not deadlock. See "Reader-slot exhaustion" for the one narrow case this recovery cannot cover.
An interrupted create is also recovered. A creator killed after the backing file is sized but before its header is committed leaves a full-size, all-zero file; new re-initializes such a file automatically, but only when it is owned by your effective uid and is still entirely zero, so a file holding data is never re-initialized. If the creator got as far as writing part of the header, the file cannot be told apart from a corrupt one and new refuses it with uninitialized file is not empty. If that path has only ever been used for this buffer, the file never held data and removing it is safe -- but the same refusal is given for any file of the right size whose first bytes are zero, so check before deleting. An initialized buffer is never re-initialized, resized, or truncated by a later new. On attach the stored geometry wins: the capacity you pass is ignored and the existing capacity is used, so check $buf->capacity if it matters. A variant, element-size or version mismatch is reported as an error.
Disk space. The backing file is created sparse: new sizes it, but blocks are allocated only as you write, so a large buffer costs almost nothing on disk until it is used. The cost of that is a late failure, and how it reaches you depends on the filesystem. Where blocks are allocated at fault time -- tmpfs, so /dev/shm and many /tmp mounts -- a write to a page that cannot be backed raises SIGBUS and kills the process, because an mmap store has no way to report ENOSPC. Where allocation is delayed to writeback (ext4, xfs), the store lands in page cache and the failure appears later: the write is lost, and sync is what reports it, croaking with the underlying error. Keep the filesystem sized for the buffer you asked for, and call sync when you need to know your writes reached disk.
SEE ALSO
Data::HashMap::Shared - concurrent hash table
Data::Queue::Shared - FIFO queue
Data::PubSub::Shared - publish-subscribe ring
Data::ReqRep::Shared - request-reply
Data::Sync::Shared - synchronization primitives
Data::Pool::Shared - fixed-size object pool
Data::Stack::Shared - LIFO stack
Data::Deque::Shared - double-ended queue
Data::Log::Shared - append-only log (WAL)
Data::Heap::Shared - priority queue
Data::Graph::Shared - directed weighted graph
Data::RingBuffer::Shared - fixed-size overwriting ring buffer
Data::BitSet::Shared - shared bitset (lock-free per-bit ops)
SECURITY
Backing files are created with mode 0600 (owner-only) by default, so only the creating user can open and attach them. To share a backing file across users, pass an explicit octal file mode such as 0660 as the last argument to new; the mode is applied only when the file is created or initialized, never to a buffer that is already in use. A pre-existing file is initialized -- and so has the mode applied to it via fchmod -- only when it is owned by your effective uid and is either empty or entirely zero (what an interrupted create leaves behind, see "CONCURRENCY AND CRASH SAFETY"); in every other case the file keeps its own permissions and its contents. The file is opened with O_NOFOLLOW, so a symlink planted at the path is refused, and created with O_EXCL; the on-disk header is validated when the file is attached. Any process you grant write access to a shared mapping is trusted not to corrupt its contents while other processes are using it.
Reader-slot exhaustion
Reader-slot exhaustion (slotless readers): dead-process recovery attributes a crashed lock holder's contribution through its reader-slot. The slot table holds 1024 entries (one per concurrent reader process). If more than that many reader processes share one mapping at once, a reader that cannot claim a slot proceeds "slotless" -- it still takes the read lock but leaves no per-process record. If such a slotless reader is then killed while holding the read lock, its share of the lock cannot be attributed to a dead process, so writer recovery cannot reclaim it and writers may block until the mapping is recreated. Reaching this needs more than 1024 concurrent reader processes on one mapping plus a crash in the brief read-lock window; the dead-process slot reclaim keeps the table from filling with stale entries, so in practice it is very unlikely. Those preconditions cover the live-process route only. The count lives in the mapping and new validates the geometry, not this transient value, so a backing file damaged at rest -- bit rot, a partial copy, or a process that scribbled on the mapping -- can present a non-zero slotless count and block every writer the same way, with none of the above. If writers hang on a file no live reader is using, recreate it.
AUTHOR
vividsnow
LICENSE
This is free software; you can redistribute it and/or modify it under the same terms as Perl itself.