NAME
Data::ReqRep::Shared - High-performance shared-memory request/response IPC for Linux
SYNOPSIS
use Data::ReqRep::Shared;
# Server: create channel
my $srv = Data::ReqRep::Shared->new('/tmp/rr.shm', 1024, 64, 4096);
# path, req_capacity, resp_slots, resp_data_max
# Server loop
while (my ($req, $id) = $srv->recv_wait) {
$srv->reply($id, process($req));
}
# Client: open existing channel
my $cli = Data::ReqRep::Shared::Client->new('/tmp/rr.shm');
# Synchronous
my $resp = $cli->req("hello");
# With timeout (single deadline covers send + wait)
my $resp = $cli->req_wait("hello", 5.0);
# Asynchronous (multiple in-flight)
my $id1 = $cli->send("req1");
my $id2 = $cli->send("req2");
my $r1 = $cli->get_wait($id1);
my $r2 = $cli->get_wait($id2);
# Integer variant (lock-free, 1.5x faster)
use Data::ReqRep::Shared::Int;
my $srv = Data::ReqRep::Shared::Int->new($path, 1024, 64);
my $cli = Data::ReqRep::Shared::Int::Client->new($path);
my $resp = $cli->req(42);DESCRIPTION
Shared-memory request/response channel for interprocess communication on Linux. Multiple clients send requests, multiple workers process them, responses are routed back to the correct requester. All through a single shared-memory file -- no broker process, no socket pairs per connection.
Linux-only. Requires 64-bit Perl.
Architecture
Request queue -- bounded MPMC ring buffer. Str variant uses a futex mutex with circular arena for variable-length data. Int variant uses a lock-free Vyukov MPMC queue.
Response slots -- fixed pool with per-slot futex for targeted wakeup and a generation counter for ABA-safe cancel/recycle.
Flow: client acquires a response slot, pushes a request (carrying the slot ID), server pops the request, writes the response to that slot, client reads it and releases the slot.
Variants
-
Variable-length byte string requests and responses. Mutex-protected request queue with circular arena. Supports UTF-8 flag preservation.
my $srv = Data::ReqRep::Shared->new($path, $cap, $slots, $resp_size); my $srv = Data::ReqRep::Shared->new($path, $cap, $slots, $resp_size, $arena); -
Single int64 request and response values. Lock-free Vyukov MPMC request queue. 1.5x faster single-process. No arena, no mutex on the request path.
my $srv = Data::ReqRep::Shared::Int->new($path, $cap, $slots);
Both variants share the same response slot infrastructure, the same generation-counter ABA protection, the same eventfd integration, and the same response-slot crash recovery. Their request queues differ: see "CRASH SAFETY".
Constructors
Server (creates or opens the channel):
->new($path, ...) # file-backed
->new(undef, ...) # anonymous (fork-inherited)
->new_memfd($name, ...) # memfd (fd-passing or fork)
->new_from_fd($fd) # open from memfd fdClient (opens existing channel):
->new($path)
->new_from_fd($fd)Constructor arguments for Str: $path, $req_cap, $resp_slots, $resp_size [, $arena [, $mode]]. For Int: $path, $req_cap, $resp_slots [, $mode]. $mode is the octal backing-file permission (default 0600); see "SECURITY". The descriptor you pass is duplicated (F_DUPFD_CLOEXEC), so it stays yours to close and closing it does not disturb the handle.
Server API
my ($data, $id) = $srv->recv; # non-blocking
my ($data, $id) = $srv->recv_wait; # blocking
my ($data, $id) = $srv->recv_wait($secs); # with timeoutReturns ($request_data, $id) or empty list. For Int, $data is an integer.
my $ok = $srv->reply($id, $response);Writes response and wakes the client. Returns false if the slot was cancelled or recycled (generation mismatch), or if $id names no slot.
Batch (Str only):
my @pairs = $srv->recv_multi($n); # up to $n under one lock
my @pairs = $srv->recv_wait_multi($n, $timeout);
my @pairs = $srv->drain;
my @pairs = $srv->drain($max);Returns flat list ($data1, $id1, $data2, $id2, ...).
Management:
$srv->clear; $srv->sync; $srv->unlink;
$srv->size; $srv->capacity; $srv->is_empty;
$srv->resp_slots; $srv->resp_size; $srv->stats;
$srv->path; $srv->memfd;eventfd (see "Event Loop Integration"):
$srv->eventfd; $srv->eventfd_set($fd);
$srv->eventfd_consume; $srv->notify;
$srv->fileno; # current request eventfd (-1 if none)
$srv->reply_eventfd; $srv->reply_eventfd_set($fd);
$srv->reply_eventfd_consume; $srv->reply_notify;
$srv->reply_fileno; # current reply eventfd (-1 if none)Client API
Synchronous:
my $resp = $cli->req($data); # infinite wait
my $resp = $cli->req_wait($data, $secs); # single deadlineAsynchronous:
my $id = $cli->send($data); # non-blocking
my $id = $cli->send_wait($data, $secs); # blocking
my $resp = $cli->get($id); # non-blocking
my $resp = $cli->get_wait($id, $secs); # blocking
$cli->cancel($id); # abandon requestcancel releases the slot only if the reply hasn't arrived yet. If it has, cancel is a no-op -- call get() to drain, or the slot stays held until the client exits. A reply still being copied in does not delay cancel: the responder frees the slot when its copy finishes. req_wait and get_wait do the cancel-and-drain for you; only a hand-rolled send/cancel loop needs it.
Convenience (Str only):
my $id = $cli->send_notify($data); # send + eventfd signal
my $id = $cli->send_wait_notify($data);Status:
$cli->pending; $cli->size; $cli->capacity;
$cli->is_empty; $cli->resp_slots; $cli->resp_size;
$cli->stats; $cli->path; $cli->memfd;eventfd (see "Event Loop Integration"):
$cli->eventfd; $cli->eventfd_set($fd);
$cli->eventfd_consume; $cli->fileno;
$cli->notify; # signal request eventfd
$cli->req_eventfd_set($fd); $cli->req_fileno;Event Loop Integration
Two eventfds for bidirectional notification. Both are opt-in -- send/reply do not signal automatically.
# Request notification (client -> server)
my $req_fd = $srv->eventfd; # create
$srv->eventfd_consume; # drain in callback
$cli->notify; # signal (or send_notify)
$cli->req_eventfd_set($fd); # set inherited fd
# Reply notification (server -> client)
my $rep_fd = $srv->reply_eventfd;
$srv->reply_notify; # signal after reply
$cli->eventfd; # create (maps to reply fd)
$cli->eventfd_consume; # drain in callback
$cli->eventfd_set($fd); # set inherited fdThe *_eventfd_set methods duplicate the descriptor, as new_from_fd does: the one you pass stays yours to close. One that is not open croaks.
For cross-process use, create both eventfds before fork() so child inherits the fds:
my $srv = Data::ReqRep::Shared->new($path, 1024, 64, 4096);
my $req_fd = $srv->eventfd;
my $rep_fd = $srv->reply_eventfd;
if (fork() == 0) {
my $cli = Data::ReqRep::Shared::Client->new($path);
$cli->req_eventfd_set($req_fd);
$cli->eventfd_set($rep_fd);
$cli->send_notify($data); # wakes server
# EV::io $rep_fd for reply ...
exit;
}
# parent = server
my $w = EV::io $req_fd, EV::READ, sub {
$srv->eventfd_consume;
while (my ($req, $id) = $srv->recv) {
$srv->reply($id, process($req));
}
$srv->reply_notify;
};Crash Safety
Stale mutex -- if a process dies holding the request queue mutex, other processes detect it via PID tracking and recover within 2 seconds.
Stale response slots -- if a client dies while holding a slot (ACQUIRED or READY state), the slot is reclaimed automatically during the next slot acquisition scan.
ABA protection -- response slot IDs carry a generation counter. A cancelled-and-reacquired slot has a different generation, so stale
reply/get/cancelcalls are safely rejected.
Tuning
req_cap-- request queue capacity (power of 2). Higher for bursty workloads (1024-4096), lower for steady-state (64-256). Memory: 24 bytes/slot + arena (Str) or 24 bytes/slot (Int).resp_slots-- max concurrent in-flight requests across all clients. One slot per outstanding async request. For synchronousreq(), one per client suffices. Memory: 64 bytes/slot (Int) or (32 +resp_sizerounded up to 64) bytes/slot (Str).resp_size-- max response payload bytes (Str only). Fixed per slot. Responses exceeding this croak. Pick the 99th percentile.arena-- request data arena bytes (Str only, defaultreq_cap * 256). Increase for large requests. Monitorarena_usedinstats().
Benchmarks
Linux x86_64. Run perl -Mblib bench/vs.pl 50000 to reproduce.
SINGLE-PROCESS ECHO (200K iterations)
ReqRep::Int (lock-free) 1.8M req/s
ReqRep::Str (12B, mutex) 1.2M req/s
ReqRep::Str batch (100x) 1.4M req/s
CROSS-PROCESS ECHO (50K iterations, 12B payload)
Pipe pair (1:1) 240K req/s
Unix socketpair (1:1) 222K req/s
ReqRep::Int 202K req/s *
ReqRep::Str 177K req/s *
IPC::Msg (SysV) 165K req/s
TCP loopback 115K req/s
MCE::Channel 96K req/s
Socketpair via broker 82K req/s
Forks::Queue (Shmem) 5K req/s* = MPMC with per-request reply routing. Pipes and sockets are faster for simple 1:1 echo but require dedicated fd pairs per client-worker connection and cannot do MPMC without a broker (which halves throughput).
CRASH SAFETY
Response slots are recovered from dead owners, and the Str request queue recovers a mutex held by a dead process. The Int request queue is not crash-safe: it is lock-free, and a producer or consumer killed between claiming a queue position and publishing its sequence number leaves a hole that wedges the queue from that position on. Nothing reclaims it -- clear is the only recovery. clear on the Int variant is likewise safe only when no peer is mid-enqueue; the Str variant takes the mutex and has no such caveat.
Destroying a client with requests still in flight abandons their slots: they are held by a live process, so death-based recovery never reclaims them, and they stay held until that process exits. Drain or cancel outstanding ids before dropping a client in a long-lived process.
A slot's generation counter is 32-bit. It guards against a stale id being honoured after the slot is recycled, which it does for any realistic run; after 2^32 re-acquisitions of the same slot an ancient id would compare equal again.
Recovery tells a dead process from a live one with kill($pid, 0), so a PID reused before recovery runs is taken for the process that died. Until that unrelated process exits, a Str queue mutex its predecessor held stays held -- every Str send and recv blocks -- and a response slot it held is not recovered. Linux hands out PIDs in sequence, so this needs the PID space to wrap between the death and the next attempt to recover; a larger kernel.pid_max makes it rarer.
A responder killed in the few instructions between claiming a reply slot and recording its PID leaves that slot held until clear, which gives the PID two seconds to appear before reclaiming it. Nothing else can tell it from a responder merely descheduled there, and reclaiming a live one would deliver its reply to the wrong request. For the same reason clear leaves a slot a live responder is still writing into to that responder, which frees it when done.
An interrupted create is recovered too. A creator killed after the backing file is sized but before its header is committed leaves a full-size, all-zero file. new re-initializes such a file automatically, but only when it is exactly the size the requested geometry needs, is owned by your effective uid, and is still entirely zero -- a file holding data is never re-initialized. If the creator got as far as writing part of the header, the file cannot be told apart from a corrupt one and new croaks with incomplete reqrep file left by an interrupted create; remove it and retry. A file left behind by an interrupted create never held data, so removing it is safe -- but a file whose header was corrupted after the fact reaches the same croak, so confirm it is an abandoned create before deleting anything you care about.
CONTAINERS
Stale-slot and stale-mutex recovery identify peers by PID, and a PID only means something inside one PID namespace. A peer attaching from another namespace would read live processes as dead -- taking their slots and misdelivering replies -- and unrelated local processes as alive, never recovering a real casualty. None of that is detectable after the fact, so new refuses it: the header records the creating process's PID namespace and the current boot id, and attaching from anywhere else croaks.
All peers must therefore share a PID namespace -- docker run --pid=container:NAME, or a Kubernetes pod with shareProcessNamespace: true. Sharing only the filesystem or the IPC namespace is not enough. The same check rejects a file left over from a previous boot, whose recorded PIDs now name unrelated processes.
Set DATA_REQREP_SHARED_UNSAFE_PIDNS=1 to attach anyway. Only do that if you do not depend on recovery -- for example a fixed set of peers that never die mid-request -- because the failure mode it re-enables is silent corruption.
For sharing across containers without a shared filesystem, create the segment with new_memfd and pass the descriptor over a unix socket with SCM_RIGHTS; a memfd crosses namespaces natively. If you use a file, note that a container's default /dev/shm is often only 64 MB. Under a user namespace, new's ownership checks compare uids as mapped in the caller's namespace, so peers need a common id mapping.
SEE ALSO
Data::Buffer::Shared - typed shared array
Data::HashMap::Shared - concurrent hash table
Data::Queue::Shared - FIFO queue
Data::PubSub::Shared - publish-subscribe ring
Data::Sync::Shared - synchronization primitives
Data::Pool::Shared - fixed-size object pool
Data::Stack::Shared - LIFO stack
Data::Deque::Shared - double-ended queue
Data::Log::Shared - append-only log (WAL)
Data::Heap::Shared - priority queue
Data::Graph::Shared - directed weighted graph
Data::BitSet::Shared - shared bitset (lock-free per-bit ops)
Data::RingBuffer::Shared - fixed-size overwriting ring buffer
SECURITY
Backing files are created with mode 0600 (owner-only) by default, so only the creating user can open and attach them. To share a backing file across users, pass an explicit octal file mode such as 0660 as the final $mode argument to new -- for Str after the optional $arena (new($path, $req_cap, $resp_slots, $resp_size, $arena, 0660)), for Int as the fourth argument (new($path, $req_cap, $resp_slots, 0660)); the mode is applied when the file is created, and when a file left behind by an interrupted create is re-initialized (see "CRASH SAFETY"); a file already in use keeps its own permissions. The file is opened with O_NOFOLLOW, so a symlink planted at the path is refused, and created with O_EXCL; the on-disk header is validated when the file is attached. Attaching refuses a world-writable file owned by another user; share with a group mode such as 0660 instead. Any process that can open the file can hold its lock, so new waits for the lock for at most 10 seconds and then croaks. Any process you grant write access to a shared mapping is trusted not to corrupt its contents while other processes are using it.
AUTHOR
vividsnow
LICENSE
This is free software; you can redistribute it and/or modify it under the same terms as Perl itself.