/ 
Linux-Event-0.102
/ docs/DEVELOPMENT-HISTORY.md

Development History

This file preserves the benchmark/optimization phase notes that previously occupied the project README. It is historical material, not the current public API documentation.

Essential completion release (0.101)

Version 0.101 turns the development series into one stable public release. The shared Timer scheduler, asynchronous Resolver and Happy Eyeballs connection path, Signal service, established Stream deadlines, declarative TLS Streams, Listener acceptance, and native framing catalog remain intact.

The release adds four final production surfaces. Wakeup provides a restricted eventfd notification boundary for native producers, fork children, and cloned ithread signalling handles. Common Stream socket policy covers local binding, TCP and buffer configuration, interface binding, and a cached advanced hook. Datagram adds packet-preserving UDP and Unix sockets. Process adds posix_spawnp, pidfd lifecycle and signalling, decoded status, and asynchronous standard I/O.

All high-level resources follow detached construction plus Loop->add() or the equivalent loop constructor option. The only low-level exception is raw Loop->watch(), which continues to return an opaque registration. The current API and guarantees are documented in module POD and the focused design files; the older sections below remain an implementation history only.

Public hierarchy simplification (0.100_025)

The 0.100_024 Watcher/IO/Connect/Listen migration exposed too many overlapping public concepts. Version 0.100_025 removes those compatibility layers. Loop, Stream, and Listener now express ownership directly; raw watch() results are opaque native registration handles. MyStream->connect() is the outbound entry point and Linux::Event::Listener with a configured stream_class is the inbound entry point.

Both loop => $loop and detached construction plus Loop->add() are supported for Stream and Listener. Neither adds a hot-path Perl dispatch layer. Errors, addresses, and framing declarations also moved to their final common Linux::Event::Error, Linux::Event::Address, and Linux::Event::Framer::* namespaces. The sections below intentionally retain the names and decisions of the release they describe.

Unified Watcher lifecycle (0.100_024)

The public model now names every loop-owned activity a Watcher. The canonical Linux::Event::Loop owns raw Linux::Event::_Registration, Stream, Listener, Connector, and future composite Watchers through add(). Watcher is a lifecycle/type contract only; native concrete dispatch remains direct.

Stream now represents outbound acquisition itself. MyStream->connect() preserves one object identity through connect, TLS readiness, message I/O, and close, including bounded writes issued before readiness. Listener creates and attaches accepted Stream instances without application socket plumbing. Connect, Listen, XSLoop, and XSWatcher remain compatibility/advanced implementation names during the migration.

Inbound Listen integration (0.100_020)

The earlier callback-configured listener design was replaced by Linux::Event::Listen subclasses with cached on_accept and on_error methods. Listen now owns TCP and filesystem Unix socket setup or validates an adopted listening handle. Each accepted nonblocking, close-on-exec handle is generic and may be handed directly to Stream, a raw watcher, or another consumer.

A separate native extension drains accept4, while XSLoop continues to own readiness and Stream continues to own established byte transport. Bounded level-triggered batches preserve fairness; resource exhaustion pauses accept readiness before error notification. A permanent lifecycle benchmark compares raw accepted-handle disposal with listener-to-Stream construction.

That benchmark exposed descriptor reuse inside Connect callbacks: closing a watched transferred handle can remove its file description from epoll before the old watcher record is cancelled, and a new socket may immediately reuse the number. Same-fd watcher replacement now retries ADD when MOD reports ENOENT, preserving safe ownership transfer for this reuse case.

Outbound Connect integration and watcher transfer (0.100_019)

The earlier callback-configured Linux::Event::Connect distribution was redesigned and merged into Linux-Event. Concrete subclasses now provide cached on_connect and on_error methods, while each request owns only target, candidate, attempt, deadline, error, and application state. Success transfers a generic connected filehandle and does not require Stream.

Sockets request nonblocking and close-on-exec behavior atomically. A small Connect-native timerfd helper implements the default deadline and defers immediate terminal outcomes until loop dispatch. Same-fd watcher replacement was changed from DEL plus ADD to one MOD with an inert old handle, allowing a success callback to construct Stream without a registration gap or a later Connect cancellation removing the new watcher.

Synchronous hostname resolution and sequential candidate fallback remain explicit initial limitations. Candidate storage is separate from attempt state so asynchronous Resolver and Happy Eyeballs can follow without another public API change.

TLS SIGPIPE isolation (0.100_018)

OpenSSL's default socket BIO may use a signal-generating write after an abrupt peer close. The TLS provider now supplies a non-owning socket BIO backed by Linux recv() and send(MSG_NOSIGNAL). Broken connections therefore enter the existing typed Stream error lifecycle without installing a process-wide signal handler or adding signal-mask syscalls to every TLS operation.

Integrated TLS packaging (0.100_017)

The previously adjacent Linux::Event::TLS distribution moved into the main Linux-Event release. One top-level configuration now builds XSLoop, XSStream, and the separate TLS native extension, and one test run covers all three. TLS uses XSStream's canonical transport ABI header rather than a duplicate copy. This packaging change makes TLS a supported part of Linux::Event without putting OpenSSL calls or state into XSLoop, XSStream, or ordinary plain Streams.

Transport deadline lifecycle (0.100_016)

Cold-path provider lifecycle hooks let an adjacent transport register one deadline watcher, disarm it after setup, reuse it for writable shutdown, and destroy it with the Stream. This supports real stalled-handshake and stalled- shutdown deadlines without putting timerfd or OpenSSL policy into Stream's native parser/write engine or XSLoop.

External transport attachment (0.100_015)

Stream published an exact-version native ABI and transport => $provider construction for established descriptors. Provider lifetime is retained by native connection state, and readiness can cross directions during handshake, I/O, and shutdown without changing XSLoop or parser semantics. The adjacent Linux::Event::TLS distribution is the first provider: it links OpenSSL and implements secure client defaults, server mode, SNI, hostname verification, ALPN, nonblocking handshake, and close_notify.

Native transport boundary foundation (0.100_014)

Direct fd reads, writes, vectored queue drains, retry classification, and writable shutdown moved behind one connection-local native operation contract. The ordinary plain provider remains specialized: XS checks its identity and issues the original syscall directly, avoiding a Perl callback and avoiding an indirect function call on every plain operation.

The result model distinguishes WANT_READ and WANT_WRITE, which ordinary EAGAIN does not need but a nonblocking TLS provider does. This release does not publish provider attachment or add an OpenSSL dependency. The complete handshake, verification, ALPN, buffering, shutdown, deadline, and STARTTLS requirements are recorded in TRANSPORT-BOUNDARY.md before the adjacent Linux::Event::TLS implementation froze the first exact-version ABI in the following release.

Hard pending-output limits (0.100_013)

Class transport policy gained the optional max_pending_bytes hard boundary. High and low watermarks remain cooperative: false from write() still means the bytes were accepted. When an unsent remainder would exceed the hard limit, XS does not allocate its queue segment and reports a typed output_limit error before Stream closes. The error retains the attempted pending count and limit.

The zero default keeps the previous unlimited behavior. The existing Stream transport benchmark now runs otherwise identical capped and uncapped classes to expose the cost of enabling the native check. Protocol transitions also validate existing queued output against the target descriptor before changing any live state.

In-place protocol transitions (0.100_012)

The subclass descriptor became replaceable on a live connection without making configuration per-object again. transition_to() retains the connection's fd, watcher, XSState, output queue, lifecycle, application data, counters, and unread native input while changing the Perl type and shared descriptor.

Native parser loops stop when a callback changes descriptors, then the input driver resumes under the target parser. This specifically supports handshake and upgrade traffic where bytes from the new protocol are already present in the same kernel read. Raw callbacks can transfer their unconsumed suffix with the explicit input option. A separate contract-1 benchmark measures live descriptor changes without mixing them into construction results.

Stream subclass-descriptor redesign (0.100_011)

The object-configured Stream API was replaced after capturing a versioned constructor and retained-memory baseline. Stream behavior is now defined by ordinary subclasses with named callbacks. Each concrete type resolves one shared immutable XS descriptor, while connection objects retain only mutable I/O, parser, queue, lifecycle, and application data state.

The same redesign removed factory-created and arbitrary custom framer objects. Native framers are declared by exact final package name; application-specific protocols parse raw on_data bytes. Current API details live in the README, STREAM-DESIGN.md, and FRAMING.md; the older material below remains only to document how the preceding reactor work was measured.

Linux::Event Phase33C Bounded Callback Scope Experiment

XS-first Linux::Event loop core with Phase29-style performance defaults and experimental knobs kept out of the normal user-facing path.

Build

perl Makefile.PL
make
make test
export PERL5LIB=$PWD/blib/lib:$PWD/blib/arch

Public comparison benchmark

The local Linux::Event target is phase33c, which explicitly loads Linux::Event::Loop from blib/lib and blib/arch. This avoids accidentally benchmarking the globally installed Linux::Event.

The comparison harness isolates every system/client/repeat case in a fresh worker process. This prevents AnyEvent/EV default-loop state from leaking between systems and prevents VmHWM RSS from accumulating across earlier benchmark cases. Failed or timed-out runs retain diagnostic attempt rates in JSON, but are not assigned official ranked msg/s or MiB/s. Summary rows use successful repeats only.

The default timeout is 60 seconds and the default repeat count is 3. For stressed 20k-client runs, use --timeout 90.

The scale workload uses:

Check dependencies first:

perl bench/run-async-comparison.pl \
  --systems phase33c,anyevent,ev,ioasync,mojo \
  --check-deps

Recommended scale run:

ulimit -n 100000

perl bench/run-async-comparison.pl --build \
  --systems phase33c,anyevent,ev,ioasync,mojo \
  --clients 1000,2500,5000,10000,15000,20000 \
  --warmup 1 \
  --messages 10 \
  --bytes 64 \
  --client-driver async \
  --repeats 3 \
  --timeout 90 \
  --out bench/results/comparison-scale-phase33c.html \
  --json bench/results/comparison-scale-phase33c.json

Open or send back:

bench/results/comparison.html
bench/results/comparison.json

Older focused benches

The previous Phase18F scripts are still present for quick internal checks, but the public comparison should use:

bench/run-async-comparison.pl

Phase20 XSLoop update

The generated comparison HTML tables are now browser-sortable. Click any column header to sort by throughput, latency, clients, RSS, backend, correctness, or hot-path counters. A small offline filter box is also included.

Phase20 XSLoop benchmark harness

Phase20 XSLoop keeps the XS-first loop/watchers and expands the public comparison harness.

Key benchmark additions:

Example:

perl bench/run-async-comparison.pl --build   --systems phase33b,anyevent,ev,ioasync   --clients 1,10,50,100   --warmup 100   --messages 1000   --bytes 64,512,4096,16384   --repeats 3   --out bench/results/comparison.html   --json bench/results/comparison.json

For a faster smoke run:

perl bench/run-async-comparison.pl --build   --systems phase20,ev   --clients 1,10   --warmup 10   --messages 100   --bytes 64   --repeats 1

Phase18M: fixes sortable HTML with delegated JavaScript click handling and avoids Perl interpolation-sensitive JavaScript regex literals. Keeps light per-library row coloring.

Phase18M note: HTML sorting now uses inline header onclick bindings and ES5-compatible JavaScript; row colors have stronger contrast and colored left borders.

Phase18N harness additions

The async comparison harness now supports safer high-client-count runs:

perl bench/run-async-comparison.pl \
  --systems phase33b,anyevent,ev,ioasync \
  --clients 5000 \
  --warmup 1 \
  --messages 10 \
  --bytes 64 \
  --repeats 1 \
  --pause-between-systems 5 \
  --out bench/results/comparison-5k.html \
  --json bench/results/comparison-5k.json

For 10k-client runs, running each backend separately is often more reliable:

perl bench/run-async-comparison.pl --systems phase19b --clients 10000 --warmup 1 --messages 10 --bytes 64 --repeats 1 --out bench/results/phase19b-10k.html  --json bench/results/phase19b-10k.json
perl bench/run-async-comparison.pl --systems anyevent --clients 10000 --warmup 1 --messages 10 --bytes 64 --repeats 1 --out bench/results/anyevent-10k.html --json bench/results/anyevent-10k.json
perl bench/run-async-comparison.pl --systems ev       --clients 10000 --warmup 1 --messages 10 --bytes 64 --repeats 1 --out bench/results/ev-10k.html       --json bench/results/ev-10k.json
perl bench/run-async-comparison.pl --systems ioasync  --clients 10000 --warmup 1 --messages 10 --bytes 64 --repeats 1 --out bench/results/ioasync-10k.html  --json bench/results/ioasync-10k.json

Then merge those JSON files into one sortable/color-coded report:

perl bench/run-async-comparison.pl \
  --merge-json bench/results/phase19b-10k.json,bench/results/anyevent-10k.json,bench/results/ev-10k.json,bench/results/ioasync-10k.json \
  --out bench/results/comparison-10k.html \
  --json bench/results/comparison-10k.json

Phase20 instrumentation

Phase20 is intended to be a measurement branch, not a hot-path rewrite. Normal benchmark runs leave nanosecond timing disabled so the results stay comparable with Phase19B. Cheap counters are still exposed through $loop->stats.

New XSLoop methods:

$loop->enable_profile(1);   # enable nanosecond timing buckets
$loop->enable_profile(0);   # disable timing
$loop->reset_stats;         # clear counters/timers, preserving profile flag
my $stats = $loop->stats;

Additional stats include:

epoll_ctl_add_calls / epoll_ctl_mod_calls / epoll_ctl_del_calls
watcher_lookup_calls
dispatch_events
epoll_wait_ns
epoll_ctl_add_ns / epoll_ctl_mod_ns / epoll_ctl_del_ns
watcher_lookup_ns
callback_ns
dispatch_ns

Run a profiled XSLoop-only smoke benchmark:

perl bench/run-async-comparison.pl --build \
  --systems phase20 \
  --clients 1000,2500,5000,10000 \
  --warmup 1 \
  --messages 10 \
  --bytes 64 \
  --client-driver async \
  --xsloop-profile \
  --out bench/results/comparison-scale-phase20-profile.html \
  --json bench/results/comparison-scale-phase20-profile.json

For public throughput comparison, omit --xsloop-profile.

Phase24

Phase24 stores the native watcher pointer directly in epoll_event.data.ptr, so dispatch can go straight from epoll_wait() to the le_watcher_t * without fd-indexed registry lookup in the hot path. The registry remains for public unwatch_fd() and lifecycle operations. Stats include direct_watcher_events.

Phase29 batching/coalescing instrumentation

Phase29 keeps Phase24's direct epoll_event.data.ptr -> le_watcher_t * dispatch path and adds batching/coalescing diagnostics. It is intended as the measurement step before changing read-drain or event-buffer behavior.

New XSLoop methods:

my $n = $loop->event_capacity;
$loop->set_event_capacity(4096);   # optional batching experiment knob

New $loop->stats fields include:

The benchmark target is now phase29:

perl bench/run-async-comparison.pl --build \
  --systems phase29,anyevent,ev,ioasync,mojo \
  --clients 1000,2500,5000,10000 \
  --warmup 1 \
  --messages 10 \
  --bytes 64 \
  --client-driver async \
  --out bench/results/comparison-scale-phase29.html \
  --json bench/results/comparison-scale-phase29.json

Optional event-buffer experiment:

perl bench/run-async-comparison.pl --build \
  --systems phase29 \
  --clients 1000,2500,5000,10000 \
  --warmup 1 \
  --messages 10 \
  --bytes 64 \
  --client-driver async \
  --xsloop-event-cap 4096 \
  --out bench/results/comparison-scale-phase29-cap4096.html \
  --json bench/results/comparison-scale-phase29-cap4096.json

Phase29 note

Phase29 changes the default XSLoop event buffer from 1024 to 8192 based on Phase25 capacity benchmarking. The set_event_capacity API and --xsloop-event-cap benchmark override remain available for experiments.

Phase29 notes

Phase29 is based on the Phase26 production baseline. It adds an opt-in lean => 1 watcher mode for callback_args => 0 callbacks. Lean watchers do not retain loop/fh/data/self accessor references because the callback closure is expected to capture all required state. The benchmark target phase29 enables this lean mode for the echo workload.

Phase30 watcher reclaim/free-list experiment

Phase30 is based on Phase29 and keeps the 8192 event buffer plus lean no-arg watcher mode. It adds an opt-in watcher reclaim/free-list path intended for high-connection churn workloads:

$loop->enable_watcher_reclaim(1);

When enabled, cancel() removes the fd from epoll, clears retained SV references, and recycles the watcher struct after the current dispatch batch. This keeps the default API behavior conservative while letting the benchmark exercise a memory-layout/lifecycle optimization.

The benchmark target is now phase30:

perl bench/run-async-comparison.pl --build \
  --systems phase30,anyevent,ev,ioasync,mojo \
  --clients 1000,2500,5000,10000,15000,20000 \
  --warmup 1 \
  --messages 10 \
  --bytes 64 \
  --client-driver async \
  --out bench/results/comparison-scale-phase30.html \
  --json bench/results/comparison-scale-phase30.json

New stats:

Phase31 watcher reuse benchmark

Phase31 keeps the Phase30 reclaim implementation experimental and adds a benchmark that actually exercises watcher reuse:

perl bench/run-watcher-reuse-bench.pl --build \
  --systems phase29,phase31 \
  --watchers 1000,5000,10000,20000 \
  --cycles 5 \
  --out bench/results/watcher-reuse-phase31.html \
  --json bench/results/watcher-reuse-phase31.json

The normal async comparison target also accepts phase31. It behaves like the Phase30 reclaim path for comparison purposes.

Phase32 API cleanup

Phase32 treats Phase29-style lean no-argument watcher dispatch as the performance baseline and keeps watcher reclaim/free-list work experimental.

Normal application code should not be expected to choose between benchmark-era knobs. The intended direction for the public API is:

my $loop = Linux::Event->new;

and the loop should select the best safe defaults internally.

Phase32 benchmark behavior:

Recommended public comparison command:

perl bench/run-async-comparison.pl --build \
  --systems phase33c,anyevent,ev,ioasync,mojo \
  --clients 1000,2500,5000,10000,15000,20000 \
  --warmup 1 \
  --messages 10 \
  --bytes 64 \
  --client-driver async \
  --out bench/results/comparison-scale-phase33c.html \
  --json bench/results/comparison-scale-phase33c.json

Phase33C bounded callback-scope tuning

Phase33C keeps Phase32 terminal-event semantics and the Phase33B callback fast path, but bounds the number of Perl callbacks that may share one ENTER/SAVETMPS/LEAVE scope. The normal Phase33C default is 128 callbacks per scope. A limit of 0 reproduces the Phase33B whole-epoll-batch experiment.

The tuning aliases are benchmark-only controls:

phase33c-1
phase33c-8
phase33c-16
phase33c-32
phase33c-64
phase33c-128
phase33c-batch

The loop reports callback_scope_limit, callback_scope_rotations, and callback_scope_max_callbacks in stats. The intended workflow is to tune the bounded scope on the target machine, select one safe default, and keep the tuning aliases out of the normal public API.

Phase34 persistent XS run benchmark

Phase34 keeps the Phase33C default of 128 callbacks per Perl scope and uses the persistent XS loop for the benchmark hot path. The benchmark enters XS once via run_for(seconds); callbacks call stop() when the final connection closes. This removes the Perl while/run_once boundary between epoll batches while retaining a monotonic native timeout for failed or stalled cases.

Phase34C benchmark note: phase34c exercises the pure persistent XS run() path with the catastrophic timeout enforced by the parent case supervisor. This avoids the in-worker fork/watchdog and extra epoll fd used by the Phase34B diagnostic path.

Phase35 callback ceiling decomposition

Phase35 adds benchmark-only native echo modes to separate the remaining echo server cost into measurable layers while keeping the real TCP workload.

For the Phase35 experiment, B minus A estimates Perl read-callback entry cost. C minus B estimates the additional cost of doing echo I/O and accounting in Perl instead of XS. Terminal/error callbacks stay on the same Perl path in all three modes so connection shutdown behavior remains comparable.

The _bench_native_echo watcher option is private benchmark instrumentation, not a supported application API.

For authoritative Phase35 ceiling measurements use bench/run-phase35-ceiling.pl. It pre-connects and accepts every client before the timed interval, resets XS statistics, then releases multiple independent async client workers. This avoids measuring connect/accept setup and prevents a single load-generator process from becoming the ceiling before the server does.

Strict same-work reactor ceiling comparison

bench/run-reactor-ceiling-comparison.pl is the fairness-focused continuation of Phase35. It is intentionally separate from the native-echo ceiling modes. Every ranked framework executes the same Perl echo_read() body and therefore does the same server-side sysread, syswrite, buffer, and counter work.

The benchmark removes framework-specific setup and teardown from the timed interval:

The client protocol remains serial request/reply: each client has at most one request outstanding. The payload, number of requests, number of clients, and load-generator workers are identical for every system.

Main leaderboard

The default Linux leaderboard intentionally spans several independent reactor implementations while keeping the application body identical:

UV::Poll is deliberately used instead of UV::TCP because this benchmark is a reactor comparison. Moving socket reads/buffering into libuv's TCP abstraction would be a higher-level stream/runtime comparison rather than the same work.

Likewise, IO::Async and Mojo are driven through their low-level readiness APIs, not IO::Async::Stream or Mojo::IOLoop::Stream. Every main row therefore enters the same named Perl echo_read() function to do the measured sysread/syswrite work.

AnyEvent diagnostics

The AnyEvent EV adaptor has two materially different watcher-construction paths. Its AE::io function is directly aliased to EV::_ae_io when that EV fast path exists. The older AnyEvent->io method API calls public EV::io. The benchmark keeps both paths available, but only the fast path is in the default main leaderboard:

There is also an optional diagnostic system, anyevent-ae-evrun, which keeps the AE::io watcher path but drives it directly with EV::run. This isolates AnyEvent condvar loop-driving overhead from its watcher/callback path. It is a diagnostic, not the primary AnyEvent result.

The direct EV row uses public EV::io and EV::run. EV public watcher callbacks receive watcher and revents arguments; the shared benchmark callback ignores framework callback arguments so the application body remains identical.

Run

First verify dependencies:

perl bench/run-reactor-ceiling-comparison.pl --check-deps

Typical dependency install command:

cpanm EV AnyEvent UV IO::Async IO::Async::Loop::Epoll Mojolicious Mojo::Reactor::Epoll

Authoritative comparison:

ulimit -n 100000

perl bench/run-reactor-ceiling-comparison.pl --build \
  --systems linuxevent,ev,anyevent-ae,uv,ioasync-epoll,mojo-epoll \
  --clients 1000,2500,5000,10000,15000,20000 \
  --warmup 1 \
  --messages 10 \
  --bytes 64 \
  --client-workers 4 \
  --repeats 6 \
  --timeout 90 \
  --out bench/results/reactor-ceiling-comparison.html \
  --json bench/results/reactor-ceiling-comparison.json

The JSON records a fairness contract and a per-result work signature. Important comparison metrics are median messages/second, p50/p95/p99 latency, server CPU microseconds/message, RSS, context switches, reactor iterations when exposed, read callbacks per message, sysread calls per message, and syswrite calls per message.

reactor_iterations is not a portable cross-framework metric. Linux::Event records epoll-wait/run_once activity and EV exposes its own loop-iteration counter, while UV, IO::Async, and Mojo do not expose an equivalent total through the APIs used here. Treat this field as an implementation-local diagnostic, not as a leaderboard metric.

For a fair reactor conclusion, compare only rows where the work signature and exact byte/syscall correctness checks pass. Do not compare Phase35 native-echo modes against EV/AnyEvent Perl echo as if they were the same abstraction level.

Balanced competitor execution order

The competitor ceiling harness runs one client-count/repeat block at a time and rotates the system order deterministically. With the default six systems and six repeats, each system appears in each execution position once per client count. This prevents long-run CPU-frequency, thermal, and scheduler drift from being systematically assigned to one framework simply because all of its cases ran earlier or later than another framework's cases. If diagnostics are added, use a repeat count that is a multiple of the selected system count for perfect position balance; the harness warns when it is not.

Each JSON row records execution_order_mode, execution_order_position, execution_order_width, and execution_block.