Proc::Spawner and its graph: the split of the executor (2026-11-02)

Status: EXPERIMENTAL. The user endorsed every choice experimentally (2026-11-02, 00:24) and asked for this split (00:32: "Please start"). The decisions taken while writing it, G1–G6 in section 6, are experimental too and wait for approval.

Until 2026-11-01 one module, ShellSpawn, did everything: the shell, the starting of children, the event loop, and a graph that only recorded and drew what happened. Now there are three layers, and the graph is what gets executed:

| Module | Lines | What it does | Depends on | |---|---|---|---| | lib/Proc/Spawner.pm (+ Spawner.xs) | 148 code lines (230 with its POD) | starts a child with a given fd table and any posix_spawn setting; runs the one event loop that reports "a child ended", "an fd is readable/writable"; its XS part is the former binding PosixSpawn | – | | lib/Proc/Spawner/Loop.pm (+ Loop/POSIX.pm, Loop/Win32.pm, win_loop.h) | post-0.10 | the one event loop (W5): queued calls, timers, reads, writes, ended children, one interface on POSIX (select, SIGCHLD) and on Windows (an I/O completion port); Proc::Spawner's on_read, on_write, defer, wait_until, poll are its old names | Proc::Spawner's XS | | lib/Proc/Spawner/Graph.pm | 263 | the graph: open files (ends) with hold counts, process nodes, Perl's own reads and writes; each process carries what to do when it ends; the notation, pictures, running hand-written graphs | Proc::Spawner | | lib/Proc/ShellSpawn.pm (+ ShellSpawn/Large.pm) | 638 (+199) | only the shell: words, redirections, contexts, builtins, traps, jobs | both |

lib/SpawnGraph.pm is gone; its notation, drawing and runner are in Proc::Spawner::Graph.

Renamed 2026-11-02, 07:30 (the user: "PosixSpawn should be subsumed by Proc::Spawner. The rest should be put into Proc::"; names option A). One distribution, Proc-Spawner, built at the top of the bundle with perl Makefile.PL && make. Older docs use the old names:

| Old name | New name | |---|---| | PosixSpawn (XS binding, its own directory and Makefile.PL) | part of Proc::Spawner: Spawner.xs at the top; the raw call PosixSpawn::spawn is now Proc::Spawner::spawn, an alias of the XSUB __posix_spawn (the user: the same interface should have a chance to work on POSIX and Windows, so no posix_ in its name), with a sixth, optional argument { OPTION => 'drop' | 'fail' }: what a port lacking an option (Windows) does with it, defaults in %Proc::Spawner::UNSUPPORTED ('*' => 'fail', EXPERIMENTAL); the call with the fd table and on_exit, formerly spawn, is now start; fpe_at_exec, chld_to_fd, chld_restore are in Proc::Spawner | | ShellSpawn, ShellSpawn::Large | Proc::ShellSpawn, Proc::ShellSpawn::Large (lib/Proc/) | | ShellSubset, ShellSubsetSmall, ShellSubsetLarge (in parser/) | Proc::ShellSubset, Proc::ShellSubset::Small, Proc::ShellSubset::Large (lib/Proc/); the test scripts are in t/parser/ (since 09:58) and find lib/ themselves |

Include paths are now lib and blib/arch. The A4 message says "Proc::ShellSubset::Large: … is not supported" since 0.02 (it said "ShellSubsetLarge: …" before).

1. Proc::Spawner: the small one

Five functions; nothing about pipes, files or the shell.

| Call | Meaning | |---|---| | start(PATH, ARGV, ENVP, FDS, %opt) (was spawn) | starts a child; returns its pid, or undef with $!. FDS = { CHILD_FD => PARENT_FD, or undef for "closed" }; fds not listed are inherited as they are (if not close-on-exec). FDS values may also be [PATH, FLAGS, MODE]: opened in the child. Options (2026-11-02, the user: "more or less all the fields of posix_spawn should be supported"): chdir, fchdir, sigdefault => [signals], sigmask => [signals], pgroup, setsid, tcsetpgrp, resetids, sched, cgroup, search, pidfd (then in list context the pid and a pidfd), umask, unsupported => HASH (passed to spawn), on_exit => CODE. Signals by name or number; an unknown name dies. The full list is in the module's header comment. | | on_read(FD, N, CODE) | one read of at most N bytes when FD is readable: CODE->(BYTES), or CODE->(undef) at end of file | | on_write(FD, BYTES, CODE) | writes BYTES as the reader takes them, never blocking Perl; CODE->() when done or the reader is gone | | defer(CODE, ARGS) | queue CODE->(ARGS) | | wait_until(CODE) | run the queue and wait for events until CODE->() is true |

Example: ls with its output in a pipe whose read end Perl reads, while ls's fd 2 is closed:

pipe my $r, my $w;
my ($out, $st, $eof) = ('');
Proc::Spawner::spawn('/bin/ls', ['ls'], undef, { 1 => fileno $w, 2 => undef }, on_exit => sub { $st = shift });
close $w;
Proc::Spawner::on_read($r, 4096, sub { defined $_[0] ? ($out .= $_[0], Proc::Spawner::on_read($r, 4096, __SUB__)) : ($eof = 1) });
Proc::Spawner::wait_until(sub { $eof && defined $st });

The rules, as decided on the page "Child Exit Wake-up":

2. Proc::Spawner::Graph: the executable graph

The graph says, for the current moment, which of our children, and which of Perl's own reads and writes, use which open file. Each process node carries what to do when it ends.

Example (from t/spawner.t): echo hello writes into pipe 1. When it ends, its handler starts cat reading pipe 1 and writing pipe 2, which Perl reads.

my $g = Proc::Spawner::Graph->new;
my ($r, $w) = $g->new_pipe; my ($pr, $pw) = $g->new_pipe;
sub then_cat { my ($g, $node, $st, $r, $pw) = @_;
  $g->start('/bin/cat', ['cat'], undef, { 0 => $r, 1 => $pw });
  Proc::Spawner::Graph::release($_) for $r, $pw }
$g->start('/bin/sh', ['sh', '-c', 'echo hello'], undef, { 1 => $w }, on_exit => ['main::then_cat', $r, $pw]);
Proc::Spawner::Graph::release($w);

Hand-written graphs in the notation (sgraph/*.sg) now run on this engine. after P1 fail becomes a named handler on P1 (Proc::Spawner::Graph::then) that starts the processes waiting for P1, or marks them never and releases their ends. sgraph/run-graph.pl checks every process's fds with the reporter program: 17 of 17 processes as drawn (6 examples).

Pictures. Proc::Spawner::Graph->new(dot => DIR) writes the graph after every event (start, exit, close) as DIR/trace-001.dot, …; the executor does this with TRACE=dot.

3. How ShellSpawn uses them

4. Behaviour changes

  1. Only our own children are reaped (G2). The loop calls waitpid(PID, WNOHANG) for each child it started, instead of waitpid(-1, …). So it no longer takes the statuses of the caller's own children (finding C2). If other code takes a status of ours first, the child is reported with status undef instead of being waited for forever; the shell gives $? = 1 for it (G3). On Windows other code cannot take it: the exit code stays with the process handle the module keeps until it has reported the child, so other code's waitpid and ours both get it (checked under Wine, post-0.10, t/spawner.t).
  2. The trace says reaped PID: wait status W (the raw status) instead of sh's $?.
  3. In pictures, an ended process shows exited N or signal N, and the Perl node also shows Perl's writes (here-documents), not only its reads.

Nothing else changed: on every battery the old and the new executor print the same output, messages and status for all 234 command lines (only pid lengths differ).

5. Tested

In 0.03 (2026-11-03), t/dash.t gives: largebattery.txt 111 of 114 agree with dash, battery.txt 42/43, fdbattery.txt 14/14, sigenv.txt 12/12, t1.txt 39/45, t2.txt 42/44 (one line's order varies, as in dash), t3.txt 16/17, pspawn-d2.txt 9/10, pspawn-l10.txt 6/9, large-subset-b13.txt 37/38; every other line is a known difference (t/dash-known.txt). Post-0.03 adds large-subset-p6.txt: 19 of 19 (21 of 21 in post2-0.03); the other batteries, run again on post-0.03, give the same counts as 0.03. In 0.04: largebattery.txt 108 of 114 (three lines more now differ from dash on purpose: getopts leaves OPTARG unset, rule E), large-subset-p6.txt 25 of 25, and the new large-subset-e.txt 2 of 10 (the other 8 are rule-E choices listed in t/dash-known.txt); the rest as in 0.03. The counts below are pre-0.01 (2026-11-02).

6. Decisions taken while writing it (EXPERIMENTAL, waiting for approval)

7. Not done

8. Windows (decided 2026-11-02, not written yet; in the distribution since post-0.04: no CPAN release had it, it was only in the Project's later copy of 0.01; checked against CPAN's 0.01 to 0.04 on 2026-11-05)

The user's rule: what Windows can do fully and natively is implemented; everything else is reported as not implemented, so the caller's unsupported hash (or %Proc::Spawner::UNSUPPORTED) decides between drop and fail, and a caller may use a workaround that is safe only in single-threaded execution (as the umask is set around the call on Linux). No "approximate" category.

| Setting | posix_spawn (glibc 2.39) | Windows, natively (CreateProcessW) | When missing: done around the call | |---|---|---|---| | dup2, close, open, closefrom (the child's fds) | yes (closefrom: 2.34) | fds 0-2 (STARTUPINFO), the handle list (close, closefrom), a file opened by the parent; fds 3 and up through the C runtime's lpReserved2 | this process arranges its own fds as the child should have them (close: close-on-exec), saving and putting back what it changes; one of the four missing means all four, and chdir, fchdir too (the actions keep their order) | | chdir, fchdir | 2.29, 2.29 | lpCurrentDirectory (fchdir: the directory handle's path) | this process changes directory and comes back (through an fd of .) | | search | posix_spawnp | our own lookup through PATH and PATHEXT | PATH looked up here | | pidfd | pidfd_spawn (2.39) | the process handle | pidfd_open right after the call (Linux 5.3); none: the pid alone | | sigdefault | yes | nothing to do (nothing is inherited) | an ignored signal set back to its default around the call | | sigmask | yes | no signals | this process's mask set around the call | | pgroup => PGID (joining a group) | yes | no | this process joins the group around the call | | pgroup => 0 (a new group) | yes | CREATE_NEW_PROCESS_GROUP | not possible (the child's pid is known only after): unsupported | | resetids | yes | nothing to do (no set-user-id) | the effective ids set to the real ones around the call | | setsid | 2.26 | no | not possible (no way back): unsupported | | tcsetpgrp | 2.35 | no | not possible (the child's group is known only after): unsupported | | sched, cgroup | yes, 2.35 | no | not undone reliably: unsupported | | umask | (none) | (none) | always set around the call | | sigignore, rlimit, nice (post-0.08, M1) | (none) | no | always set around the call, in C by the XSUB (below); unsupported where it cannot be put back | | posix_spawn itself (post-0.08, M2) | yes | (win_spawn.h) | fork and exec; without fork either, Perl's own system { FILE } 1, ARGV; everything else around the call |

What posix_spawn has no option for (post-0.08; the user, 2026-11-06, decision M1: "do the around-the-call settings where they can be undone, and (temporarily) fail for the rest"). start(..., sigignore => [SIGNALS], rlimit => { NAME => SOFT | [SOFT, HARD] }, nice => INCREMENT); in spawn's attributes sigignore => [SIGNUMS], rlimit => [[RESOURCE, SOFT, HARD], ...] (undef: as ours; 'unlimited'), nice => INCREMENT. The XSUB sets them in this process right around the call, in C (nothing is allocated and no Perl code runs in between, Spawner.xs, around_parse), with every signal blocked meanwhile, and puts them back right after; the child gets the signal mask from before. Unsupported (dropped or failed, as UNSUPPORTED says), because it could not be put back or would harm this process meanwhile:

Failing as the child would: a lower nice value without privilege (EACCES), a limit above our hard one without privilege (EPERM), an unknown resource (EINVAL); a signal both in sigignore and sigdefault, SIGKILL, SIGSTOP: EINVAL. A memory limit (AS) lower than what this process uses makes glibc's posix_spawn fail (ENOMEM: it maps the child's stack in our address space), where a child setting it before exec would not (checked: AS 10 MB fails, ours is put back, later calls work; with fork and exec, M2, the same call succeeds). On Linux nice and the signal mask are per thread, so only the calling thread sees them; the signal actions and the limits are the process's (bullet-proof only in non-threaded execution, as the umask). A signal arriving meanwhile stays pending (Linux keeps a blocked signal pending although it is ignored) and is delivered after.

nice and rlimit on Windows (post-0.09; the user, 2026-11-08, decision W4: "(c) and (b); (c) beats (b) if present"). Windows' own settings, exact, per child: win_priority (a priority class: idle, below_normal, normal, above_normal, high, realtime), set with SetPriorityClass on the child while it is still suspended (Wine ignores the class given as a creation flag, Windows honours both), and win_job (limits of a job of the child's own: ProcessMemoryLimit, JobMemoryLimit in bytes of committed memory, PerProcessUserTimeLimit, PerJobUserTimeLimit in 100-ns ticks of user time, ActiveProcessLimit; the job is nested in the kill_with_parent one, which needs Windows 8, and closed with the child's handle). nice and rlimit are mapped to them, approximately: nice from our class's nice value (idle 19, below_normal 5, normal 0, above_normal -5, high -10, realtime -20) plus the increment, to idle (10 and up), below_normal (1 to 9), normal (0), above_normal (-1 to -9) or high (-10 and below), never realtime; rlimit AS and DATA to ProcessMemoryLimit (committed memory, not address space; the smaller wins), CPU to PerProcessUserTimeLimit, NPROC to ActiveProcessLimit (the processes in the child's job, not the user's), each from the soft value if set, else the hard one (Windows has one limit and ends the child at it, where Unix sends SIGXCPU at the soft CPU limit); NOFILE, CORE, FSIZE, STACK and the rest have no counterpart: unsupported. win_priority and win_job beat the mapping where both are given; elsewhere they are unsupported. Checked under Wine with a child that prints its priority class: win_priority, nice 5, 15, -12, and nice with win_priority give the classes above; a resource with no counterpart fails with ENOTSUP or is dropped; an unknown class fails with EINVAL. Wine stores no job limits (a plain C program setting them reads back none), so their effect is not checked here, only that the child starts with them; the mapping is tested on Linux (t/spawner.t).

No posix_spawn at all (post-0.08; the user, 2026-11-06, decision M2: as Run 0.06's forkless mode, branch project-2026-11-03, claude/Run-0.06/). Makefile.PL now probes posix_spawn itself (POSIX_SPAWN); without it, or with PROC_SPAWNER_MISSING=posix_spawn (tests), the primitive is the XSUB __spawn_nowait: fork and exec, in C, with nothing in between but the signal mask put back (as spawnvp(P_NOWAIT) on a system with a plain spawn; exec's errno comes back through a close-on-exec pipe). Every other setting is then missing and done around the call as above (setsid, pgroup => 0, tcsetpgrp, sched, cgroup unsupported); rlimit, sigignore, nice as M1. argv reaches the child exactly (execve). Checked: the whole suite passes so (58 tests), and a build without posix_spawn (PROC_SPAWNER_WITHOUT=POSIX_SPAWN) passes t/spawner.t and t/dash.t. Where there is no fork either (Windows when posix_spawn is masked, OS/2, VMS; on Linux with PROC_SPAWNER_MISSING="posix_spawn fork": $Proc::Spawner::PRIMITIVE is system1), the redirections are delegated to Run (the user, 2026-11-07: "one of the major points of the exercise"; "it must delegate the whole code of redirection to Run"): Run 0.07, in repository ilyaz-ai/run (0.02 to 0.06 as released, then 0.07), to be released before Proc-Spawner needs it. Run::start(1, ARGV, REDIRS, ERRS, FILE) applies the redirections in this process around Perl's own spawn, system { FILE } 1, ARGV (P_NOWAIT; on Unix Run emulates it by fork and exec), and puts our fds back; the fd actions become Run's [FD => {fd => M}], {close => 1}, and a file opened here as {filehandle => FH, pin => 1}; chdir and fchdir are done here, in order, so an open's relative path is taken from the directory of that moment. What Perl's spawn does on Windows (win32/win32.c of 5.38.2, do_spawnvp_handles, create_command_line):

The Windows primitive (post-0.06, EXPERIMENTAL, not yet built into the module). win_spawn.h is posix_spawn on CreateProcessW, in plain C; Spawner.xs includes it under _WIN32 and #defines the posix_spawn names to it, so __posix_spawn stays one source. It does the Windows column of the table above: the child's fds 0-2 through STARTUPINFO, fds 3 and up through the C runtime's lpReserved2 table, exactly the handles the child should have through PROC_THREAD_ATTRIBUTE_HANDLE_LIST (inheritable copies made for the call, so the parent's handles keep their flags); the file actions in order on a table of the child's fds (an open by the parent, relative to the child's directory); argv quoted as the C runtime parses it back; posix_spawnp through PATH and PATHEXT (a relative program name is taken from the child's directory, after chdir, as glibc does); pgroup => 0 as CREATE_NEW_PROCESS_GROUP; ENOTSUP for the rest. The process handle is kept, so the exit code survives until the child is waited for (__win_spawn_handle, __win_spawn_forget). Checked with MinGW-w64 under Wine; compiled for x86, x64 and ARM64. Still to do on Windows: building the module with a Windows Perl, waiting for children through the kept handles, and the event loop without SIGCHLD.

Compared with Perl's own spawn on Windows (do_spawnvp_handles(), create_command_line(), qualified_path(), win32_waitpid(), win32_kill() in win32/win32.c of perl 5.38.2; 2026-11-05). Fixed in win_spawn.h and Spawner.xs after the comparison:

Run under Wine with Strawberry Perl 5.38.2 (the user's approval, 2026-11-05): Spawner.dll built with MinGW-w64 against its headers and libperl538.a loads; Proc::Spawner::spawn starts Windows' perl.exe, an open action gives it a file as stdout, Perl's own waitpid reaps it (status 3), and %Proc::Spawner::WINDOWS works. This found two faults, fixed: spawn gave the count of its results, not the pid, in scalar context (its wrapper; on Linux only when a setting is missing); Proc::Spawner used fcntl and SIGCHLD at load time. On Windows the loop now polls for ended children every $TICK; Perl's select there works on sockets only, so on_read, on_write of pipes are not done yet.

How the child sees its arguments (a Windows perl.exe printing @ARGV; given: a b, q"x, c\\, the empty string, x "y" z, tail\, d\"e, %PATH%, a&b):

| Started by | The child's @ARGV | |---|---| | Perl's own system LIST | two arguments: [a b] [qx c\\ " x y z tail\ d"e %PATH% a&b] (quotes taken as quoting, the empty one gone; a -e program with " in it does not even compile) | | Proc::Spawner::spawn | all nine, as given |

Choices (the user, 2026-11-05), in %Proc::Spawner::WINDOWS (win_spawn.h: WIN_OPT_*), every child:

Batch files (.bat, .cmd) run (the user: most arguments can be passed safely, and the others can be recognized): through the system's cmd.exe /e:ON /v:OFF /d /c ""SCRIPT" ARGS" (not a cmd.exe found on PATH), each argument quoted for cmd as Rust's standard library does since BatBadBut (CVE-2024-24576): quoted unless it has only letters, digits and #$*+-./:?@\_; " doubled; % written %%cd:~,%, which cmd expands to a single %, so no %NAME% is expanded; a trailing \ quotes it too. Refused (EINVAL): an argument with CR or LF (cmd ends the line there), a script path with " or ending in \. The script sees each argument as cmd shows it (%1 with the quotes, %~1 without). Under Wine, & | < > and quotes arrive literally and nothing else runs; % does not (Wine's cmd reads %% on a command line as one %, real cmd does not); not checked on real Windows.

Unicode: the child is started with CreateProcessW (UTF-16 command line, environment and paths), Perl's own with CreateProcessA (the ANSI code page). A string Perl holds as UTF-8 therefore reaches the child as it is, whatever the code page; one it holds as bytes is read in the ANSI code page, as Perl's spawn reads it.

Windows versions: the explicit handle list needs Vista; Windows 7's console handles are pseudo-handles that the list refuses, so they are left out of it (the child reaches the console through STARTUPINFO); nested job objects need Windows 8. Not tested before Windows 10 (Wine 9 imitates 10).

The kept process handle is released once the loop has the child's status (post-0.08: reap calls Proc::Spawner::reaped, which calls __win_spawn_forget; since post-0.10 start's child watch does; a caller of the raw spawn who waits itself calls reaped(PID)). Checked under Wine with Strawberry Perl 5.38.2: one child's handle is gone after its status (0.08: still open), and 1,100 children in a row all give their status. With 1,100 children started before any is waited for, all ended by then, 0.09's fixed table of 1,024 kept handles lost 76 statuses (the children whose handles did not fit were closed at once; known issue X34); since post-0.09 the table grows, and none is lost (checked under Wine both ways). rlimit, sigignore, nice fail there with ENOTSUP, or are dropped.

Done post-0.09: Win32::SetChildShowWindow is honoured as by Perl's own spawn (STARTF_USESHOWWINDOW and wShowWindow from Perl's w32_use_showwindow, w32_showwindow; checked under Wine with a child that prints its startup info: the same through both primitives, before, after SetChildShowWindow(0) and (7), and after resetting it); more than 64 children at once (the loop polls each child, so Windows' limit of 64 handles per wait does not apply: 100 at once checked; Perl's own spawn, system1, still stops at 64, Perl's table); building through Makefile.PL (no POSIX probes there; checked under Wine with Strawberry Perl's own gcc and gmake: perl Makefile.PL, gmake without a warning, gmake test of t/windows.t and t/parsers.t).

Done post-0.10, W5: the loop is Proc::Spawner::Loop, with a Windows backend on an I/O completion port; how it works and what was checked: pspawn-loop.md. The shell layer on top runs there too since post-0.10: pspawn-shell-windows.md.

Proc::ShellSpawn on Windows, before its port (the port came later on 2026-11-09: pspawn-shell-windows.md).** What runs there: Proc::Spawner (the primitive, win_spawn.h), the loop (pspawn-loop.md), the parsers, and the programs Proc::ShellSpawn runs inside Perl (X21: echo, printf, test, [, true, false). The shell layer itself did not, before the port: found under Wine on 2026-11-09, when the snippet library was to run there (large-subset/large-subset-state.md, section 7, "W5"), every run() died at once, first at getppid (for $PPID; Windows Perl has none), then, past that in a scratch copy, in Proc::Spawner::Graph at fcntl with F_GETFL (a handle's mode), and the graph's pipes were made as FIFOs and reopened through /proc, which Windows lacks too (the port takes $PPID from NtQueryInformationProcess and does without fcntl and /proc there). Under Wine with Strawberry Perl 5.38.2.2, on the code of post-0.10, these pass: t/loop.t, t/windows.t, t/parsers.t, t/programs.t (skipped there: Ctrl-C without a console, the comparisons with dash, bash and coreutils), and the large parser over all 45,501 extracted corpus samples gives the same trees and rejections as on Linux.

Still not done: the inheritable copies of handles exist during the call, so another thread's CreateProcess with inheritance could take them (one thread assumed).

While this process's fds or directory are changed, every signal is blocked, so no handler of ours sees them; the child's mask is set by the primitive. If sigmask is missing too, the signals cannot be blocked without the child inheriting that: then a handler running in that moment would see the changed fds. Tests: t/missing-matrix.pl runs the whole suite once per entry of the bitmap, then with all emulable settings missing, with the Windows column's gaps, and with everything missing.