ShellSpawn: running ShellSubset parse trees with posix_spawn (2026-11-01)

Status: written 2026-11-01; sections 2, 7, 8 and 9 describe the state of that day (section 6 is kept current). The current state is in pspawn-spawner.md and in the modules' POD.

Code (today's files): the executor lib/Proc/ShellSpawn.pm, the large subset and the extension points (EXPERIMENTAL) lib/Proc/ShellSpawn/Large.pm, described in pspawn-extensions.md; the posix_spawn binding Spawner.xs, part of lib/Proc/Spawner.pm (starting children; the event loop is lib/Proc/Spawner/Loop.pm since W5) and lib/Proc/Spawner/Graph.pm (the executable spawn graph), both EXPERIMENTAL and described in pspawn-spawner.md; t/try.pl; t/dash.t compares with dash (in place of the old compare.pl) on battery.txt, fdbattery.txt and the other batteries, with the helper t/fds; hand-written graphs: t/sgraph/, described in pspawn-graph.md; tests of the two modules: t/spawner.t. Build: perl Makefile.PL && make. Run: TRACE=1 perl t/try.pl 'COMMAND'. (On 2026-11-01 these were files of the old Project: an XS binding PosixSpawn, lib/ShellSpawn.pm, compare.pl, hyg.pl.)

1. The scenario

As decided on 2026-11-01: this module is the only code that starts children of the process; it may set SIGCHLD as it likes; posix_spawn is available. So: no fork in Perl, no wraparound, waitpid(-1, WNOHANG) is allowed (every child is ours).

2. The posix_spawn binding

Neither CPAN module exposes file actions: Proc::FastSpawn has spawn, spawnp and fd_inherit; POSIX::RT::Spawn has one spawn like system LIST. Hence PosixSpawn (XS, ~150 lines):

PosixSpawn::spawn(FILE, \@ARGV, \@ENV_OR_UNDEF, \@ACTIONS, \%ATTR) (today Proc::Spawner::spawn; the full list of its attributes is in lib/Proc/Spawner.pm's POD) returns the pid, or the empty list with $!. Actions in order: [dup2 => FROM, TO], [close => FD], [open => FD, PATH, FLAGS, MODE], [chdir => DIR], [fchdir => FD], [closefrom => FD] (the last three are glibc extensions). [tcsetpgrp => FD] (glibc). Attributes: search (posix_spawnp), sigdefault => [SIGNUMS], sigmask => [SIGNUMS], pgroup => PGID, setsid => 1, resetids => 1, sched => [POLICY, PRIORITY], cgroup => FD (glibc), pidfd => 1. With pidfd, glibc 2.39's pidfd_spawn is used and spawn returns (PID, PIDFD).

PosixSpawn::fpe_at_exec() says what Perl's own exec would give a child for SIGFPE: 1 ignored, 0 default, -1 this Perl leaves SIGFPE alone. Perl ignores SIGFPE for itself at startup (PERL_FPU_INIT) and, around exec, restores the disposition it found then (PERL_FPU_PRE_EXEC, the global PL_sigfpe_saved).

Facts checked on glibc 2.39 (Linux):

3. Objects

4. From a command to posix_spawn

For a simple command:

  1. Expand the settings (NAME=value, left to right, each seeing the ones before), then the words. A $( … ) runs first, asynchronously.
  2. Apply the redirections to a copy of the context's map, in the order written, in the parent: n>&m copies entry m, n>&- sets undef, n>file opens the file in the parent (sysopen, close-on-exec) and puts the new end in entry n. So 2>&1 >f and >f 2>&1 come out as in sh, by construction.
  3. Builtins (:, exit, break, continue, read) run in the parent with that map. Otherwise: find the program in PATH, and spawn.
  4. File actions from the map (Proc::Spawner::actions() since 2026-11-02, on fd numbers; the graph leaves out a parent fd that a child gets under its own number): for every target fd t whose end has parent fd f ≠ t, dup2(f, t); for undef, close(t). A source fd f that is itself a target with another end would be overwritten before its turn: it is first copied to a scratch fd above every fd involved, and the scratch fds are closed at the end. A target whose end is already at the same number gets dup2(t, t) only if the parent's fd is close-on-exec (to clear the flag). Then chdir if the virtual current directory is not the real one; sigmask => []; sigdefault = the signals the Perl program has set to IGNORE since the module was loaded.
  5. Release the command's own opened files (their creator's holds), then the contexts in $tail.

Example from traces.txt: echo $x 3>&1 1>&2 2>&3 >/dev/null gives the map {1: '/dev/null' at parent fd 3, 2: our fd 1, 3: our fd 1}. Both parent fds 3 and 1 are targets with other ends, so: dup2 3->10, dup2 1->11, dup2 10->1, dup2 11->2, dup2 11->3, close 10, close 11.

5. When the parent forgets an end

Every construct is run as x_*(context, node, $tail, $k): $k is the continuation that gets the status; $tail lists the contexts to release right after the last process the construct can start has started. That is the rule of 2026-10-30: forget an end once all processes that could use it have started.

Example (traces.txt), { false || cat /etc/hostname; } | wc -l >/dev/null: the read end is closed right after wc starts; the write end stays open while false runs (because cat may follow) and is closed right after cat starts.

6. The loop

One loop, loop_until(DONE) (since 2026-11-02 Proc::Spawner::wait_until; since W5 the loop is the module Proc::Spawner::Loop, pspawn-loop.md): runs queued continuations; reaps with waitpid(PID, WNOHANG) for each of its own children (until 2026-11-01 waitpid(-1, WNOHANG), which took other code's statuses); then selects on a self-pipe written by the SIGCHLD handler (installed with SA_RESTART while the loop runs, the previous handler restored after) and on the fds being read for $( … ) and read. Since 2026-11-01 the byte is written by a C handler in the binding (Proc::Spawner::chld_to_fd, until 2026-11-02 PosixSpawn::chld_to_fd; decision D1, endorsed experimentally by the user on 2026-11-02, pspawn-extensions.md), so select waits without limit. The fallback, used only without that function, is a deferred Perl handler with select waking every 50 ms ($Proc::Spawner::Loop::POSIX::TICK): an immediate handler corrupted Perl's heap, and a deferred one can miss a child that ends just before select starts waiting (known issue X14; perl-signals-findings.md, S1–S3). The final fix is decision D1 on the page "Child Exit Wake-up". Every continuation is queued, never called from another one, so a loop of builtins (while :; do :; done) runs in constant memory. Each on_read is one read; the callback asks again. read reads one byte at a time, so later commands get the rest of the input, as in sh.

run(STRING) returns when the foreground commands have finished. A background chain started by it goes on at once, also between run calls (since post-0.04, the user's choice B of question K3, 2026-11-03): while background jobs remain, Proc::ShellSpawn keeps a deferred SIGCHLD handler installed, which calls Proc::Spawner::poll() (what is ready now, without waiting) when a child ends; it keeps the program's $? and $! as they were (X55). The costs, documented: while such jobs remain, the program's own sleep and select return early whenever a child ends (they are interrupted by the signal; doc/perl-signals-findings.md, the consequence of item L5); a program's own $SIG{CHLD} code is called after ours, and a $SIG{CHLD} set later replaces ours (the jobs then go on only in later run calls). A chain that waits for Perl to read a pipe ($( ), read) may still stall if the writing child fills the pipe and so never ends. At exit (since post-0.12, K5, the user, 2026-11-15: "as shell") an END block waits only while ours has something left to do for a job: a command still to start (ours starts each command of a job itself, without fork: the next one when the one before ends), a read or write of its own, a call queued; then Perl exits and the job's running processes go on, as the shells leave theirs. Until post-0.12 it waited for every job to end. PROC_SHELLSPAWN_END (or $Proc::ShellSpawn::END_WAIT at run time; a stopgap until such choices are arguments of the API) is MODE[:SECONDS]: shell (the default), all (wait for every job to end, as before), none (exit at once: a job's commands not yet started never start); SECONDS caps the wait. What remains unlike the shells: they exit at once and leave a copy of themselves to start the rest of a job, so ours exits later by as long as a job needs before its last command starts ((sleep 60; reboot) &: 60 s), and a job that never runs out of commands to start (while :; do date; sleep 60; done &) keeps ours from exiting unless SECONDS is set. Without fork ours cannot leave a copy of itself; t/at-exit.t tests the modes; over the library's 44 samples with a background job, all came no closer to the shells than shell in any, and worse in one (section 7 of large-subset/large-subset-state.md, "Ours at exit"). Until 0.04 the chains went on only inside run and at exit (known issue X1).

7. Tested (2026-11-01)

Sections 7 and 8 are of 2026-11-01. The current tests: t/dash.t (with the known differences in t/dash-known.txt), t/memory.t and the other files of t/; what did not pass at each level of the spawn primitive: test-results/.

8. Not done, or different (the current list: pspawn-known-issues.md)

9. posix_spawn's features: what is used (2026-11-01)

| Feature | Used | Why, or why not | |---|---|---| | adddup2 | yes | every pipe end and redirection; scratch fds for conflicts (section 4) | | addclose | yes | n>&-, and the scratch fds | | addopen | no | the parent opens files itself: it can then say which redirection failed, and the commands of a group share one open file | | addchdir_np | yes | when the virtual current directory (after cd in a subshell) is not the real one | | addfchdir_np | no | would survive a renamed or removed directory (known issue X9) | | addclosefrom_np | no | sh passes fds it inherited without close-on-exec, and so do we | | addtcsetpgrp_np | no (bound) | job control only; fails with ENOTTY without a terminal | | setsigmask | yes | always empty | | setsigdefault | yes | SIGFPE unless it was ignored when Perl started (as Perl's own exec does); every signal Perl code has set to IGNORE since the module was loaded; SIGINT and SIGQUIT for foreground children only | | setpgroup | no (bound) | a non-interactive sh puts everything in one group; needed to kill a whole background job | | RESETIDS, setsid, scheduling, setcgroup_np | no (bound) | sh does not change them either; tested in the binding | | pidfd_spawn (glibc 2.39) | no (bound) | a pollable fd per child would replace the SIGCHLD self-pipe; but the child is still an ordinary child, so other code's waitpid(-1) can still take its status | | environment (envp) | yes | the exported variables (every name from %ENV that is an identifier, as dash; PWD kept up to date by cd) plus the command's own settings |

Signal dispositions, checked against dash with sigenv.txt (11 of 12 agree; the 12th is umask, outside the subset): a foreground child has nothing ignored or blocked; a background child, also inside a subshell or a pipeline, has SIGINT and SIGQUIT ignored; a signal ignored when Perl started stays ignored, SIGFPE included (checked both ways with trap '' FPE in the parent of Perl). While run() runs, Perl itself ignores SIGINT and SIGQUIT, as Perl's system does.

Not set through posix_spawn, inherited as in sh: umask, resource limits, nice value, process group and session, controlling terminal.