NAME

Proc::Queue - limit the number of child processes running

SYNOPSIS

use Proc::Queue size => 4, debug => 1;

package other;

# this loop creates new childs, but Proc::Queue makes it wait every
# time the limit (4) is reached until enough childs exit
foreach (1..10) {
  my $f=fork;
  if(defined ($f) and $f==0) {
    print "-- I'm a forked process $$\n";
    sleep rand 5;
    print "-- I'm tired, going away $$\n";
    exit(0)
  }
}

Proc::Queue::size(10); # changing limit to 10 concurrent processes
Proc::Queue::trace(1); # trace mode on
Proc::Queue::debug(0); # debug is off
Proc::Queue::delay(0.2); # set 200 miliseconds as minimum
                         # delay between fork calls

package other; # just to test it works in any package

print "going again!\n";

# another loop with different settings for Proc::Queue
foreach (1..20) {
  my $f=fork;
  if(defined ($f) and $f==0) {
    print "-- I'm a forked process $$\n";
    sleep rand 5;
    print "-- I'm tired, going away $$\n";
    exit(0)
  }
}

1 while wait != -1;

DESCRIPTION

This module lets you parallelise a perl program using the fork, exit, wait and waitpid calls as usual and without the need to take care of not creating too many processes that could overload the machine.

It redefines perl fork, exit, wait and waitpid core functions. Old programs do not need to be modified, only the use Proc::Queue sentence is needed.

Additionally, the module have two debugging modes (debug and trace) that seem too be very useful when developing parallel aplications:

debug mode:

when active dumps lots of information about processes being created, exiting, being caught be parent, etc.

trace mode:

just prints a line every time one of the fork, exit, wait or waitpid functions is called.

It is also possible to set a minimun delay time between calls to fork to stop consecutive processes for starting in a short time interval.

Child processes continue to use the modified functions, but their queues are reset and the maximun process number for them is set to 1 (childs can change their proccess queue size themselves later).

Proc::Queue doesn't work if CHLD signal handler is set to IGNORE. You have to call wait or waitpid to get rid of zombie processes even if you are not interested in their exit status.

EXPORT

This module redefines the fork, wait, waitpid and exit calls.

EXPORT_OK

Functions fork_now, waitpids, run_back, run_back_now, all_exit_ok, running_now, system_back and system_back_now can be imported. Tag :all is defined to import all of them.

FUNCTIONS

There are several not exported functions that can be used to configure the module:

size(), size($number)

If an argument is given the maximun number of concurrent processes is set to it and the number of maximun processes that were allowed before is returned.

If no argument is given, the number of processes allowed is returned.

delay(), delay($time)

delay lets you set a minimun time in seconds to elapse between every consecutive calls to fork. This is usefull for not creating too many processes in a short time.

If Time::HiRes module is available it is used and delays shorted that 1 second are allowed.

If no arg is given, the current delay is returned.

To clear it use Proc::Queue::delay(0).

debug(), debug($boolean), trace(), trace($boolean)

Change or return the status for the debug and trace modes.

Other utility subroutines that can be imported from Proc::Queue are:

fork_now()

Sometimes you would need to fork a new child without waiting for other childs to exit if the queue is full, fork_now does that. It is exportable so you can do...

use Proc::Queue size => 5, qw(fork_now), debug =>1;

$f=fork_now;
if(defined $f and $f == 0) {
    print "I'm the child\n"; exit;
}

waitpids(@pid)

Will wait for all the processes in @pid to exit. It returns an array with pairs pid and exit values (pid1, exit1, pid2, exit2, pid3, exit3,...) as returned by individual waitpid calls.

run_back(\&code), run_back { code }

Runs the argument subrutine in a forked child process and returns the pid number for the new process.

run_back_now(\&code), run_back_now { code }

A mix between run_back and fork_now.

system_back(@command)

Similar to the system call but runs the command in the background and waits for some childs to exit first if there are already too many childs running.

system_back_now(@command)

As system_back but without checking if the maximun number of childs allowed has been reached.

all_exit_ok(@pid)

Do a waitpids call and test that all the processes exit with code 0.

running_now()

Returns the number of child processes currently running.

import(pkg,opt,val,opt,val,...,fnt_name,fnt_name,...)

The import function is not usually explicitally called but by the use Proc::Queue statement. The options allowed are size, debug and trace and they let you configure the module instead of using the size, debug or trace module functions as in...

use Proc::Queue size=>10, debug=>1;

Anything that is not size, debug or trace is expected to be a function name to be imported.

use Proc::Queue size=>10, ':all';

BUGS

Proc::Queue is a very stable module, and no bugs have been reported in a long time.

Child (forking) behaviour althought deterministic could be changed to something better. I would accept any suggestions on it.

SEE ALSO

perlfunc(1), perlipc(1), POSIX, perlfork(1), Time::HiRes, Parallel::ForkManager. The example.pl script contained in the module distribution.

AUTHOR

Salvador Fandiño <sfandino@yahoo.com>

COPYRIGHT AND LICENSE

Copyright 2001, 2002, 2003 by Salvador Fandiño

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

cpanm Proc::Queue

perl -MCPAN -e shell
install Proc::Queue