/ 
Schedule-Parallel-1.1
/ Schedule::Parallel::Thread

NAME

Schedule::Parallel

SYNOPSIS

  use Schedule::Parallel;

  @unused_portion =  
	  runqueue( jobcount, closure, [closure ...] );

DESCRIPTION

Fork with a bit of boilerplate and a maximum number of jobs to run at one time (jobcount). The queue (whatever's left on @_ after shifting off the count) is run in parallel by forking perl and exiting the sub-process with the closure's status.

The caller gets back the unexecuted portion of the queue. In a scalar context this will return false if the entire execution succeeded; in an array context it returns the unused portion for your money bac... er, in order to simplify re-execution where the calling code can fix the problems (e.g., if the closures store recovery information).

fork + exit semantics require that code called from the closures exits zero if it succeeds (i.e., shell-like returns). Any non-zero exit from a forked job will abort further processing.

Notes

Running N jobs in parallel makes the assumption that jobs will consume roughly constant system resource during execution. If this is not true it may be useful to submit a large que in sections with some monitoring to adjust the jobcount parameter as pieces of the queue are sumitted. Examples would be raising the value of jobcount for long-running, mostly-blocked queues (e.g., web searches) or reducing it to avoid bombarding the network if return times are faster.

The leading newlines may look a bit odd but help separate status notices from any sub-proc output that they get buried in. Ditto the leading "$$" on all of the messages.

One minor modification to the arguments improves the logging. Instead of passing in an array of closures, use an array of arrays, with the title and a closure. The title is printed, the closure is executed. In cases where the called subroutine doesn't give sufficient logging information this might help. The required change is simply:

my $item = shift;
my ( $title, $morsel ) = @$item;

and print $title in the status messages:

"\n$$: Forked $forkfull running $title\n";

and

"\n$$: Running $title";

Should provide sufficient logging information for most purposes.

AUTHOR

Steven Lembark Knightsbridge Solutions, LLC

slembark@knightsbridge.com

Copyright

(C) 2001-2002 Steven Lembark, Knightsbridge Solutions

This code is released under the same terms as Perl istelf. Please see the Perl-5.6.1 distribution (or later) for a full description.

In any case, this code is release as-is, with no implied warranty of fitness for a particular purpose or warranty of merchantability.

SEE ALSO

perl(1).