NAME

Thread::DependentJob - Run interdependent jobs using a pool of worker threads

SYNOPSIS

use Thread::DependentJob;
use Thread::Semaphore ();

my $jobsch = Thread::DependentJob->new();

sub do_work_func(...) {
  ... consume whatever is represented by input tokens ...
  ... produce whatever is represented by output tokens ...
  return ...;
}

my $sema = Thread::Semaphore->new(3);

$jobsch->sched(subname => "do_work_func",
               args    => [arguments for &do_work_func],
               inputs  => [input tokens],
               outputs => [output tokens]),
               semas   => [$sema],
               context => "list"|"scalar"|"void",
              );
$jobsch->sched( ... );
$jobsch->sched( ... );

my $results = $jobsch->wait();
foreach my $jobid (keys %$results) {
  print "Job $jobid returned ", $results->{$jobid}, "\n";
}

DESCRIPTION

This package allows self-contained "jobs" to be executed concurrently, initiated using an arbitrary graph of dependencies.

A single pool of "worker" threads is used to execute all jobs.

Each job has sets of input and output "tokens". A job will not start until its input tokens are available, and upon termination (normal or abnormal), it's output tokens are produced.

Jobs may also be regulated by semaphore(s), which are "downed" when the job starts and "upped" at completion.

NOTE: Jobs may not depend on each other while running, i.e., once started a job must complete regardless of what other jobs do. The only way to coordinate jobs is for a common predecessor job (or the main program) to schedule the jobs with appropriate token dependencies.

See perldoc threads for limitations of the thread model. In particular, chdir is global and affects all threads, and system et. al. always use the main thread's %ENV.

Aborting Jobs

Jobs may terminate abnormally by calling die() in the usual way. Jobs should never call exit(), as the entire process will terminate.

If a job dies or is aborted (see abort_job), it's output tokens are immediately produced but marked "poisoned", and other jobs which depend on them will be cancelled.

$jobsch = Thread::DependentJob->new(OPTIONS);

Create a job scheduler object. Options are key => value pairs.

OPTIONS

NUMTHREADS => $number
NUMTHREADS => 0

Set the number of worker threads. The default is 1 greater than the number of CPUs.

If zero is specified then concurrent threads are not used and jobs are executed sequentially when $jobsch->wait is called. This is useful for debugging.

Configure global options may also be passed to new, but still affect all Job Schedulers.

$jobid = $jobsch->sched(PARAMETERS)

($jobid) = $jobsch->sched(PARAMETERS)

Schedule a job to run when specified inputs and semaphores are available and which produces specified outputs (with no inputs or semaphores the job can can start immediately).

PARAMETERS are key => value pairs:

subname => "name" (required)

The name of the function to be called on a worker thread (a sub ref will not work). It must be in the caller's package unless package:: qualifiers are included.

The function will be called as follows:

name(args => [args], inputs => [inputs], outputs => [outputs],
     js => job_scheduler_object)

The first three parameters are the corresponding sched parameter values (actually the results of calling shared_clone).

The context of the function call (list, scalar, or void) is determined by the 'context' parameter, if specified, otherwise the context of the call to $jobsch->sched(...). This contols how return values are handled, see <wait>.

args => [...]

"User" arguments for the function.

inputs => [token,...]

The tokens, if any, which must be produced before this job can start.

outputs => [token,...]

The tokens which are implicitly produced when this job finishes.

semas => [SEMSPEC,...]

Semaphores which are all "downed" when the job starts, deferring the job until that is possible; and "upped" when the job completes.

Each SEMSPEC may be just a Thread::Semaphore object ref (which will be downed/upped by 1) or a tuple [semaphore, down_up_count] or [semaphore, down_count, up_count].

context => "list" | "scalar" | "void"

The context to use when calling subname. In the function, wantarray will be true, false, or undefined respectively.

If not specified, "list" context is assumed if $jobsch->sched() is called in list context, otherwise "scalar" is assumed.

Return value(s) are saved and returned in the results from <wait>.

RETURNS: A Job ID identifier string. Job IDs are never recycled.

The job will run in an indeterminate thread, possibly in the main thread when $jobsch->wait() is called.

Tokens

Tokens are arbitrary strings. For example, if a job writes a file to be read by subsequent job(s), the pathname might be used as the corresponding token.

Tokens may be produced only once but consumed any number of times. That is to say: They may (actually: must) be an output token for a single job but may be an input token for any number of dependent jobs.

Tokens may (but need not) be explicitly forgotten when no longer needed using delete_tokens.

Semaphores

Semaphores are used to allocate "reusable" resources to jobs.

For example, you might want to limit the number of disk-intensive jobs to prevent thrashing. This could be done using a semaphore initialized to the maximum concurrency allowed, and specifying that semaphore when scheduling disk-intensive jobs.

A job can start only when its semaphore(s) can all be immediately "downed" by 1 (or by the indicated count) without becoming negative. When a job finishes, it's semaphores are "upped" (by 1 or the indicated count).

$results = $jobsch->wait();

Wait until all jobs (managed by Thread::DependentJob instance $jobsch) have completed.

If called in void context (i.e. the results will not be examined), then warnings are issued about any jobs which aborted. Silent if called in non-void context.

The return value is a ref to a hash with these members:

jobid => {RESULT => value | [list of values],
          ABORTMSG => "...",
          EXCEPTION => anything,
         }

RESULT gives the return value from the function specified when scheduling the job. If the job had "list" context (see sched), the RESULT value will be an array reference to the values returned. If the job had void context RESULT will be absent.

ABORTMSG will be an explanatory string if the job terminated abnormally for any reason. The reasons include:

1)

The job called die: String(s) passed to 'die' will be included in ABORTMSG. EXCEPTION will contain exactly $@, or rather a shared_clone of it (could be an object or unblessed container).

2)

The job depended on a poisoned input. This occurs if an 'input' token was to be produced by a precursor job which itself aborted.

3)

abort_job() was called for the job.

If jobs schedule other jobs, $jobsch->wait() will wait until all of them complete.

$jobsch->abort_job($jobid, [$msg]);

Nothing is done if the job has already completed. Otherwise the job is killed if running or cancelled if not yet started, and ABORTMSG set in the job's result hash (see wait). If $msg is provided it will be included in the ABORTMSG string.

Returns true if the job was cancelled or killed, false if the job was already complete.

$jobsch->delete_tokens(token)

$jobsch->delete_tokens([token, ...]

Tokens must be unique and by default may not be re-used. A fatal error occurs if an attempt is made to produce the same token twice.

delete_tokens may be called to forget tokens which can no longer influence any scheduled job; thereafter the same identifier strings may be used for new tokens. It is not necessary to do this unless the same token string must be re-used, or to reduce memory usage.

A running job may delete it's own input tokens, but otherwise tokens used by incomplete jobs may not be deleted.

In practice, delete_tokens may only be by called by the main thread after return from $jobsch->wait, or from a job which consumes (specifies as it's inputs) an output token from every other job which might not yet be complete.

@jobids = $jobsch->all_jobids()

Returns the identifiers for all jobs which have not been "reaped" by calling $jobsch->wait().

$jid = my_jobid()

Returns the current job's jobid, or undef if not called from a running job.

$jobsch->delete()

Waits for any incomplete jobs to finish, and issues warnings about failed jobs (see wait). Then internal references are broken, allowing the job-scheduler object to be destroyed.

This is not normally necessary, as it is done implicitly at process exit.

Thread::DependentJob->Configure(global_options...)

Set global options, which affect all job scheduler instances. These options may also be passed to Thread::DependentJob->new but affect all instances.

ALLSERIAL => $boolean

If true, NUMTHREADS defaults to zero (see sched).

DEBUG => $boolean

If true, trace messages are printed.