NAME

AnyEvent::ProcessPool::Pipeline - A simplified, straightforward way to parallelize tasks

VERSION

version 0.07

SYNOPSIS

use AnyEvent::ProcessPool::Pipeline;

pipeline workers => 4,
  in {
    my @task = get_next_task(); # list of (CODE ref $task, ARRAY ref $args)
    return @task;
  },
  out {
    process_result(shift->recv);
  };

EXPORTED SUBROUTINES

pipeline

Launches an AnyEvent::ProcessPool and immediately starts processing tasks it receives when executing the code specified by in. As results arrive (and not necessarily in the order in which they were queued), they are delivered as condition variables (ready ones, guaranteed not to block) via the code supplied by out. The pipeline will continue to run until in returns an empty list, after which it will continue to run until all pending results have been delivered. pipeline returns the total number of tasks processed.

Optionally, an existing pool may be specified using the pool parameter.

pipeline pool => $pool,
  in  {...},
  out {...};

Aside from in and out, all other arguments are passed unchanged to AnyEvent::ProcessPool's constructor.

in

Supplies the pipeline with a code ref representing the caller's task queue. When called, the code ref returns a two item list containing: 1) a code ref to be executed in a worker sub-process, and, 2) an (optional) array of arguments to be passed to the code ref when called.

The pipeline will call in until an empty list is returned, after which the pool will continue to run until all remaining tasks have completed and been passed to out.

out

Specifies the callback to which results are passed. Results may arrive in any order.

AUTHOR

Jeff Ober <sysread@fastmail.fm>

COPYRIGHT AND LICENSE

This software is copyright (c) 2017 by Jeff Ober.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.