NAME
Ekorn::Pipeline::Gather - a stage made afresh for each run, told when its values run out
SYNOPSIS
use Ekorn::Pipeline::Gather;
# Values in batches of $size, the last batch given at the end
sub batches_of ($size) {
return gather {
my @batch; # this run's
return (
sub ($x) { push @batch, $x; @batch == $size ? [splice @batch] : () },
sub () { @batch ? [@batch] : () },
);
};
}
my $next = Ekorn::Pipeline->new(batches_of(500))->iterator($source);
while (my($batch) = $next->()) { ... }
DESCRIPTION
A plain stage, a code reference, is called with each value in turn. It can't tell when the last value has gone by. Whatever it keeps in a closure stays there from one run of its pipeline to the next. A stage that gathers values, to give them in batches or all together, needs both: fresh state on each run, and a call at the end.
gather makes such a stage. Its block is the stage's set-up, called at the start of each run, so whatever the block declares is fresh each time. A run is a call of a pipeline with values, or an iterator or a feed made from it. The block returns:
the step: a code reference, called with each value as any stage is, or a pipeline;
optionally, the flush, called with no arguments, and
$_undefined, once its input has run out;optionally, the end, called once, when the run has ended, however it ended, told how, as "Ends" describes.
It returns them as a list, in that order, giving undef for the flush to have only an end, or by name, as a hash reference, as in {step => $step, end => $end}: "Made for a run" in Ekorn::Pipeline has how.
What the flush returns goes through the rest of the stages, just like what a step returns: nothing, one value, or several. It's called even when there were no values, so a flush can give an empty result, such as []. Its input runs out when the values run out, or after a stop in a stage before it.
The flushes are called in order, first stage first. What one flush returns goes through the stages after it before the next flush is called, so a flush later in the pipeline gets everything sent to it.
Make any stage that keeps state, such as a count, with gather, even without a flush, so that it starts afresh on each run.
The flush is what v0.2.0 called the end function. Its place in what the block returns hasn't changed, so a gather written for v0.2.0 works as it did.
Ends
The end is for what must be done however the run ends, such as closing a file or committing a transaction. It's called once the run has ended, after every flush, and is given how the run ended, from where the stage stands:
died, if the run died, anywhere;stopped, if its input was cut short, by a stop at or after it, or by the run being ended from outside, as "finish" in Ekorn::Pipeline::Iterator does;done, if its input ran out.
What it returns is dropped. The ends are called in order, the source's first, then each stage's, first stage first. An end that dies, in a run that hadn't died, ends the run with its error, and every end after it is told died. In a run already dying, it's reported as a warning, naming its stage, and the first error is kept. "How a run ends" in Ekorn::Pipeline has the rest.
A pipeline as the stage
A pipeline returned as the stage has its stages made for the run in its place, as if they had been given to the pipeline there. A gather stage inside it is made afresh too, and a stop inside it ends the whole run. So the block can build a pipeline for each run, or choose one:
# The first $n values of each run, numbered from 1
sub first_numbered ($n) {
return gather {
my $i = 0; # this run's
Ekorn::Pipeline->new(
sub { ++$i . ". $_" },
take($n),
);
};
}
The block's flush, if it returns one, is called after the flushes of the pipeline's own stages, and its end after their ends. A stage inside the pipeline is named in messages by its position in each, as #2.3 for the third stage of the pipeline returned by stage #2.
Stopping
After a Ekorn::Pipeline::Stop, the flushes of the stages after the one that stopped are called. They get the values from before the stop, and any it carries. The flushes of the stage that stopped, and of the stages before it, aren't called, as nothing else is processed. Their ends are, told stopped, and the ends of the stages after it are told done.
A flush can stop the pipeline too. The values its stop carries go on, and the flushes after it are still called.
Context
Inside a flush, and the stages it sends values to, "ending" in Ekorn::Pipeline is true. Code that tells its stages where a value came from can use it to say the value comes from many, not one.
FUNCTIONS
gather
my $stage = gather { ...; return ($step, $flush, $end) };
Returns a stage that a pipeline makes afresh for each run, by calling the block. It's an Ekorn::Pipeline::Gather, which can be called as a code reference, so it can go wherever a stage can. gather is exported by default. To keep it out of your namespace:
use Ekorn::Pipeline::Gather ();
METHODS
new
my $stage = Ekorn::Pipeline::Gather->new($setup);
Returns a gather of $setup, a code reference, as gather does. Anything else croaks. Called as a code reference, a gather calls its set-up.
EKORN_STAGE
my @parts = $stage->EKORN_STAGE;
Calls the set-up, for a new run, and returns what it returns: the parts, as a list or by name. A run calls it as it starts; you needn't.
DIAGNOSTICS
- Expected a code reference
-
gatherwas given something other than a block or code reference. - Expected stage #%s, made with gather, to return a code reference or a pipeline, and optionally a flush and an end
-
A block given to
gatherreturned something else. It's reported when a run starts: when the pipeline is called with values, or an iterator is made from it. The stage is named by its position, as "DIAGNOSTICS" in Ekorn::Pipeline describes.
SEE ALSO
Ekorn::Pipeline, Ekorn::Pipeline::Stop, and the canned stages, many made with it: Ekorn::Pipeline::Stages::Flow, Ekorn::Pipeline::Stages::Batch and Ekorn::Pipeline::Stages::Taps
AUTHOR
Iain Campbell <cpanic@cpan.org>
LICENSE
Copyright (c) 2024-2026 by Iain Campbell.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.