NAME

Ekorn::Pipeline - run values through a list of stages

SYNOPSIS

use Ekorn::Pipeline;

my $p = Ekorn::Pipeline->new(
    sub { $_->{active} ? $_ : () },          # filter
    sub ($user) { @{ $user->{roles} } },     # flatMap
    sub { uc },                              # map
);

# Push: values in, a list out
my @roles = $p->(@users);

# Pull: one value per call, from a source
my $next = $p->iterator(sub { shift @queue // () });
while (my($role) = $next->()) { ... }

# A function of your own, taking stages from its caller
sub lines_of ($file, @stages) {
    open my $fh, '<', $file or die "Can't read $file: $!\n";
    return Ekorn::Pipeline->new(@stages)->iterator(sub { scalar(<$fh>) // () });
}

DESCRIPTION

A pipeline is a list of stages. A stage is a code reference, or another pipeline, which stands for its own stages (see "A pipeline as a stage").

A code reference is called with one value, both as its argument and in $_. It can name the value with a signature, sub ($row) { $row->[1] }, or use $_, sub { $_->[1] }. $_ is localised for each call, so the caller's is left alone. It returns a list:

  • nothing, to drop the value (a filter);

  • one value, to pass it on (a map);

  • several, each of which goes through the rest of the stages on its own (a flatMap).

Each value goes through every stage before the next one starts. The values come out in order, and each stage sees them in the order a for loop would.

A stage can end the pipeline early with Ekorn::Pipeline::Stop. The values already out are kept, and any values the stop carries go on through the rest of the stages. Nothing else is processed:

sub ($row) { $row->{id} == $wanted ? Stop($row) : () }

A stage made with Ekorn::Pipeline::Gather is made afresh for each run of the pipeline. It can also have an end function, called once the values run out, whose values go through the rest of the stages. Each call of a pipeline with values is a run, and so is each iterator.

Stages from your caller

A pipeline is for the author of any function that hands out values. The function takes stages from its caller, as its last arguments, and runs what it would return through them. The caller says what they want done with each value, beside the call, and gets back only what they want:

sub bands (@stages) {
    return Ekorn::Pipeline->new(@stages)->(load_bands());
}

my @names = bands(sub { $_->{name} }, take(5));

With "iterator", the values are made one at a time, as the caller asks for them. A function reading a file or a socket reads no more than it has to. Once a stage has stopped the pipeline, the source isn't called again.

The canned stages, in Ekorn::Pipeline::Stages::Flow, Ekorn::Pipeline::Stages::Batch and Ekorn::Pipeline::Stages::Taps, work in any pipeline. Ekorn::Pipeline::Pipes gives a class of your own pipe and tap methods, which run an object through stages.

METHODS

new

my $p = Ekorn::Pipeline->new(@stages);

Returns a pipeline that runs values through @stages, in order. Each stage must be a code reference or a pipeline (see "A pipeline as a stage"); anything else croaks. With no stages, a pipeline returns its values unchanged.

A pipeline is an object that can be called as a code reference. Called with values, it returns the list that comes out of the last stage:

my @results = $p->(@values);

In scalar context, it returns how many results came out, as an array would:

my $count = $p->(@values);

A pipeline as a stage

You can give new a pipeline wherever it takes a stage. Its stages are spliced in where it stands, so these two run the same:

my $clean = Ekorn::Pipeline->new(sub { s( \A \s+ | \s+ \z )()xgr }, sub { length ? $_ : () });

Ekorn::Pipeline->new($clean, take(10));
Ekorn::Pipeline->new($clean->stages, take(10));

The spliced stages belong to the new pipeline's run. A take inside $clean limits the whole run, not each value.

The only difference is in error messages. Given $clean itself, a stage inside it is named by its position in each pipeline: the third stage of a pipeline given second is #2.3, and three levels down, #1.2.3. Given $clean->stages, it's named by its position in the combined list.

The stages spliced in are whatever the pipeline's "stages" method returns, so a subclass can override it. If the override returns a different list, its stages are named by their position in that list.

To run a pipeline on each value separately instead, call it from a stage: sub { $clean->($_) }. Each call is then a run of its own, and a take inside limits each call.

iterator

my $next = $p->iterator($source);

Returns a closure that hands out the values that come out of the stages, one per call. $source is a code reference that returns the next value or values each time it's called, and an empty list when there are no more. It's called only once the values from its last call have all been handed out.

The closure returns an empty list when the values run out, and on every call after that. After a stop, it returns an empty list once the values the stop carried are handed out, and the source isn't called again. Read the closure in list context, so that an undef value isn't mistaken for the end:

while (my($x) = $next->()) { ... }

The end functions of stages made with Ekorn::Pipeline::Gather are called once the source runs out, or after a stop, before the closure returns an empty list.

If a stage or an end function dies, the call dies with it, and the iterator has ended, as a pushed run does. Every call after returns an empty list. Nothing waiting goes on through the stages, no end function is called, and the source isn't called again. To skip a bad value and carry on, catch the error inside the stage.

stages

my @stages = $p->stages;

Returns the pipeline's stages, in order. A pipeline that was given as a stage appears as the stages it was made of. In scalar context, it returns how many there are.

The list can go anywhere stages go, including to a function that accepts only code references:

my $clean  = Ekorn::Pipeline->new(sub { s( \A \s+ | \s+ \z )()xgr }, sub { length ? $_ : () });
my @titles = Ekorn::Pipeline->new($clean->stages, take(10))->(@raw);

To give new the pipeline itself, see "A pipeline as a stage". Either way, the stages run as if they had been written out in full:

  • A stop in one of them ends the whole run.

  • A stage made with Ekorn::Pipeline::Gather starts afresh on each run, so pipelines that share it don't share its state.

  • A plain closure is the same code reference wherever it goes, so anything it keeps is shared. Keep state in gather instead.

  • A stage that dies is named by its position, as "A pipeline as a stage" describes.

FUNCTIONS

is_stage

use Ekorn::Pipeline v0.1.3 qw(is_stage);

is_stage($thing)

True if $thing can be used as a stage: a code reference (including a stage made with Ekorn::Pipeline::Gather), or a pipeline of this class or a subclass. It's the same test "new" uses.

Use it in a function that takes its stages from the end of its arguments, so that it accepts the same stages new does, pipelines included:

sub fetch ($url, @args) {
    my @stages;
    unshift @stages, pop @args while @args && is_stage($args[-1]);
    ...
}

It's exported on request. It's new in v0.1.3, so ask for that version on the use line, as above: v0.1.2 has no import, so without the version, use Ekorn::Pipeline qw(is_stage) compiles under it, and fails only when is_stage is called.

A pipeline is a blessed hash, so is_hashref is true of one too. A function that takes a hash reference of options before its stages should test is_stage first, or it takes a pipeline for the options.

Exporter provides the import method, and a subclass inherits it, so use My::Pipeline qw(name) croaks unless the subclass exports name itself.

ending

Ekorn::Pipeline::ending()

True while the values going through the stages come from an end function, rather than from the source. It's for code that tells its stages more about each value, such as where it came from: a value from an end function has no such origin. It isn't exported.

DIAGNOSTICS

Expected stage #%d to be a code reference

A stage given to "new" wasn't a code reference or a pipeline. It's also raised for a pipeline whose overridden stages method returned something that isn't a code reference.

Expected a code reference

The source given to "iterator" wasn't a code reference.

Expected stage #%s, made with gather, to return one or two code references

The block given to "gather" in Ekorn::Pipeline::Gather for that stage returned something other than a stage (a code reference or a pipeline), or a stage and an end function.

Pipeline stage #%s failed at %s line %d:

A stage, or its end function, died. The message it died with follows, indented. An exception object is passed on as it is.

In this message and the one above, a stage is named by its position: #2 for the second stage, or #2.3 for the third stage of a pipeline given as the second stage.

SEE ALSO

Ekorn::Pipeline::Gather, Ekorn::Pipeline::Stop, Ekorn::Pipeline::Pipes, and DBIx::Squirrel, where the pipeline began

Ekorn::Pipeline::Manual::HowTo shows how to do common tasks, one recipe at a time, and Ekorn::Pipeline::Manual::Reference describes every module, function and method in full.

AUTHOR

Iain Campbell <cpanic@cpan.org>

LICENSE

Copyright (c) 2024-2026 by Iain Campbell.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.