Actions Status CPAN Kwalitee

Ekorn-Pipeline

Stages for any function: let the caller finish the job.

A function that hands out values can't know everything its callers will want done with them. With Ekorn::Pipeline, it takes a list of stages from its caller: small functions that each see one value at a time. It runs what it would return through them. The caller says what they want, beside the call, and the function does only what it alone can.

use Ekorn::Pipeline;
use Ekorn::Pipeline::Tooling qw(:all);

# The author's function: it reads log lines, and nothing else.
sub log_lines ($file, @stages) {
    return Ekorn::Pipeline->new(@stages)->iterator(from_file($file));
}

# A caller's use of it: the first ten errors, as their messages.
my $errors = log_lines(
    'app.log',
    sub { m( \A ERROR \  (.*) )x ? $1 : () },    # only errors, as messages
    take(10),                                    # ten, then stop reading
);
while (my($message) = $errors->()) { say $message }

The file is read only as far as the tenth error, and closed there, however the caller's loop ends.

Every stage, source, sink and word here comes from one module, Ekorn::Pipeline::Tooling, by tags, and the examples below assume it; the sections say which module makes each.

The problem it solves

Every function that returns a list meets the same pressure. One caller wants only some of the values. Another wants them in a different shape, a third wants the first ten, and a fourth wants them in batches of 500. The author has two ways to answer, and both cost something.

Stages are a third way. The function takes a list of them as its last arguments, and runs each value it makes through them. The caller's processing happens inside the function, one value at a time, as the values are made:

# Options
sub users (%opt) {
    my @users = load_users();
    @users = grep { $_->{active} } @users if $opt{active_only};
    @users = map  { $_->{name} } @users   if $opt{names_only};
    @users = splice @users, 0, $opt{limit} if $opt{limit};
    return @users;
}

# Stages
sub users (@stages) {
    return Ekorn::Pipeline->new(@stages)->(load_users());
}

my @names = users(
    take_if { $_->{active} },
    sub { $_->{name} },
    take(10),
);

The second users does everything the first does, and everything it doesn't, without a line more. Its author is done with it. Whatever a caller wants next, they can write as a stage, without asking.

What everyone gets

The shape of a run

Every run of a pipeline has the same shape. Values come from somewhere, go through stages one at a time, and end up somewhere:

  source  -->  stage  -->  stage  -->  stage  -->  sink
 (where       (change, drop, multiply,          (where they
they come      or stop: one value at a time)      end up)
  from)

The stages are the heart of it, and the only part a function that takes stages needs. The other two are optional. A source is where a pulled pipeline gets its values, such as a file, opened as the run starts and closed however it ends. A sink, given last, takes every value that comes out and gives one result, such as a count, the first value, or the number of lines written to a file. Without a sink, the run gives back whatever comes out of the last stage.

The sections below take the parts in that order, after the stages: how a pipeline runs, how a stage stops it, parts made afresh for each run, sources, sinks, and joining them with |.

Stages

A stage is most often a plain code reference. It's called with one value, as its argument and in $_, and returns a list:

One shape covers all three. A pipeline is just a list of stages, and a stage needs no wrapper to say which kind it is:

sub { $_ % 2 ? $_ : () }                # filter: the odd numbers
sub { $_ * 10 }                         # map: ten times each
sub ($user) { @{ $user->{roles} } }     # flatMap: each of a user's roles

Returning a list, rather than a value or undef, is what makes this work. An empty list plainly means "nothing", so undef, 0 and the empty string are ordinary values, and can be passed on. A test, which returns true or false, isn't a filter: give it to take_if or drop_if, from Ekorn::Pipeline::Stages::Flow, as in take_if { $_->{active} }.

A stage can also be:

So stages you've put together once can be reused, in another pipeline or with any function that takes stages:

my $active_names = Ekorn::Pipeline->new(
    take_if { $_->{active} },
    sub { $_->{name} },
);
my @names = users(
    $active_names,
    take(10),
);

If a function accepts only code references, give it the pipeline's stages instead: users($active_names->stages, take(10)).

One value at a time

Each value goes through every stage before the next one starts, as it would in a for loop. The values come out in order, and each stage sees them in order. Nothing is held between stages, so a pipeline over a million values needs no more memory than one over ten.

A pipeline runs two ways:

The simplest source is a sub that returns the next value, or an empty list when there are no more. An iterator can also give everything that's left, with all, or just the next, with first; end its run early, with finish; and say how the run ended, with ended. An iterator let go before its values run out ends its run, so a source closes what it opened.

Stopping early

A stage can end the pipeline with Ekorn::Pipeline::Stop. The values already out are kept, and any values the stop carries go on through the rest of the stages. Nothing else is made: once a stage has stopped it, a pulled pipeline never calls its source again.

sub ($user) { $user->{id} == $wanted ? Stop($user) : () }    # find one, then stop

This is how a caller's stages save the author's work. With take(10), the function in the first example reads up to the tenth error line, and leaves the rest of the file unread. Nobody wrote an option for it, and the caller wrote no last.

It matters more where each value costs more. A function over a paged web API fetches a page only when its caller wants a result from it:

sub search ($query, @stages) {
    my $page = 0;
    return Ekorn::Pipeline->new(@stages)->iterator(sub {
        return @{ fetch_page($query, ++$page) };    # a request; empty after the last page
    });
}

my $hits = search('squirrels', take(10));
while (my($hit) = $hits->()) { say $hit->{title} }    # one request, if a page holds ten

Without stages, that function either fetches every page, or grows max_results, max_pages and on_result options, for its author to write and maintain.

Parts made for each run

A stage is a closure, so it can keep state, such as a count. But a plain closure keeps its state from one run to the next, and can't tell when the last value has gone by, or how the run ended. Ekorn::Pipeline::Gather makes a stage afresh at the start of each run. Its block is the stage's set-up, and returns the stage's step, called with each value, and, if it needs them:

The parts are given by name, in a hash reference, or as a list in that order:

sub batches_of ($size) {
    return gather {
        my @batch;    # this run's
        return {
            step  => sub ($x) { push @batch, $x; @batch == $size ? [splice @batch] : () },
            flush => sub ()   { @batch ? [@batch] : () },    # the last, short batch
        };
    };
}

That's how batch, collect, take and drop are made. Sources and sinks are made the same way, from a set-up called at the start of each run, so every part of a run can have state of its own, and be told how the run ended.

Sources

A plain sub makes a fine source, but it's the same sub on every run, and it's never told when the run is over: a sub reading a file can't close it when the caller stops early. The sources in Ekorn::Pipeline::Sources are made afresh for each run, and told when it ends:

The word source, from Ekorn::Pipeline::Factories, makes your own. Its block is the source's set-up, and returns its step, which gives the next value or an empty list at the end, and, if it needs one, its end:

sub lines_of ($file) {
    return source {
        open my $fh, '<', $file or die "Can't read $file: $!\n";
        return {
            step => sub () { my $line = readline $fh; defined $line ? $line =~ s( \n \z )()xr : () },
            end  => sub ($how) { close $fh },
        };
    };
}

The set-up can also give a context, anything its stages may want to know about the run, such as a file's column names, which a stage asks for with Ekorn::Pipeline::run_context(). And for a producer that's easiest written as a loop, the word generator takes a block that calls yield with each value.

An iterator is a source too, so one pipeline's results can be read as another's values, and ending the outer run ends the inner one.

Sinks

A sink is the opposite of a source: given last in a pipeline, it takes every value that comes out, and gives one result, which is what the run gives. Ekorn::Pipeline::Sinks has them ready-made:

my $errors = Ekorn::Pipeline->new(
    take_if { m( \A ERROR )x },
    count_items,
)->(@lines);
my $first = Ekorn::Pipeline->new(
    take_if { m( \A ERROR )x },
    keep_first,
)->(@lines);

A sink that has what it needs, such as keep_first, stops the run there, so a source is read no further.

The words sink_one and sink_many make your own, giving one value or a list. The set-up returns the step, called with each value; the flush, giving the result; and optionally an end. A sink whose end is told done knows every value arrived, so it can commit a transaction, or move a file into place, only then.

Joining parts with |

With Ekorn::Pipeline::Operators in scope, sources, stages and sinks join with |, as commands do in a Unix shell. A chain from a source to a sink runs at once, and gives the sink's result:

use Ekorn::Pipeline::Operators;

my $errors = from_file('app.log') | take_if { m( \A ERROR )x };
say $errors | count_items;
my $messages = $errors
    | stage { s( \A ERROR \  )()xr }
    | take(10)
    | keep_all;
say for @$messages;

A source joined to stages, with no sink, is a new source, which can be kept and ended in different sinks, each chain a run of its own. Stages joined with no source make a pipeline, and |= adds to it. @{ $source } reads a source to its end, and gives every value.

It's a shorthand for the commonest shape, and everything it does can be done with new and iterator. Two things to mind, both Perl's: start a chain with one of Ekorn's types, a source, a stage made with stage, or a ready-made one, as sub {...} | sub {...} is Perl's bitwise OR of two addresses; and put a chain inside a larger expression in parentheses, as | binds more loosely than . and +.

Stages are values

Stages can be kept in variables, built by functions and passed around. A function that returns a stage is a word in a vocabulary, and the vocabulary grows in the caller's code, not the author's:

sub only_active ()    { take_if { $_->{active} } }
sub value_of ($name)  { sub { $_->{$name} } }

my @page  = (drop(20), take(10));
my @names = users(
    only_active,
    value_of('name'),
    @page,
);

A stage written for users works for log_lines too, or for a function on CPAN whose author has never heard of it.

A pipeline you use often can be given a name, and used by it. The word pipeline makes a pipeline in which a name stands for the pipeline it names:

Ekorn::Pipeline->named(active_names =>
    only_active,
    value_of('name'),
);
my @names = users(pipeline(
    'active_names',
    take(10),
));

A name belongs to the package that defined it, unless it's qualified, so two modules can each have their own. A stage that dies inside a named pipeline is reported with its name. Ekorn::Pipeline::Sugar names pipelines as the program compiles, and gives your package a function of each name:

use Ekorn::Pipeline::Sugar pipeline => {
    active_names => [only_active, value_of('name')],
};
my @names = active_names(@users);

Writing a function that takes stages

A class of your own can have pipe and tap methods, from Ekorn::Pipeline::Pipes, which run an object through stages. pipe returns what comes out. tap returns the object, so a chain of method calls can carry on after it.

Parts for other packages

Your own classes can be sources, stages and sinks, with no need to load Ekorn or inherit from anything. An object says what it can be by its methods, named in capitals because Ekorn calls them, not your users: EKORN_SOURCE, to be read from; EKORN_STAGE, to be a stage with state of its own; and EKORN_SINK_ONE or EKORN_SINK_MANY, to be written to. Each is called at the start of a run, and returns the parts, as a set-up does:

package My::Queue {
    sub new ($class, @items) { bless {items => [@items]}, $class }

    # Read: each item, taken off the front of the queue.
    sub EKORN_SOURCE ($self) {
        return {step => sub () { @{ $self->{items} } ? shift @{ $self->{items} } : () }};
    }
}

my @loud = Ekorn::Pipeline->new(sub { uc })->iterator(My::Queue->new(qw(a b)))->all;

The methods are the whole contract, so your package and Ekorn can each change their insides freely. A source class can also overload | and @{} with Operators' chain and drain, so its users can start a chain with it. The reference's "Making each kind of part" has what each method is called with and must return, with a worked example of each.

Errors

A stage that dies is named by its position in the pipeline, and reported at the line of the caller's code that ran it, with its message below:

Pipeline stage #2 failed at app.pl line 12:
    No name for user 7 at app.pl line 9.

A stage inside a pipeline given as a stage is numbered within it, as #2.1, and one inside a named pipeline is named with it, as #2.1 (main::clean#1). An exception object is passed on unchanged. Either way the run ends, pushed or pulled, an iterator returns nothing more, and every part's end is called, told died, before the error is thrown. A mistake in making a pipeline or a ready-made part, such as a stage that's neither a code reference nor a pipeline, or take(-1), is also reported at the caller's line.

What comes with it

Stage modules of your own

Anyone can make stages, in any namespace. Where a module of stages belongs depends on what its stages need to know:

Wherever it goes, a module of stages is built from the same parts as the ready-made stages:

Here's a small one:

package My::Stages::Words;

use v5.36;
use Exporter qw(import);
use Ekorn::Pipeline::Errors qw(croakf);

our @EXPORT_OK = qw(longer_than);

sub longer_than ($length) {
    croakf 'Expected a length of 0 or more, not %s', $length // 'undef'
        unless defined $length && $length =~ m( \A \d+ \z )xa;
    return sub ($word) { length $word > $length ? $word : () };
}

When to use something else

Where it came from

The pipeline began in DBIx::Squirrel, where stages written beside a query shape the rows it returns. None of it needs a database, so it's here on its own, for any function to use, and to grow beyond what rows need. DBIx::Squirrel now uses it, in place of its own copy, from v2.0.5.

Documentation

The user guide tours the package from the bottom up, from a single stage to functions and packages built on it, with recipes for what each part can do, and where it falls short. The reference describes every module, function and method in full, and how a pipeline behaves, with notes for stage authors. Each module's POD, starting with Ekorn::Pipeline, says the same more briefly, and is the authority where they differ.

Installing

From CPAN:

cpanm Ekorn::Pipeline

Or the latest from GitHub:

cpanm https://github.com/nukopian/Ekorn-Pipeline.git

It needs Perl 5.36 or later.

Licence

Copyright (c) 2024-2026 by Iain Campbell. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. See the file LICENSE.

Acknowledgements

Designed and created by Iain Campbell, in collaboration with Claude, Anthropic's AI model.