Actions Status MetaCPAN Release Kwalitee

Ekorn-Pipeline

Stages for any function: let the caller finish the job.

A function that hands out values can't know everything its callers will want done with them. With Ekorn::Pipeline, it takes a list of stages from its caller: small functions that each see one value at a time. It runs what it would return through them. The caller says what they want, beside the call, and the function does only what it alone can.

use Ekorn::Pipeline;
use Ekorn::Pipeline::Stages::Flow;    # take and drop

# The author's function: it reads log lines, and nothing else.
sub log_lines ($file, @stages) {
    open my $fh, '<', $file or die "Can't read $file: $!\n";
    return Ekorn::Pipeline->new(@stages)
        ->iterator(sub { scalar(<$fh>) // () });
}

# A caller's use of it: the first ten errors, as their messages.
my $errors = log_lines(
    'app.log',
    sub { m( \A ERROR \  (.*) )x ? $1 : () },    # only errors, as messages
    take(10),                                    # ten, then stop reading
);
while (my($message) = $errors->()) { say $message }

The problem it solves

Every function that returns a list meets the same pressure. One caller wants only some of the values. Another wants them in a different shape, a third wants the first ten, and a fourth wants them in batches of 500. The author has two ways to answer, and both cost something.

Stages are a third way. The function takes a list of them as its last arguments, and runs each value it makes through them. The caller's processing happens inside the function, one value at a time, as the values are made:

# Options
sub users (%opt) {
    my @users = load_users();
    @users = grep { $_->{active} } @users if $opt{active_only};
    @users = map  { $_->{name} } @users   if $opt{names_only};
    @users = splice @users, 0, $opt{limit} if $opt{limit};
    return @users;
}

# Stages
sub users (@stages) {
    return Ekorn::Pipeline->new(@stages)->(load_users());
}

my @names = users(sub { $_->{active} ? $_ : () }, sub { $_->{name} }, take(10));

The second users does everything the first does, and everything it doesn't, without a line more. Its author is done with it. Whatever a caller wants next, they can write as a stage, without asking.

What everyone gets

Stages

A stage is either a code reference or another pipeline.

A code reference is called with one value, as its argument and in $_, and returns a list:

One shape covers all three. A pipeline is just a list of stages, and a stage needs no wrapper to say which kind it is:

sub { $_ % 2 ? $_ : () }                # filter: the odd numbers
sub { $_ * 10 }                         # map: ten times each
sub ($user) { @{ $user->{roles} } }     # flatMap: each of a user's roles

Returning a list, rather than a value or undef, is what makes this work. An empty list plainly means "nothing", so undef, 0 and the empty string are ordinary values, and can be passed on.

A pipeline used as a stage stands for its own stages, spliced in where it's given. So stages you've put together once can be reused, in another pipeline or with any function that takes stages:

my $active_names = Ekorn::Pipeline->new(sub { $_->{active} ? $_ : () }, sub { $_->{name} });
my @names = users($active_names, take(10));

If a function accepts only code references, give it the pipeline's stages instead: users($active_names->stages, take(10)).

One value at a time

Each value goes through every stage before the next one starts, as it would in a for loop. The values come out in order, and each stage sees them in order. Nothing is held between stages, so a pipeline over a million values needs no more memory than one over ten.

A pipeline runs two ways:

Stopping early

A stage can end the pipeline with Ekorn::Pipeline::Stop. The values already out are kept, and any values the stop carries go on through the rest of the stages. Nothing else is made: once a stage has stopped it, a pulled pipeline never calls its source again.

use Ekorn::Pipeline::Stop;

sub ($user) { $user->{id} == $wanted ? Stop($user) : () }    # find one, then stop

This is how a caller's stages save the author's work. With take(10), the function in the first example reads up to the tenth error line, and leaves the rest of the file unread. Nobody wrote an option for it, and the caller wrote no last.

It matters more where each value costs more. A function over a paged web API fetches a page only when its caller wants a result from it:

sub search ($query, @stages) {
    my $page = 0;
    return Ekorn::Pipeline->new(@stages)->iterator(sub {
        return @{ fetch_page($query, ++$page) };    # a request; empty after the last page
    });
}

my $hits = search('squirrels', take(10));
while (my($hit) = $hits->()) { say $hit->{title} }    # one request, if a page holds ten

Without stages, that function either fetches every page, or grows max_results, max_pages and on_result options, for its author to write and maintain.

Stages with state, and with an end

A stage is a closure, so it can keep state, such as a count. But a plain closure keeps its state from one run to the next, and can't tell when the last value has gone by. Ekorn::Pipeline::Gather makes a stage afresh at the start of each run. The stage can also have an end function, called once the values run out, whose values go on through the rest of the stages:

use Ekorn::Pipeline::Gather;

sub batches_of ($size) {
    return gather {
        my @batch;    # this run's
        return (
            sub ($x) { push @batch, $x; @batch == $size ? [splice @batch] : () },
            sub ()   { @batch ? [@batch] : () },    # the last, short batch
        );
    };
}

That's how batch, collect, take and drop are made.

Stages are values

Stages are code references or pipelines, so they can be kept in variables, built by functions and passed around. A function that returns a stage is a word in a vocabulary, and the vocabulary grows in the caller's code, not the author's:

sub only_active ()    { sub { $_->{active} ? $_ : () } }
sub value_of ($name)  { sub { $_->{$name} } }

my @page  = (drop(20), take(10));
my @names = users(only_active, value_of('name'), @page);

A stage written for users works for log_lines too, or for a function on CPAN whose author has never heard of it.

Writing a function that takes stages

A class of your own can have pipe and tap methods, from Ekorn::Pipeline::Pipes, which run an object through stages. pipe returns what comes out. tap returns the object, so a chain of method calls can carry on after it.

Errors

A stage that dies is named by its position in the pipeline, and reported at the line of the caller's code that ran it, with its message below:

Pipeline stage #2 failed at app.pl line 12:
    No name for user 7 at app.pl line 9.

An exception object is passed on unchanged. Either way the run ends, pushed or pulled, and an iterator returns nothing more. A mistake in making a pipeline or a canned stage, such as a stage that's neither a code reference nor a pipeline, or take(-1), is also reported at the caller's line.

Canned stages

Stage modules of your own

Stages are plain code references, or pipelines of them, so anyone can make them, in any namespace. Where a module of stages belongs depends on what its stages need to know:

Wherever it goes, a module of stages is built from the same parts as the canned stages:

Here's a small one:

package My::Stages::Words;

use v5.36;
use Exporter qw(import);
use Ekorn::Pipeline::Errors qw(croakf);

our @EXPORT_OK = qw(longer_than);

sub longer_than ($length) {
    croakf 'Expected a length of 0 or more, not %s', $length // 'undef'
        unless defined $length && $length =~ m( \A \d+ \z )xa;
    return sub ($word) { length $word > $length ? $word : () };
}

When to use something else

Where it came from

The pipeline began in DBIx::Squirrel, where stages written beside a query shape the rows it returns. None of it needs a database, so it's here on its own, for any function to use, and to grow beyond what rows need. DBIx::Squirrel now uses it, in place of its own copy, from v2.0.5.

Documentation

The how-to shows how to do common tasks, one recipe at a time, for callers, function authors and stage authors. The reference describes every module, function and method in full, and how a pipeline behaves, with notes for stage authors. Each module's POD, starting with Ekorn::Pipeline, says the same more briefly, and is the authority where they differ.

Installing

From CPAN:

cpanm Ekorn::Pipeline

Or the latest from GitHub:

cpanm https://github.com/nukopian/Ekorn-Pipeline.git

It needs Perl 5.36 or later.

Licence

Copyright (c) 2024-2026 by Iain Campbell. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. See the file LICENSE.

Acknowledgements

Written by Iain Campbell, with Claude, Anthropic's AI model. Claude reviews the code, and writes parts of it where Iain asks, often from his code in DBIx::Squirrel. It also helps with the Git Ops, the repository's setup, and the tests and documentation.