NAME
Ekorn::Pipeline - run values through a list of stages
SYNOPSIS
use Ekorn::Pipeline;
my $p = Ekorn::Pipeline->new(
sub { $_->{active} ? $_ : () }, # filter
sub ($user) { @{ $user->{roles} } }, # flatMap
sub { uc }, # map
);
# Push: values in, a list out
my @roles = $p->(@users);
# Pull: one value per call, from a source
my $next = $p->iterator(sub { shift @queue // () });
while (my($role) = $next->()) { ... }
# A function of your own, taking stages from its caller
sub lines_of ($file, @stages) {
open my $fh, '<', $file or die "Can't read $file: $!\n";
return Ekorn::Pipeline->new(@stages)->iterator(sub { scalar(<$fh>) // () });
}
# Named pipelines, and pipeline() to make one from stages and names
use Ekorn::Pipeline::Factories qw(pipeline);
Ekorn::Pipeline->named(clean => sub { s( \A \s+ | \s+ \z )()xgr });
my @first = pipeline(
'clean',
sub { length ? $_ : () },
)->(@lines);
DESCRIPTION
A pipeline is a list of stages. A stage is a code reference; another pipeline, which stands for its own stages (see "A pipeline as a stage"); a stage made for each run, with Ekorn::Pipeline::Gather; or a sink, which consumes the values and gives a result (see Ekorn::Pipeline::Sink).
A code reference is called with one value, both as its argument and in $_. It can name the value with a signature, sub ($row) { $row->[1] }, or use $_, sub { $_->[1] }. $_ is localised for each call, so the caller's is left alone. It returns a list:
nothing, to drop the value (a filter);
one value, to pass it on (a map);
several, each of which goes through the rest of the stages on its own (a flatMap).
Each value goes through every stage before the next one starts. The values come out in order, and each stage sees them in the order a for loop would.
A stage can end the pipeline early with Ekorn::Pipeline::Stop. The values already out are kept, and any values the stop carries go on through the rest of the stages. Nothing else is processed:
sub ($row) { $row->{id} == $wanted ? Stop($row) : () }
Each call of a pipeline with values is a run, and so is each iterator, and each feed, which is given its values as they arrive (see "feed"). A stage made with Ekorn::Pipeline::Gather is made afresh for each run, and so are a source and a sink, so a run can open a file, or start a transaction, and close or commit it when it ends, however it ends: see "RUNS AND HOW THEY END".
Stages from your caller
A pipeline is for the author of any function that hands out values. The function takes stages from its caller, as its last arguments, and runs what it would return through them. The caller says what they want done with each value, beside the call, and gets back only what they want:
sub bands (@stages) {
return Ekorn::Pipeline->new(@stages)->(load_bands());
}
my @names = bands(
sub { $_->{name} },
take(5),
);
With "iterator", the values are made one at a time, as the caller asks for them. A function reading a file or a socket reads no more than it has to. Once a stage has stopped the pipeline, the source isn't called again.
Values can come from a source, given to "iterator": a code reference, or one made with Ekorn::Pipeline::Source, which is told when the run ends. And a pipeline can end in a sink, such as keep_all or count_items, from Ekorn::Pipeline::Sinks, whose result is what the run gives.
The canned stages, in Ekorn::Pipeline::Stages::Flow, Ekorn::Pipeline::Stages::Batch and Ekorn::Pipeline::Stages::Taps, work in any pipeline. Ekorn::Pipeline::Pipes gives a class of your own pipe and tap methods, which run an object through stages.
METHODS
new
my $p = Ekorn::Pipeline->new(@stages);
Returns a pipeline that runs values through @stages, in order. Each stage must be something "is_stage" is true for: a code reference; a pipeline (see "A pipeline as a stage"); or a part offering one of the role methods EKORN_STAGE, EKORN_SINK_ONE and EKORN_SINK_MANY, such as a gather or a sink. Anything else croaks, a source included. With no stages, a pipeline returns its values unchanged.
A pipeline is an object that can be called as a code reference. Called with values, it returns the list that comes out of the last stage:
my @results = $p->(@values);
In scalar context, it returns how many results came out, as an array would:
my $count = $p->(@values);
Unless its last stage is a sink: then a run gives the sink's result. In list context, that's the sink's list, or a sink of one's one value. In scalar context, a sink of one gives its value, and a sink of many its list as an array reference:
my $n = Ekorn::Pipeline->new(count_items)->('a', 'b'); # 2
my $kept = Ekorn::Pipeline->new(keep_all)->('a', 'b'); # ['a', 'b']
A pipeline as a stage
You can give new a pipeline wherever it takes a stage. Its stages are spliced in where it stands, so these two run the same:
my $clean = Ekorn::Pipeline->new(
sub { s( \A \s+ | \s+ \z )()xgr },
sub { length ? $_ : () },
);
Ekorn::Pipeline->new(
$clean,
take(10),
);
Ekorn::Pipeline->new(
$clean->stages,
take(10),
);
The spliced stages belong to the new pipeline's run. A take inside $clean limits the whole run, not each value.
The only difference is in error messages. Given $clean itself, a stage inside it is named by its position in each pipeline: the third stage of a pipeline given second is #2.3, and three levels down, #1.2.3. Given $clean->stages, it's named by its position in the combined list.
The stages spliced in are whatever the pipeline's "stages" method returns, so a subclass can override it. If the override returns a different list, its stages are named by their position in that list.
To run a pipeline on each value separately instead, call it from a stage: sub { $clean->($_) }. Each call is then a run of its own, and a take inside limits each call.
named
Ekorn::Pipeline->named(clean => @stages); # define from stages
Ekorn::Pipeline->new(@stages)->named('clean'); # name one you have
Names a pipeline, and returns it. On the class, it makes a pipeline of the stages after the name, each a code reference, a pipeline or the name of one, as "pipeline_in" takes them, looked up in its caller's package; a subclass makes one of its own class. On a pipeline, it names that one, and croaks if given stages after the name.
The name is kept in the caller's package, unless it's qualified, as "Named pipelines" explains. A name is identifiers, as Perl's, joined by ::, such as clean or My::Module::clean. It's new in v0.3.0.
A name can't be defined twice for another pipeline, which croaks, whatever class named was called on. Naming the same pipeline again, with $p->named('clean'), is no mistake; but named on a class always makes a new pipeline, even from a single pipeline, so calling Ekorn::Pipeline->named(clean => $p) twice croaks. A name can't be removed, so define each once, as a program or module loads.
Named pipelines
A named pipeline is kept for the life of the program, in a package, which says whose name it is, so two modules can each have a clean.
namedkeeps a name in the package that called it, and the wordpipelineandnamedlook one up in the package that called them, so two modules can each have acleanof their own.A qualified name, such as
My::Module::clean, is that package's, wherever it's used. Code that names or looks up pipelines on another package's behalf qualifies them:"${package}::clean".A name is resolved when the pipeline is made, not when it runs, and the pipeline it names is spliced in, as a pipeline given to "new" is. So a pipeline, once made, keeps the pipelines it found, whatever's defined or overridden later. A name alone,
pipeline('clean'), gives that pipeline itself, andnamedon a class always makes a new one, even of one name, so a name never aliases another.Only "pipeline_in", which the word
pipelinecalls, andnamedread names. "new" doesn't, and "is_stage" is false for a string.
name
my $name = $p->name;
Returns the first name the pipeline was given with "named", qualified by its package, as main::clean, or undef if it has none. Errors in its stages name it (see "DIAGNOSTICS"). It's new in v0.3.0.
pipeline_in
Ekorn::Pipeline->pipeline_in($package, @stages)
Returns a new pipeline of its invocant's class, of @stages, each a code reference, a pipeline, or the name of one, looked up in $package unless it's qualified, as "Named pipelines" explains. It's the only method that reads names, and it's what the word pipeline, from Ekorn::Pipeline::Factories, calls, with its caller's package. Given one name and nothing else, it returns the pipeline named itself, if it's of the invocant's class or a subclass, so its stages are numbered where it's given, and in a new pipeline of the invocant's class otherwise:
use Ekorn::Pipeline::Factories qw(pipeline);
# A new pipeline, from stages:
pipeline(
sub { uc },
take(5),
);
# The pipeline named clean, itself:
pipeline('clean');
# A new pipeline: the one named clean, then more stages:
pipeline(
'clean',
take(5),
);
# Another package's, qualified:
pipeline(
'My::Module::clean',
sub {...},
);
Call it from code the user calls directly, passing the user's package, as the word does with scalar caller. Code that passes stages through, such as a function taking stages from its caller, gives them to "new", which reads no names: the user has already turned any names into pipelines with pipeline, where they wrote them. If new read names, it would look them up in the package of whatever called it, which, for a function passing its caller's stages on, isn't the caller's.
A function whose values can be code references can't tell them from stages with "is_stage", but it can take a pipeline given last as its stages, which a caller makes with pipeline:
sub call_all (@callbacks) {
my $stages = @callbacks && $callbacks[-1] isa 'Ekorn::Pipeline' ? pop @callbacks : pipeline();
return $stages->(map { $_->() } @callbacks);
}
call_all(@callbacks, pipeline(
sub { $_ * 10 },
take(2),
));
It's new in v0.3.0, as is the word pipeline.
iterator
my $next = $p->iterator($source);
Returns an iterator, an Ekorn::Pipeline::Iterator, which hands out the values that come out of the stages, one per call. $source is a code reference that returns the next value or values each time it's called, and an empty list when there are no more; or anything offering an EKORN_SOURCE method, such as an Ekorn::Pipeline::Source, made for the run and told when it ends, or another iterator. It's called only once the values from its last call have all been handed out.
The iterator returns an empty list when the values run out, and on every call after that. After a stop, it returns an empty list once the values the stop carried are handed out, and the source isn't called again. Read it in list context, so that an undef value isn't mistaken for the end:
while (my($x) = $next->()) { ... }
The flushes of gathers and sinks are called once their input runs out, before the iterator returns an empty list. The run's ends are called as it ends, as "RUNS AND HOW THEY END" describes. The iterator has methods of its own, to pull every value still to come, or the first, and to end the run early: see Ekorn::Pipeline::Iterator.
If a stage or a flush dies, the call dies with it, and the run has ended, as a pushed run does. Every call after returns an empty list. Nothing waiting goes on through the stages, no flush is called, and the source isn't called again; each end is called, told died, before the error is thrown again. To skip a bad value and carry on, catch the error inside the stage.
feed
my $feed = $p->feed;
my @results = $feed->push(@values); # as often as values arrive
my @rest = $feed->close; # the flushes', and a sink's
Returns a feed, an Ekorn::Pipeline::Feed: one run of the stages, which you give values as they arrive, with push, getting back what came out of them, and end with close, getting what the flushes give. The run is made when feed is called, so a gather's block is called then, and an error in it is thrown then. Fed the same values, a feed gives what a pushed run gives, a push at a time.
Use it where the values don't come from you, a value at a time: from a library's callbacks, an event loop, or a stage running its values through another pipeline. Its ended says when a stage has stopped the run, so the caller can stop feeding; a push after that croaks. Ekorn::Pipeline::Feed has the rest.
stages
my @stages = $p->stages;
Returns the pipeline's stages, in order. A pipeline that was given as a stage appears as the stages it was made of. In scalar context, it returns how many there are.
The list can go anywhere stages go: to "new", or to a function that takes stages. Each is one of Ekorn's types, an object called as a code reference: a code reference given to new comes back as an Ekorn::Pipeline::Stage of it. So a function that takes only plain code references, testing them with is_coderef or ref, won't take them; v0.2.0 gave the code references themselves.
my $clean = Ekorn::Pipeline->new(
sub { s( \A \s+ | \s+ \z )()xgr },
sub { length ? $_ : () },
);
my @titles = Ekorn::Pipeline->new(
$clean->stages,
take(10),
)->(@raw);
To give new the pipeline itself, see "A pipeline as a stage". Either way, the stages run as if they had been written out in full:
A stop in one of them ends the whole run.
A stage made with Ekorn::Pipeline::Gather starts afresh on each run, so pipelines that share it don't share its state.
A plain closure is the same code reference wherever it goes, so anything it keeps is shared. Keep state in
gatherinstead.A stage that dies is named by its position, as "A pipeline as a stage" describes.
RUNS AND HOW THEY END
A run is one call of a pipeline with values, or one iterator or feed made from it. Each run has its own state. A part of the pipeline that needs state, such as a batch it's filling or a file it has open, is made afresh for each run.
Made for a run
A gather (Ekorn::Pipeline::Gather), a source (Ekorn::Pipeline::Source) or a sink (Ekorn::Pipeline::Sink) has a set-up, a code reference called at the start of each run. It declares whatever the run needs, with my, and returns closures sharing what it declared:
the step, called with each value (a gather or a sink), or each time the source is pulled (a source);
the flush, for a gather or a sink, called once its input runs out, what it returns going on through the stages after it. A sink's gives its result, and is required;
the end, called once, when the run has ended, told how. It's optional;
a context, for a source only, and optional: anything its stages may ask for with "run_context".
Two iterators of the same pipeline each get their own, from their own call of the set-up. A plain code reference is never made for a run: it's the same code, with the same closure, in every run.
The set-up returns its parts as a list, in the order above, leaving out those it doesn't give from the end, and giving undef for one it leaves out before one it gives. Or it returns them by name, as a hash reference whose keys are step, flush, end and context, giving only those it has, in any order:
gather {
my $n = 0; # this run's
return {
step => sub ($x) { $n++; $x },
end => sub ($how) { $$total = $n },
};
}
A key Ekorn doesn't read is ignored, and a hash without what its part needs, a step, and a sink's flush, croaks, as a list without them does. Only a plain hash reference returned alone is taken for parts by name. Parts by name are new in v0.3.0.
Any object can take part, by offering a role method that does what a set-up does, returning the same parts, as a list or by name: EKORN_SOURCE, EKORN_STAGE, EKORN_SINK_ONE or EKORN_SINK_MANY. "Delivering parts for Ekorn" in Ekorn::Pipeline::Manual::Reference has how.
How a run ends
A run ends in one of three ways, and each part's end is told which, from where the part stands:
died, if the run died, anywhere, before it ended;stopped, if its input was cut short, by a stop at or after it, or by the run being ended from outside, as an iterator's finish does;done, if its input ran out, and the run didn't die. A part whose input had run out before a stop is tolddone, and so is a sink that stops the run itself.
So a stop ends the stream early for what's before it, and normally for what's after it. In
my $n = Ekorn::Pipeline->new(
take(5),
count_items,
)->iterator($lines)->();
$lines is told stopped, and closes its file, and count_items, after take(5), is told done, and gives 5.
Flushes and ends
A flush is called once its part's input runs out normally: when the values ran out, after a stop in a stage before it, or, for a sink, when the sink stopped the run itself. What it returns goes on through the stages after it, and while it runs, and they do, "ending" is true. It isn't told how the run ended, as the run hasn't.
An end:
always runs, once, when the run has ended, however it ended, and is given how, as its argument:
done,stoppedordied. A run has ended as soon as nothing in it can still act on a value. With a sink last, that's as the sink's result is ready, so a sink that commits only when tolddonecommits then, and never for a run that dies after it;gives nothing: what it returns is dropped;
runs in order: the source's end first, then each stage's, first stage first, so a source that fails to close is known before any sink commits.
After a stage dies, every end still runs, told died, and then the error is thrown again, as it would have been. An end that dies then is reported as a warning, naming its stage, and the first error is kept. An end that dies in a run that hadn't died ends the run with its error, and the ends after it are told died.
An iterator dropped before its values run out ends its run when the last reference to it goes, as Ekorn::Pipeline::Iterator describes.
FUNCTIONS
is_stage
use Ekorn::Pipeline v0.1.3 qw(is_stage);
is_stage($thing)
True if $thing can be used as a stage: a code reference; a pipeline of this class or a subclass; or an object offering one of the role methods EKORN_STAGE, EKORN_SINK_ONE and EKORN_SINK_MANY, such as a stage made with Ekorn::Pipeline::Gather, or a sink. It's the same test "new" uses. It's false for a string, even the name of a pipeline, and for a source.
Use it in a function that takes its stages from the end of its arguments, so that it accepts the same stages new does, pipelines included:
sub fetch ($url, @args) {
my @stages;
unshift @stages, pop @args while @args && is_stage($args[-1]);
...
}
It's exported on request. It's new in v0.1.3, so ask for that version on the use line, as above: v0.1.2 has no import, so without the version, use Ekorn::Pipeline qw(is_stage) compiles under it, and fails only when is_stage is called.
A pipeline is a blessed hash, so is_hashref is true of one too. A function that takes a hash reference of options before its stages should test is_stage first, or it takes a pipeline for the options.
is_sink
use Ekorn::Pipeline qw(is_sink);
is_sink($thing)
Returns one if $thing is a sink of one, giving one value, and many if it's a sink of many, giving a list; for a pipeline, the sink it ends in, if any. Both are true. It's false for anything else. A sink is anything offering EKORN_SINK_ONE or EKORN_SINK_MANY, as Ekorn::Pipeline::Sink describes. It's exported on request.
ending
Ekorn::Pipeline::ending()
True while a flush runs, and while the values it gave go through the stages after it, rather than values from the source. It's for code that tells its stages more about each value, such as where it came from: a value from a flush has no such origin. It isn't exported.
run_context
Ekorn::Pipeline::run_context()
Returns the context the run's source gave, if any: its context, or the third part its EKORN_SOURCE method returned as a list, such as a statement's column names. It's for stages that need to know about the values' source, called as they run. Each run has its own. A pushed run has none, and an iterator used as a source passes on its own run's. A feed has none of its own: its stages see the context of the code pushing into it. It isn't exported.
sub pluck ($name) {
return sub ($row) {
my $columns = Ekorn::Pipeline::run_context()->{columns};
my($at) = grep { $columns->[$_] eq $name } 0 .. $#$columns;
return $row->[$at];
};
}
DIAGNOSTICS
- Expected stage #%d to be a stage: a code reference, a pipeline, or an object offering a stage's role method
-
A stage given to "new" was something else, such as a string.
- Expected stage #%d, a pipeline, to give only stages from its stages method
-
A stage given to "new" was a pipeline whose overridden
stagesmethod returned something that isn't a stage. - Expected stage #%d to be a stage: a code reference, a pipeline or its name, or an object offering a stage's role method
-
A stage given to "pipeline_in", and so to the word
pipeline, or to "named" after the name, was none of them. Fornamed, stages are counted from the first after the name. - No pipeline named %s
-
A name given to "pipeline_in", and so to the word
pipeline, or to "named", isn't defined in that package. The name is given qualified, so it says which package it was looked up in. It may be defined in another package, which a qualified name reaches. - A pipeline named %s is already defined
-
"named" was given a name already defined for another pipeline.
- Expected a name, made of words joined by ::
-
"named" was given something other than a name.
- Expected only a name, naming a pipeline you have
-
"named", on a pipeline, was given stages after the name.
- Expected stage #%d to be a stage, not a source
-
A source, something offering
EKORN_SOURCEand none of a stage's role methods, was given as a stage. Give it to "iterator" instead. - Expected stage #%d to offer one of EKORN_STAGE, EKORN_SINK_ONE and EKORN_SINK_MANY, not %s
-
A stage offered more than one of the role methods, so it couldn't say which it is.
- Expected a source: a code reference, or an object offering EKORN_SOURCE
-
The source given to "iterator" was something else, such as a stage, or a string.
- Expected the source to return a code reference, and optionally an end and a context
-
A source's
EKORN_SOURCEreturned something else. It's reported as the run starts. - Expected stage #%s to return a code reference, and optionally a flush and an end
-
A stage's
EKORN_STAGEreturned something else. It's reported as the run starts. - Expected sink #%s to return a code reference, a flush, and optionally an end
-
A sink's
EKORN_SINK_ONEorEKORN_SINK_MANYreturned something else. It's reported as the run starts. - Expected no stop in a set-up, as the run has nothing yet to stop
-
A set-up, or a role method, threw a Ekorn::Pipeline::Stop. The ends of the parts already made are called, told
died. - Pipeline end of %s failed at %s line %d:
-
An end died, in a run that hadn't died. The message it died with follows, indented, and the end is named:
the source, or a stage by its position, asstage #2. An exception object is passed on unchanged. - Expected no stop in an end, as the run has ended
-
An end threw a Ekorn::Pipeline::Stop, which is an error like any other there, reported as the message above.
- The end of %s died while the run was already dying: %s
-
A warning: an end died in a run that had died. The run's own error is thrown, once every end has run.
- Expected stage #%s, made with gather, to return a code reference or a pipeline, and optionally a flush and an end
-
The block given to "gather" in Ekorn::Pipeline::Gather for that stage returned something else.
- Pipeline stage #%s failed at %s line %d:
-
A stage, or its flush, died. The message it died with follows, indented. An exception object is passed on unchanged.
In this message and the one above, a stage is named by its position:
#2for the second stage, or#2.3for the third stage of a pipeline given as the second stage.A stage within a named pipeline, or of the named pipeline run, is named with it too: after its position, the innermost named pipeline it's in, by its "name", and its position there, as
#1.2 (main::clean#2)for the second stage of the pipeline namedclean, given as the first stage. It's the name the pipeline has when the stage dies.
SEE ALSO
Ekorn::Pipeline::Tooling, to import everything from one place; Ekorn::Pipeline::Factories, the words; Ekorn::Pipeline::Sugar, pipelines named as the program compiles; Ekorn::Pipeline::Gather, Ekorn::Pipeline::Stop, Ekorn::Pipeline::Source, Ekorn::Pipeline::Sink, Ekorn::Pipeline::Iterator and Ekorn::Pipeline::Feed, the types; Ekorn::Pipeline::Stages::Flow, Ekorn::Pipeline::Stages::Batch, Ekorn::Pipeline::Stages::Taps, Ekorn::Pipeline::Sources, Ekorn::Pipeline::Sinks and Ekorn::Pipeline::JSON, the ready-made parts; Ekorn::Pipeline::Operators, for |; Ekorn::Pipeline::Pipes; and DBIx::Squirrel, where the pipeline began
Ekorn::Pipeline::Manual::HowTo, the user guide, tours the package from the bottom up, with recipes, and Ekorn::Pipeline::Manual::Reference describes every module, function and method in full.
AUTHOR
Iain Campbell <cpanic@cpan.org>
LICENSE
Copyright (c) 2024-2026 by Iain Campbell.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.