NAME
Ekorn::Pipeline::Manual::HowTo - Ekorn::Pipeline user guide
DESCRIPTION
Ekorn::Pipeline lets a function hand its values, one at a time, through a list of small steps chosen by whoever called it. The steps are called stages. A stage can change a value, drop it, turn it into several, or say "that's enough", and the function stops making values.
That one idea goes a long way. A function that reads a log file no longer needs options for "only errors", "only the first ten" or "just the message part": its caller writes those as stages, beside the call, and the function stays the size of what only it can do. The same stages work with every function that takes them, whoever wrote it.
This guide shows what you can do with Ekorn::Pipeline, and why you might want to. It starts with the smallest piece, a single stage, and works upwards, a layer at a time, to functions, modules and packages built on it. Each layer uses only what the layers before it have shown, so you can read it from the start, or dip into the layer you need. Every example runs as it stands, and its comments say what it prints.
The reference describes every function and method in full. This guide is the tour; the reference is the map.
Contents
The layers, at a glance
Every run of a pipeline has the same shape. Values come from somewhere, go through stages one at a time, and end up somewhere:
source --> stage --> stage --> stage --> sink
(where (change, drop, multiply, (where they
they come or stop: one value at a time) end up)
from)
Ekorn::Pipeline is built in layers around that shape, and this guide follows them from the bottom up:
- Stages
-
What it gives you: an ordinary sub that sees one value and returns what should go on
Chapter: 1
- Pipelines
-
What it gives you: stages in order, which you call with values, or pull results from one at a time
Chapter: 2
- Stopping
-
What it gives you: a way for any stage to say "enough", ending the work before it
Chapter: 3
- Parts made for each run
-
What it gives you: stages with their own state, told when the values run out and when the run ends
Chapter: 4
- Sources
-
What it gives you: where values come from, such as a file, closed however the run ends
Chapter: 5
- Sinks
-
What it gives you: where values end up: a count, the first, every one, or a file
Chapter: 6
- Composition
-
What it gives you: pipelines inside pipelines, and pipelines with names
Chapter: 7
- Operators
-
What it gives you:
source | stage | sink, for when a pipeline reads best as one lineChapter: 8
- Functions that take stages
-
What it gives you: the reason it all exists: functions whose callers finish the job
Chapter: 9
- Parts for others
-
What it gives you: modules of stages, and your own classes as sources and sinks
Chapter: 10
Chapter 11 covers errors and debugging, and chapter 12 answers the questions people ask first: how fast it is, how it compares with map and grep, and what it doesn't do.
The examples' data
Every example in this guide starts from the code below: six tracks, a function tracks that hands them out through whatever stages its caller gives, and a file of their titles, one per line, in a temporary folder that's removed when the program ends.
To try an example, put this at the top of a file, and the example after it. It imports every stage, source, sink and word the guide uses, from Ekorn::Pipeline::Tooling, so the examples needn't (see Import everything from one place). Each example stands alone, given this, apart from a few that use a function an earlier one defined, which they say. To try several in one file, put each in braces, { ... }, so their variables don't clash.
use v5.36;
use Ekorn::Pipeline;
use Ekorn::Pipeline::Tooling qw(:all); # every stage, source, sink and word
use File::Temp qw(tempdir);
my @tracks = (
{artist => 'Abba', title => 'Dancing Queen', plays => 9100},
{artist => 'Abba', title => 'Knowing Me, Knowing You', plays => 4200},
{artist => 'Blur', title => 'Girls & Boys', plays => 6100},
{artist => 'Blur', title => 'Parklife', plays => 5800},
{artist => 'Blur', title => 'Country House', plays => 3900},
{artist => 'Cream', title => 'Sunshine of Your Love', plays => 7700},
);
# The tracks, through whatever stages the caller gives.
sub tracks (@stages) {
return Ekorn::Pipeline->new(@stages)->(@tracks);
}
# setlist.txt: each title on a line of its own.
my $dir = tempdir(CLEANUP => 1);
my $setlist = "$dir/setlist.txt";
open my $out, '>', $setlist or die "Can't write $setlist: $!\n";
print {$out} map { "$_->{title}\n" } @tracks;
close $out or die "Can't write $setlist: $!\n";
Import everything from one place
What you use to do work with Ekorn::Pipeline is made in several modules, one for each kind of thing, and the chapters below say which. You needn't remember them: Ekorn::Pipeline::Tooling exports all of it, by tags that follow the modules:
:core-
What it gives you:
StopandgatherMade in: Ekorn::Pipeline::Stop and ::Gather
:flow-
What it gives you:
take,drop,take_if,take_whileand the restMade in: Ekorn::Pipeline::Stages::Flow
:batch-
What it gives you:
batchandcollectMade in: Ekorn::Pipeline::Stages::Batch
:taps-
What it gives you:
peek,count_intoanddump_toMade in: Ekorn::Pipeline::Stages::Taps
:stages-
What it gives you:
:flow,:batchand:taps :sources-
What it gives you:
from_array,from_file,from_fhandcounterMade in: Ekorn::Pipeline::Sources
:sinks-
What it gives you:
count_items,keep_all,keep_first,to_fileand the restMade in: Ekorn::Pipeline::Sinks
:json-
What it gives you:
from_json,from_jsonl,to_jsonandto_jsonlMade in: Ekorn::Pipeline::JSON
:words-
What it gives you:
pipeline,stage,source,generator,sink_oneand the restMade in: Ekorn::Pipeline::Factories
:all-
What it gives you: all of the above
Ask for what you need, by tag or by name. Nothing comes by default:
package My::Report {
use Ekorn::Pipeline::Tooling qw(:stages :sinks);
say scalar Ekorn::Pipeline->new(
take_if { $_->{plays} > 6000 },
count_items,
)->(@tracks);
# 3
}
Each name is the very sub its own module exports, so it makes no difference which you import it from. Two things Tooling leaves out: what's for extending Ekorn rather than using it, such as is_stage and croakf, which you import from their own modules when chapters 9 and 10 need them; and |, which only use Ekorn::Pipeline::Operators puts in force, in its own scope, as chapter 8 explains.
1. Stages: one value at a time
A stage is an ordinary Perl sub. It's given one value, and it returns a list of what should go on in its place. That's the whole contract, and everything else in Ekorn::Pipeline is built on it. What a stage returns decides what it is:
- one value
-
So it: changes the value, or passes it on as it is
Often called: a map
- nothing, an empty list
-
So it: drops the value: the stages after it never see it
Often called: a filter
- several values
-
So it: turns one value into several, each going on by itself
Often called: a flatMap
The names in the last column come from other languages, and you'll meet them in the reference. You don't need them: a stage is just a sub that answers "what should go on, given this?"
Change each value
Return one value, and it goes on in place of the one the stage was given:
my @shouted = tracks(sub { uc $_->{title} });
say $shouted[0]; # DANCING QUEEN
Drop the values you don't want
Return an empty list, and the value is dropped:
my @hits = tracks(
sub { $_->{plays} >= 6000 ? $_ : () }, # only the hits
sub { $_->{title} }, # as their titles
);
say join ', ', @hits; # Dancing Queen, Girls & Boys, Sunshine of Your Love
The stages run in the order you wrote them, top to bottom, so the second stage only ever sees the hits.
One thing to watch: a test on its own isn't a filter. sub { $_->{plays} >= 6000 } returns what the test returns, true or false, and that's what goes on, in place of the track. Return the value, or nothing. Or give the test to take_if, from chapter 3, which passes on the value: take_if { $_->{plays} >= 6000 }.
Turn one value into several
Return several values, and each goes on through the rest of the stages, in order:
my @words = tracks(
sub { $_->{artist} eq 'Abba' ? $_ : () },
sub { split ' ', $_->{title} },
sub { lc },
);
say "@words"; # dancing queen knowing me, knowing you
Use $_, or give the value a name
A stage is given its value twice: in $_, and as its only argument. Use $_ for a short stage, and a signature when a name reads better:
my @short = tracks(sub { $_->{title} });
my @named = tracks(sub ($track) { $track->{title} });
say "@short" eq "@named" ? 'the same' : 'different'; # the same
$_ is made local to each call, so a stage can use it freely, and call code that uses it too, without disturbing yours.
Use a sub you already have
Any code reference is a stage, including one to a named sub. So the functions you already have can be stages as they are, if they take one value and return what should go on:
sub trim ($text) { $text =~ s( \A \s+ | \s+ \z )()xgr }
my @clean = Ekorn::Pipeline->new(\&trim)->(' Parklife ', ' Tender');
say join '|', @clean; # Parklife|Tender
Keep undef, 0 and the empty string
Only an empty list drops a value. undef, 0 and '' are values like any other, and go on:
my @kept = Ekorn::Pipeline->new(sub { $_ || undef })->(3, 0, '', 5);
say scalar @kept; # 4
So return undef passes undef on. To drop a value, write return; or ().
Leave the caller's data alone
The values given to a pipeline are copied, so assigning to $_ in a stage changes the copy, not your variable. But a reference is copied as a reference: a stage that changes a hash it's given changes your hash. To leave it alone, return a new one:
my @doubled = tracks(sub { +{ %$_, plays => $_->{plays} * 2 } });
say "$doubled[0]{plays} $tracks[0]{plays}"; # 18200 9100
Do something on the side, and pass the value on
A stage run for what it does, such as logging, must still return the value it was given, or the value is dropped:
my @seen;
my @titles = tracks(
sub { push @seen, $_->{artist}; $_ }, # note it, then pass it on
sub { $_->{title} },
);
say scalar @seen, ' seen, ', scalar @titles, ' kept'; # 6 seen, 6 kept
Forget the $_ at the end, and push returns the length of @seen, which goes on in place of the track. peek does this for you, and can't forget.
See the order values move in
Each value goes through every stage before the next value starts. When a stage returns several, the first goes all the way through, then the second:
my @trace;
Ekorn::Pipeline->new(
sub { push @trace, "1:$_"; ("$_.1", "$_.2") },
sub { push @trace, "2:$_"; $_ },
)->(qw(a b));
say "@trace"; # 1:a 2:a.1 2:a.2 1:b 2:b.1 2:b.2
This is what lets a pipeline stop early, and hold almost nothing in memory: nothing waits between stages.
Compared with map and grep
Perl's map and grep do some of this already, and for a list that's in memory, used once, they're the right tool. Here are the hits' titles both ways:
my @by_map = map { $_->{title} } grep { $_->{plays} >= 6000 } @tracks;
my @by_stages = tracks(
sub { $_->{plays} >= 6000 ? $_ : () },
sub { $_->{title} },
);
say "@by_map" eq "@by_stages" ? 'the same' : 'different'; # the same
The differences show once the job grows:
Order.
mapandgrepread from right to left, last step first. Stages read top to bottom, in the order they happen.Memory. Each
mapandgrepbuilds a whole new list before the next starts. Stages pass one value at a time, so ten steps over a million values hold one value, not ten lists of a million.Stopping.
grepcan't stop part way. A stage can, and the work before it stops too: the file isn't read further, the next page isn't fetched (see 3. Stopping early).Passing them on. You can't hand a
grepto someone else's function. You can hand it stages, and it runs its values through them (see 9. Functions that take stages).Speed.
mapandgrepare faster, by a long way, for trivial steps over values already in memory. How fast is it? has numbers.
2. Pipelines: pushing values, pulling values
A pipeline is a list of stages, in order. You can use one in two ways:
Push values in. Call it with values, and it gives back what comes out of the last stage, all at once.
Pull results out. Give it a source, something to ask for values, and it gives you an iterator, which hands out one result each time you call it, asking the source for values only as they're needed.
Each call, and each iterator, is a run. A pipeline can have as many runs as you like.
Make a pipeline, and call it
new makes a pipeline from stages. Call it like a sub:
my $titles = Ekorn::Pipeline->new(sub { $_->{title} });
my @titles = $titles->(@tracks);
say $titles[-1]; # Sunshine of Your Love
The tracks function in the examples' data is just this: Ekorn::Pipeline->new(@stages)->(@tracks).
Count the results
Called in scalar context, a pipeline gives how many results came out:
my $hits = tracks(sub { $_->{plays} >= 6000 ? $_ : () });
say $hits; # 3
Run values through no stages at all
A pipeline with no stages gives its values back unchanged. So a function that takes stages has nothing to special-case when its caller gives none:
say scalar tracks(); # 6
Use a pipeline again and again
A pipeline is a value: keep it in a variable, pass it around, and call it whenever you like. Each call is a run of its own:
my $loud = Ekorn::Pipeline->new(
sub { uc },
sub { "$_!" },
);
say join ' ', $loud->('hey'), $loud->('ho'); # HEY! HO!
A stage can even call the pipeline it's in:
my $countdown;
$countdown = Ekorn::Pipeline->new(sub { $_ > 0 ? ($_, $countdown->($_ - 1)) : 'lift-off' });
say join ' ', $countdown->(3); # 3 2 1 lift-off
Pull results one at a time
iterator takes a source, and returns an iterator. The simplest source is a sub that returns the next value each time it's called, and an empty list when there are no more. The iterator is called like a sub too, and returns one result per call:
my @queue = (3, 1, 2);
my $next = Ekorn::Pipeline->new(sub { $_ * 10 })
->iterator(sub { @queue ? shift @queue : () });
while (my($n) = $next->()) {
say $n;
}
# 30
# 10
# 20
The source is called only when a result is wanted. That's the point of pulling: a source reading a file, a socket or a database reads no more than you ask for.
Read results in list context
Read an iterator in list context, my($n) = $next->(), as above. Then a result of 0, '' or undef isn't taken for the end. Written as while (my $n = $next->()), the loop would stop at the first 0:
my @counts = (2, 0, 1);
my $next = Ekorn::Pipeline->new->iterator(sub { @counts ? shift @counts : () });
my @seen;
while (my($n) = $next->()) { push @seen, $n }
say "@seen"; # 2 0 1
Write a source that can give undef
A source ends by returning an empty list, so a source written as sub { shift @queue // () } ends at the first undef in the queue. Ask whether there's anything left instead:
my @queue = ('a', undef, 'b');
my $next = Ekorn::Pipeline->new(sub { $_ // 'none' })
->iterator(sub { @queue ? shift @queue : () });
say join ' ', $next->all; # a none b
all, one of the iterator's methods, reads every result that's left.
Read a filehandle a line at a time
A filehandle makes a source. Read it in scalar context, so each call reads one line:
my $log = <<~'END';
INFO started
ERROR disk full
INFO retrying
ERROR disk still full
END
open my $fh, '<', \$log or die "Can't read the log: $!\n";
my $errors = Ekorn::Pipeline->new(sub { m( \A ERROR \s+ (.*) )x ? $1 : () })
->iterator(sub { scalar(<$fh>) // () });
while (my($message) = $errors->()) {
say $message;
}
# disk full
# disk still full
Without scalar, <$fh> would read every line on its first call. Chapter 5 has sources that read files for you, and close them, however the run ends.
Fetch values a page at a time
A source can return several values from one call, such as a page of results from a web service. Every value from one call goes through the stages before the source is called again:
my @pages = ([qw(a b c)], [qw(d e)]);
my $calls = 0;
my $next = Ekorn::Pipeline->new(sub { uc })
->iterator(sub { $calls++; @pages ? @{ shift @pages } : () });
my($first) = $next->();
say "$first after $calls call"; # A after 1 call
Make values for ever, and take what you need
A source needn't end. Pulled, it makes only as many values as are asked for, so a source of every square number is fine, as long as something stops the run. Here take, from chapter 3, does:
my $n = 0;
my @squares = Ekorn::Pipeline->new(
sub { $_ * $_ },
take(5),
)->iterator(sub { ++$n })->all;
say "@squares"; # 1 4 9 16 25
Pushed values are a list, already made. Pulled values are made on demand. That's the difference that makes a source without end useful.
Read everything that's left, or only the first
An iterator has methods for reading in one go:
allreads every result that's left: a list, or an array reference in scalar context;firstreads the next result, and ends the run, so nothing more is read;singlereads the next, asfirstdoes, but warns if there was another, for a run that should give only one.
my @queue = @tracks;
my $next = Ekorn::Pipeline->new(sub { $_->{title} })
->iterator(sub { @queue ? shift @queue : () });
say $next->first; # Dancing Queen
say scalar @queue; # 5
say scalar @{ $next->all }; # 0
first read one track, and ended the run, so five were never read. Chapter 5's sources show why ending a run matters: it's when a file is closed.
Ask how a run ended
finish ends a run early, and ended says how a run ended, once it has: done if the source ran out, stopped if something cut it short, and died if a stage died. Until then, it gives undef:
my @queue = (1 .. 5);
my $next = Ekorn::Pipeline->new->iterator(sub { @queue ? shift @queue : () });
$next->();
say $next->ended // 'still running'; # still running
$next->finish;
say $next->ended; # stopped
An iterator you let go of, by letting its variable go out of scope, ends its run as Perl frees it, as if you'd called finish.
Run two iterators side by side
Each iterator is a run of its own, so two from one pipeline don't get in each other's way:
my $numbered = Ekorn::Pipeline->new(sub { "<$_>" });
my @left = qw(a b);
my @right = qw(x y);
my $one = $numbered->iterator(sub { @left ? shift @left : () });
my $two = $numbered->iterator(sub { @right ? shift @right : () });
say join ' ', $one->(), $two->(), $one->(), $two->(); # <a> <x> <b> <y>
Push or pull?
Push when you already have the values in a list, and want the results as a list. It's the simplest, and the most common.
Pull when the values are made one at a time, from a file, a socket, a database or a calculation, so that a stage that stops saves the work of making the rest; or when there are too many to hold at once; or when the caller wants to handle each result as it comes.
A function can offer either, from the same stages: chapter 9 shows how.
3. Stopping early
Any stage can end its run, by throwing a stop. The values already out are kept, the source isn't asked for more, and the run ends cleanly. This is what makes pulling worthwhile: a caller who wants ten results gets ten, and the work of making the eleventh is never done.
Stop at the first match
Stop, from Ekorn::Pipeline::Stop, ends the run from inside a stage. Values given to it go on through the rest of the stages, as the run's last:
my($blur) = tracks(sub { $_->{artist} eq 'Blur' ? Stop($_) : () });
say $blur->{title}; # Girls & Boys
Stop(), with nothing, ends the run without passing anything on. Ekorn::Pipeline::Stop->throw(...) does the same as Stop(...), without importing anything.
Stop without reading one more
Give the stop the last value you want, as above, rather than stopping at the value after it. Then the source isn't asked for the one more value it would take to notice:
my $read = 0;
my $next = Ekorn::Pipeline->new(sub { $_ eq 'Parklife' ? Stop($_) : $_ })
->iterator(sub { $read < @tracks ? $tracks[$read++]{title} : () });
my @up_to = $next->all;
say "@up_to[-1, 0]"; # Parklife Dancing Queen
say "read $read of 6"; # read 4 of 6
With a source reading a slow service, one value saved is one request saved.
Take the first few
take, from Ekorn::Pipeline::Stages::Flow, passes on the first few values, then stops. With an iterator, the source isn't called again:
my $read = 0;
my @two = Ekorn::Pipeline->new(
take(2),
sub { $_->{title} },
)->iterator(sub { $read < @tracks ? $tracks[$read++] : () })->all;
say join ', ', @two; # Dancing Queen, Knowing Me, Knowing You
say "read $read of 6"; # read 2 of 6
take sends its last value with its stop, as the last recipe advises, so the third track is never read.
Page through values
drop leaves out the first few values, and passes on the rest. Together with take, it gives a page:
my @page2 = tracks(
drop(2),
take(2),
sub { $_->{title} },
);
say join ', ', @page2; # Girls & Boys, Parklife
Stages run in the order written, so put a filter before drop and take to page through what the filter lets by.
Take the first, the last, or the only one
Flow has three more stages for picking one value, imported on request:
take_firstpasses on the first value, then stops;take_lastpasses on the last, once the values run out, reading every one to find it;take_singlepasses on the one value, and if there's a second, warns "Expected only one item, but there were more" and stops: for when you meant one, and want to hear about it if there are more.
say tracks(
sub { $_->{title} },
take_first,
);
# Dancing Queen
say tracks(
sub { $_->{title} },
take_last,
);
# Sunshine of Your Love
say tracks(
sub { $_->{artist} eq 'Cream' ? $_->{title} : () },
take_single,
);
# Sunshine of Your Love
Each is a stage, so more stages can follow it. Chapter 6's sinks, keep_first, keep_last and keep_single, give the same values as the run's result instead.
Keep or drop values by a test
take_if passes on each value its block is true for, and drop_if each value its block is false for, testing every value, as grep does. Compare take_while, in the next recipe, which stops at the first value its test fails:
say join ', ', tracks(
take_if { $_->{plays} >= 6000 },
sub { $_->{title} },
);
# Dancing Queen, Girls & Boys, Sunshine of Your Love
say join ', ', tracks(
drop_if { $_->{artist} eq 'Blur' },
sub { $_->{title} },
);
# Dancing Queen, Knowing Me, Knowing You, Sunshine of Your Love
say join ', ', tracks(
take_while { $_->{plays} >= 6000 },
sub { $_->{title} },
);
# Dancing Queen
The second track has fewer plays, so take_while stopped there, and read no further. Use take_if to keep every match, and take_while when the values are in an order that means nothing after the first miss is wanted, such as a file's header lines.
Take values while a test holds
take_while passes on values while its block is true for them, and stops at the first it's false for, which it doesn't pass on. Here it reads an email's headers, which end at the first blank line, and never reads the body:
my $mail = <<~'END';
From: abba@example.com
Subject: Waterloo
My, my, at Waterloo Napoleon did surrender
END
open my $fh, '<', \$mail or die "Can't read the mail: $!\n";
my @headers = Ekorn::Pipeline->new(
take_while { m( \S )x },
sub { s( \n \z )()xr },
)->iterator(sub { scalar(<$fh>) // () })->all;
say join ' | ', @headers; # From: abba@example.com | Subject: Waterloo
Skip values while a test holds
drop_while is the other half: it drops values while its block is true for them, then passes on the rest, from the first it's false for, without testing again:
my @lines = ('# tracks', '# by plays', 'Dancing Queen', '# not a comment now');
my @body = Ekorn::Pipeline->new(drop_while { m( \A \# )x })->(@lines);
say join ' | ', @body; # Dancing Queen | # not a comment now
Catch errors in a stage, but let a stop through
A stop is an exception, so a stage that catches errors itself, with eval or try, catches stops too, including those of stages it runs. Throw a stop on:
use experimental qw(try);
my @titles = tracks(sub ($track) {
try {
Stop() if $track->{artist} eq 'Blur';
return uc $track->{title};
}
catch ($e) {
die $e if $e isa 'Ekorn::Pipeline::Stop'; # a stop goes on
return 'unknown';
}
});
say join ', ', @titles; # DANCING QUEEN, KNOWING ME, KNOWING YOU
Swallow the stop, and the run carries on when it should have ended.
Compared with last and flags
Without stops, stopping early means a loop with last, or a flag tested on every pass, written around the call that made the values. That works when the loop and the values are yours. A stop works from any stage, in any order, inside a function someone else wrote, and the function's own work stops with it: its file is read no further, its next page isn't fetched. Neither a last nor a flag can reach inside a function to do that.
4. Parts made for each run
So far, every stage has been a plain sub. A plain sub is the same sub on every run, and that's fine until a stage needs to remember something: a count, a batch it's filling, a file it has open. This chapter is about parts that are made afresh for each run, so each run has state of its own; that are told when the values run out, so they can hand over what they've held; and that are told when the run ends, however it ends, so they can tidy up.
See why a plain sub remembers too much
A stage that keeps a count in a closure keeps it from one run to the next:
my $count = 0;
my $plain = Ekorn::Pipeline->new(sub { ++$count . ". $_" });
my @first = $plain->(qw(a b));
my @second = $plain->(qw(a b));
say "@first / @second"; # 1. a 2. b / 3. a 4. b
The second run should have counted from 1 again. Worse, two iterators read in turn would share one count. The stage needs a fresh count for each run.
Keep state for one run with gather
gather, from Ekorn::Pipeline::Gather, makes a stage from a block. The block is the stage's set-up: it's called at the start of every run, and returns the stage. So whatever the block declares with my is fresh for each run:
sub numbered () {
return gather {
my $n = 0; # this run's
return sub ($value) { ++$n . ". $value" };
};
}
my $p = Ekorn::Pipeline->new(numbered());
my @first = $p->(qw(a b));
my @second = $p->(qw(a b));
say "@first / @second"; # 1. a 2. b / 1. a 2. b
Make any stage that keeps state with gather, even a simple count. It costs one call per run, and it means the stage is right however it's used: run twice, given to two pipelines, or read by two iterators at once.
The catch: nothing makes you. A plain closure that keeps state works in a quick test, and goes wrong the second time it runs. It's the mistake to look for first when a stage gives odd results on its second run.
Hand over what you held when the values run out
The set-up can return a second sub, the flush. It's called once the stage's input runs out, and what it returns goes on through the rest of the stages, just as a stage's values do. So a stage can hold values back, and hand them over at the end:
sub titles_by_artist () {
return gather {
my %titles;
my $step = sub ($track) {
push @{ $titles{ $track->{artist} } }, $track->{title};
return; # hold every track; pass nothing on yet
};
my $flush = sub () {
return map { "$_: " . join(', ', @{ $titles{$_} }) } sort keys %titles;
};
return ($step, $flush);
};
}
say for tracks(titles_by_artist());
# Abba: Dancing Queen, Knowing Me, Knowing You
# Blur: Girls & Boys, Parklife, Country House
# Cream: Sunshine of Your Love
The flush is called even when no values went by, so it can give an empty result, or nothing at all. Flushes are called in order, first stage first, and what one gives goes through the stages after it before the next flush is called, so nothing is lost between them.
Hand values on in batches, or all at once
Ekorn::Pipeline::Stages::Batch has two stages built exactly this way. batch passes values on in array references of a given size, the last one shorter if need be; collect holds every value, and passes on one array reference of them all, or what its block makes of them:
use List::Util qw(sum0);
my @batches = tracks(
sub { $_->{title} },
batch(4),
);
say scalar @batches, ' batches, the last: ', "@{ $batches[-1] }";
# 2 batches, the last: Country House Sunshine of Your Love
my($total) = tracks(
sub { $_->{plays} },
collect { sum0 @_ },
);
say $total; # 36800
batch holds one batch at most, so it can feed a database 500 rows at a time from a file of millions, never holding more than 500. collect holds everything until the end, so the whole lot is in memory at once: use it when you need every value together, and batch when you don't.
Clean up however the run ends
A flush is only called when the values run out normally. If a stage before it stops the run, or anything dies, there's no flush. For what must happen however the run ends, such as closing a file, the set-up can return a third sub, the end. It's called once, when the run is over, and told how it ended: done, stopped or died.
To give an end without a flush, name the parts, in a hash reference, rather than listing them in order:
my @log;
my $logged = gather {
my $n = 0;
return {
step => sub ($track) { $n++; $track },
end => sub ($how) { push @log, "$n tracks, $how" },
};
};
tracks($logged);
tracks(
$logged,
take(2),
);
eval { tracks(
$logged,
sub { die "No more\n" if $_->{artist} eq 'Cream'; $_ },
) };
say for @log;
# 6 tracks, done
# 2 tracks, stopped
# 6 tracks, died
The keys are step, flush and end, and you give only those you have. The list form, return ($step, $flush, $end), works too, with undef for a flush you leave out before an end. Either way, a stage with only a step can just return it.
Know what each part is told
Each part is told how the run ended from where it stands. A stop ends the run early for everything before it, so they're told stopped. What comes after a stop got everything it was going to get, so it's told done:
sub told ($name, $log) {
return gather {
return {step => sub ($v) { $v }, end => sub ($how) { push @$log, "$name $how" }};
};
}
my @told;
Ekorn::Pipeline->new(
told('before', \@told),
take(2),
told('after', \@told),
)->(1 .. 5);
say "@told"; # before stopped after done
This is what lets a part act on how the run ended: a part before a stop knows it wasn't read to its end, and a part after it knows it got everything it was going to get.
Choose the stages for each run
The set-up can return a whole pipeline in place of a step. Its stages are made for the run, in the gather's place, so the set-up can choose, or build, the stages each run gets:
my $quiet = 1;
my $voice = gather {
$quiet ? Ekorn::Pipeline->new(sub { lc }) : Ekorn::Pipeline->new(
sub { uc },
sub { "$_!" },
);
};
my $p = Ekorn::Pipeline->new($voice);
my @first = $p->('Hey');
$quiet = 0;
my @second = $p->('Hey');
say "@first / @second"; # hey / HEY!
Make a stage into an object with stage
The word stage, from Ekorn::Pipeline::Factories, makes an Ekorn::Pipeline::Stage from a block. It goes wherever a stage goes, and does what the block does:
my $title = stage { $_->{title} };
say tracks(
$title,
take(1),
);
# Dancing Queen
A plain sub works as well in a pipeline, so why bother? Two reasons. A Stage is one of Ekorn's own types, so | can start a chain with one (chapter 8), where a plain sub can't. And it keeps the code it was given, so a Stage made from \&trim is unaffected if trim is later redefined.
Tell values from a flush apart
While the values going through the stages come from a flush, rather than from the run's source, Ekorn::Pipeline::ending() is true:
my @lines = tracks(
sub { $_->{plays} },
collect { sum0 @_ },
sub { Ekorn::Pipeline::ending() ? "total $_" : $_ },
);
say "@lines"; # total 36800
It's for code that tells its stages more about each value, such as which row it came from: a value from a flush came from no row.
5. Sources: where values come from
A source is what an iterator pulls values from. Chapter 2 used the simplest kind, a sub that returns the next value. A plain sub has the same weakness as a plain stage, though: it's the same sub in every run, and it's never told when the run is over. A sub reading a file can't close it when the caller stops early.
This chapter's sources are made afresh for each run, and told when the run ends, however it ends. Ekorn::Pipeline::Sources has ready-made ones, and the word source makes your own.
Read values you already have
from_array gives the values it was given, from the first, on each run:
my $source = from_array(@tracks);
my $next = Ekorn::Pipeline->new(sub { $_->{artist} })->iterator($source);
say join ' ', $next->all; # Abba Abba Blur Blur Blur Cream
It's handy wherever a source is wanted and you have a list, as in chapter 8's chains. For pushing a list through stages, calling the pipeline is simpler.
Read a file, and close it however the run ends
from_file reads a file's lines, each without its newline. It opens the file as the run starts, and closes it as the run ends: when the lines run out, when a stage stops the run, when the iterator is finished or let go, or when a stage dies:
my $next = Ekorn::Pipeline->new(take(2))->iterator(from_file($setlist));
say join ' / ', $next->all; # Dancing Queen / Knowing Me, Knowing You
say $next->ended; # stopped
The run was stopped, so from_file closed the file there and then.
Give it I/O layers as a second argument: from_file($path, ':raw') for bytes, or from_file($path, ':encoding(UTF-8)') to decode text. If the file can't be opened, it croaks as the run starts, with Perl's reason: "Can't read setlist.txt: No such file or directory".
Read paragraphs, records, or the whole file
from_file reads records as Perl's $/ divides them, at each read, just as readline does. Set $/ while you pull, and it reads paragraphs, records of your own, or the whole file at once:
my $verses = "$dir/verses.txt";
open my $out, '>', $verses or die "Can't write $verses: $!\n";
print {$out} "You can dance\nYou can jive\n\nHaving the time\nof your life\n";
close $out or die "Can't write $verses: $!\n";
{
local $/ = ''; # paragraph mode
my @paragraphs = Ekorn::Pipeline->new->iterator(from_file($verses))->all;
say scalar @paragraphs; # 2
}
{
local $/; # the whole file, as one record
my($all) = Ekorn::Pipeline->new->iterator(from_file($verses))->all;
say length $all; # 57
}
Read a filehandle you opened yourself
from_fh reads a filehandle you already have, such as STDIN or a socket. It leaves the handle open when the run ends, because you opened it, and may want to read on:
my $text = "one\ntwo\nthree\n";
open my $fh, '<', \$text or die "Can't read the text: $!\n";
my @first = Ekorn::Pipeline->new(take(1))->iterator(from_fh($fh))->all;
my @rest = Ekorn::Pipeline->new->iterator(from_fh($fh))->all;
say "@first / @rest"; # one / two three
Count, without end
counter gives numbers from a start, by a step, for ever: 1, 2, 3 by default. Give it something that stops:
say join ' ', Ekorn::Pipeline->new(take(4))->iterator(counter(10, 5))->all; # 10 15 20 25
Read JSON, or JSON Lines
from_json reads a file holding one JSON document. If it's an array, its elements are the values; otherwise the document is:
my $chart = "$dir/chart.json";
open my $fh, '>', $chart or die "Can't write $chart: $!\n";
print {$fh} '[{"title":"Waterloo","plays":8000},{"title":"Parklife","plays":5800}]';
close $fh or die "Can't write $chart: $!\n";
my $next = Ekorn::Pipeline->new(sub { $_->{title} })->iterator(from_json(file => $chart));
say join ', ', $next->all; # Waterloo, Parklife
from_jsonl reads JSON Lines, a document a line, each as it is. It reads a block at a time, as the values are pulled, so a take reads no more of a big file than it needs; from_json has to read its document whole. Each takes fh => $fh in place of file, for a filehandle you opened, :raw.
Given no file, each is a stage, decoding each value it's given:
my @plays = Ekorn::Pipeline->new(
from_jsonl(),
sub { $_->{plays} },
)->('{"plays":8000}', '{"plays":5800}');
say "@plays"; # 8000 5800
Write a source that closes what it opens
The word source makes a source of your own. Its block is the source's set-up, called at the start of each run. It opens whatever the run needs, and returns the step, which gives the next value or an empty list at the end, and, if it needs one, the end, which is told how the run ended:
# The lines of a file, opened for each run, and closed once it ends,
# noting how each run ended in @$closed.
sub lines_of ($file, $closed = []) {
return source {
open my $fh, '<', $file or die "Can't read $file: $!\n";
return {
step => sub () { my $line = readline $fh; defined $line ? $line =~ s( \n \z )()xr : () },
end => sub ($how) { close $fh; push @$closed, $how },
};
};
}
my @closed;
my $next = Ekorn::Pipeline->new(sub { uc })->iterator(lines_of($setlist, \@closed));
say $next->(); # DANCING QUEEN
say $next->first; # KNOWING ME, KNOWING YOU
say "@closed"; # stopped
Each iterator calls the set-up for itself, so two iterators reading the same file each open it, and each close it, once.
Never leave a file open
Compare that with a plain sub reading a file. If the caller stops reading early, nothing tells the sub, and the file stays open until the sub itself is freed, whenever that is. The usual answers in Perl each have a gap: closing the file at the end of a loop misses a die or an early return; a guard object closes it when a scope ends, which isn't when the reading ends if the iterator was returned from a function. A source's end is called when the run ends, however it ends, and wherever the iterator went: when it's read to its end, stopped, finished, let go, or when anything in the run dies.
my @closed;
{
my $next = Ekorn::Pipeline->new->iterator(lines_of($setlist, \@closed));
$next->();
} # $next has gone, so the run has ended
say "@closed"; # stopped
This uses lines_of, from the recipe before.
Tell the stages about the run
A source's set-up can return a context, anything its stages may want to know about the run, such as a file's column names. A stage asks for it with Ekorn::Pipeline::run_context(), and each run has its own:
my $csv = "title,artist\nParklife,Blur\nWaterloo,Abba\n";
# Rows of a CSV string, as array references, its header the run's context.
sub rows_of ($text) {
return source {
my @lines = split m( \n )x, $text;
my @columns = split m( , )x, shift @lines;
return {
step => sub () { @lines ? [split m( , )x, shift @lines] : () },
context => {columns => \@columns},
};
};
}
# A stage giving the value in the named column of each row.
sub column ($name) {
return sub ($row) {
my $columns = Ekorn::Pipeline::run_context()->{columns};
my($at) = grep { $columns->[$_] eq $name } 0 .. $#$columns;
return $row->[$at];
};
}
say join ', ', Ekorn::Pipeline->new(column('artist'))->iterator(rows_of($csv))->all; # Blur, Abba
A real CSV reader would handle quoting, of course. The shape is what matters: the source knows something about its values, and a stage anywhere in the pipeline can ask.
Write a producer as a loop with generator
Some producers are easiest to write as a loop. generator takes a block that calls yield with each value, as a Python generator does:
my $pairs = generator {
for my $x (1 .. 3) {
for my $y ($x .. 3) { yield "$x$y" }
}
};
say join ' ', Ekorn::Pipeline->new->iterator($pairs)->all; # 11 12 13 22 23 33
The catch: Perl can't pause a sub in the middle, so a run calls the block once, to its end, and hands out what it yielded afterwards. So the block mustn't loop for ever, and does all its work even if only the first value is wanted. For a producer without end, or an expensive one, write a source whose step makes one value per call.
Read one pipeline's results as another's source
An iterator is a source too. Give one to another pipeline's iterator, and it reads the first pipeline's results, one at a time, as they're wanted:
my $titles = Ekorn::Pipeline->new(sub { $_->{title} })->iterator(from_array(@tracks));
my $short = Ekorn::Pipeline->new(sub { length $_ < 10 ? $_ : () })->iterator($titles);
say join ', ', $short->all; # Parklife
An iterator is one run, so it can be read through once. Ending the outer run ends the inner one too, so a file at the bottom is closed however the top one ends.
6. Sinks: where values end up
A sink is the opposite of a source: it takes every value it's given, and when they run out, gives one result. Put one last in a pipeline, and its result is what the run gives: a count, the first value, every value, or the number of lines written to a file.
Ekorn::Pipeline::Sinks has ready-made sinks, and the words sink_one and sink_many make your own.
Get a count, the first, or the last
count_items, keep_first and keep_last give one value each:
say scalar tracks(
sub { $_->{artist} eq 'Blur' ? $_ : () },
count_items,
);
# 3
say scalar tracks(
sub { $_->{artist} },
keep_first,
);
# Abba
say scalar tracks(
sub { $_->{artist} },
keep_last,
);
# Cream
keep_first and keep_last give undef when there were no values; count_items, 0.
Get every value, or the first few
keep_all gives every value; keep($n), the first $n. In list context, a list. Asked for one value, in scalar context, an array reference, so a list sink never gives you a mere count by accident:
my @titles = tracks(
sub { $_->{title} },
keep_all,
);
my $titles = tracks(
sub { $_->{title} },
keep_all,
);
say scalar @titles, ' titles, and ', scalar @$titles, ' by reference'; # 6 titles, and 6 by reference
my $two = tracks(
sub { $_->{title} },
keep(2),
);
say join ', ', @$two; # Dancing Queen, Knowing Me, Knowing You
An array reference is true even when it's empty, so test for no values with @$titles, or ask count_items.
Get the only one, and hear if there are more
keep_single gives the one value, for a run that should give exactly one. If there's a second, it warns, stops the run, and gives the first:
say scalar tracks(
sub { $_->{artist} eq 'Cream' ? $_->{title} : () },
keep_single,
);
# Sunshine of Your Love
Asked for "the track by Blur", it would warn "Expected only one item, but there were more": a forgotten limit, caught.
Stop reading once you have the answer
keep_first and keep($n) stop the run as soon as they have what they need, so a source is read no further:
my $read = 0;
my $first = Ekorn::Pipeline->new(
sub { $_->{plays} > 6000 ? $_ : () },
keep_first,
)->iterator(sub { $read < @tracks ? $tracks[$read++] : () })->();
say "$first->{title}, after reading $read"; # Dancing Queen, after reading 1
A sink that stops the run is told done: it decided the run had had enough. The source is told stopped, and a source reading a file closes it there and then.
Write to a file, or a filehandle
to_file writes each value to a file, each followed by a newline, and gives how many it wrote. It opens the file as the run starts, emptying it, and closes it however the run ends. to_fh writes to a filehandle you opened, and leaves it open:
my $hits = "$dir/hits.txt";
my $n = tracks(
sub { $_->{plays} >= 6000 ? $_->{title} : () },
to_file($hits),
);
say "$n written"; # 3 written
open my $fh, '>', \my $text or die "Can't write the text: $!\n";
Ekorn::Pipeline->new(to_fh($fh))->iterator(from_file($hits))->();
print $text;
# Dancing Queen
# Girls & Boys
# Sunshine of Your Love
from_file takes the newline off each line, and to_file puts it back, so a file read with one and written with the other is copied line for line.
Write JSON, or JSON Lines
to_jsonl writes each value as a line of JSON Lines, and to_json every value as one array, so the file is one JSON document. Each gives how many it wrote. The json option passes options to the encoder:
my $top = "$dir/top.jsonl";
my $n = tracks(
sub { $_->{plays} >= 7000 ? {title => $_->{title}, plays => $_->{plays}} : () },
to_jsonl(file => $top, json => {canonical => 1}),
);
say "$n written"; # 2 written
open my $in, '<', $top or die "Can't read $top: $!\n";
print <$in>;
# {"plays":9100,"title":"Dancing Queen"}
# {"plays":7700,"title":"Sunshine of Your Love"}
Given no file, each is a stage passing on what a file would hold: to_jsonl each value as a line of JSON, ready for to_file, and to_json every value as one array, once they've all come:
say tracks(
sub { $_->{artist} },
take(3),
to_json(),
);
# ["Abba","Abba","Blur"]
Write a sink of your own
The words sink_one and sink_many make sinks. The block is the sink's set-up, called at the start of each run, and returns the step, called with each value, and the flush, called when the values run out, giving the result. sink_one makes a sink giving one value, and sink_many one giving a list:
my $longest = sink_one {
my $max = '';
return {
step => sub ($title) { $max = $title if length $title > length $max },
flush => sub () { $max },
};
};
say scalar tracks(
sub { $_->{title} },
$longest,
);
# Knowing Me, Knowing You
my $artists = sink_many {
my %seen;
return {
step => sub ($track) { $seen{ $track->{artist} }++ },
flush => sub () { sort keys %seen },
};
};
say join ' ', tracks($artists); # Abba Blur Cream
What a sink's step returns is ignored: a sink passes nothing on until its flush.
Commit only when the whole run went through
A sink can have an end too, told how the run ended. One that should take effect only if every value arrived checks for done. This one writes to a temporary file, and moves it into place only then, so a run that dies, even after the sink, leaves the old file as it was:
use File::Temp qw(tempfile);
sub save_to ($path) {
return sink_one {
my($fh, $tmp) = tempfile(DIR => $dir);
my $n = 0;
return {
step => sub ($line) { say {$fh} $line; $n++ },
flush => sub () { $n },
end => sub ($how) {
close $fh;
$how eq 'done' ? rename $tmp, $path : unlink $tmp;
},
};
};
}
my $titles = "$dir/titles.txt";
say scalar tracks(
sub { $_->{title} },
save_to($titles),
);
# 6
eval { tracks(
sub { die "Not today\n" if $_->{artist} eq 'Cream'; $_->{title} },
save_to($titles),
) };
say scalar Ekorn::Pipeline->new(count_items)->iterator(from_file($titles))->(); # 6
The second run died at the last track, so its five lines were thrown away, and the six from the first run are still there. The same shape commits a database transaction only when the run is done, and rolls it back otherwise.
Go on after a sink
A sink needn't be last. Anywhere else, its result goes on through the stages after it, as any value does:
say tracks(
sub { $_->{artist} eq 'Blur' ? $_ : () },
count_items,
sub { "$_ by Blur" },
);
# 3 by Blur
Use the stage that matches a sink
Each keep sink but keep_all has a matching stage in Flow, which passes on what the sink would give, so more stages can follow: take($n) for keep($n), and take_first, take_last and take_single for the other three. Use the sink when the value is the run's result, and the stage when there's more to do with it.
Build a hash, with no sink at all
There's no sink for a hash, because you don't need one: a hash is assigned from a list of keys and values, so give both from a stage:
my %plays = tracks(sub { ($_->{title}, $_->{plays}) });
say $plays{Parklife}; # 5800
Use a pipeline's result as another's source
A pipeline that reads from a source, and ends in a sink, gives the sink's result when it's pulled. So it's a source of that result, and another pipeline can read from it:
my $in_order = sink_many {
my @all;
return {step => sub ($line) { push @all, $line }, flush => sub () { sort @all }};
};
my $sorted = Ekorn::Pipeline->new($in_order)->iterator(from_file($setlist));
say join ', ', Ekorn::Pipeline->new(take(2))->iterator($sorted)->all; # Country House, Dancing Queen
Sorting needs every value before it can give the first, so the sink holds them all, and the file is read to its end, and closed, before anything comes out. That's the trade: a stage or sink that needs every value makes the run read every value, whatever comes after it.
7. Building pipelines from pipelines
Stages put together once can be used again: as part of another pipeline, in a function that makes stages, or under a name. This chapter shows the ways to reuse them, and the one difference between them that matters, which is whether the stages join the run they're put in, or run separately for each value.
Use a pipeline as a stage
A pipeline can go wherever a stage goes. Its stages are spliced in where it stands, and run as part of the new pipeline, just as if you'd written them out:
my $clean = Ekorn::Pipeline->new(
sub { s( \A \s+ | \s+ \z )()xgr },
sub { length ? $_ : () },
);
my @titles = Ekorn::Pipeline->new(
$clean,
take(2),
)->(' Parklife ', ' ', 'Tender', 'Waterloo');
say join '|', @titles; # Parklife|Tender
Because the stages join the run, a stop in any of them ends the whole run, and a gather among them starts afresh with it.
Hand over just the stages
stages gives a pipeline's stages as a list, for a function that won't take a pipeline, or for when you want to add to them:
my $clean = Ekorn::Pipeline->new(
sub { s( \A \s+ | \s+ \z )()xgr },
sub { length ? $_ : () },
);
my @more = ($clean->stages, sub { uc });
say join '|', Ekorn::Pipeline->new(@more)->(' tender ', ' '); # TENDER
say scalar $clean->stages; # 2
Run a pipeline on each value separately
Call a pipeline from inside a stage, and each call is a run of its own, for that one value. The difference shows with a stage that stops, or keeps state:
my $two_words = Ekorn::Pipeline->new(
sub { split ' ' },
take(2),
);
my @spliced = tracks(
sub { $_->{title} },
$two_words,
);
my @separate = tracks(
sub { $_->{title} },
sub { join ' ', $two_words->($_) },
);
say "@spliced"; # Dancing Queen
say join ' / ', @separate[0, 1]; # Dancing Queen / Knowing Me,
Spliced in, take(2) limited the whole run to two words. Called from a stage, it limited each title to two words, and the run went on.
Build a vocabulary of stages
A stage is a value, so a function can make one. A handful of such functions becomes a small language for your data, which reads like what it does:
sub by ($artist) { sub { $_->{artist} eq $artist ? $_ : () } }
sub value_of ($field) { sub { $_->{$field} } }
sub at_least ($plays) { sub { $_->{plays} >= $plays ? $_ : () } }
say join ', ', tracks(
by('Blur'),
at_least(5000),
value_of('title'),
);
# Girls & Boys, Parklife
The same words work with every function that takes stages, whoever wrote it. Chapter 10 shows how to package them for others.
Name a pipeline, and use it by name
named keeps a pipeline under a name, and the word pipeline, from Ekorn::Pipeline::Factories, makes a pipeline in which a name stands for the pipeline it names. A named pipeline can be made from other names:
Ekorn::Pipeline->named(hit => at_least(6000));
Ekorn::Pipeline->named(title => value_of('title'));
Ekorn::Pipeline->named(hit_titles =>
'hit',
'title',
);
say join ', ', tracks(pipeline(
'hit_titles',
take(2),
));
# Dancing Queen, Girls & Boys
This uses at_least and value_of, from the recipe before. To name a pipeline you already have, call named on it: $clean->named('clean').
A named pipeline's name also appears in error messages, which makes a failing stage easier to find (see Find the stage that failed).
Know whose name it is
A name belongs to the package that defined it, so two modules can each have a clean without clashing. pipeline looks a name up in the package that called it; to use another package's, qualify it:
package My::Music {
Ekorn::Pipeline->named(loud => sub { uc });
}
say pipeline('My::Music::loud')->('waterloo'); # WATERLOO
say eval { pipeline('loud') } ? 'found' : 'not in main'; # not in main
The catch: names last for the life of the program, can't be removed, and can't be defined twice, so define each once, as a module loads, never in a sub that may run again. And a name is looked up when the pipeline is made, not when it runs, so a pipeline keeps what it found even if a name is defined later. That's a deliberate trade: no surprises at run time, in return for a little care in setting names up.
Name pipelines as the program compiles
Ekorn::Pipeline::Sugar names pipelines as the program compiles, and gives your package a function of each name, so a pipeline you use in several places is called like any other function:
use Ekorn::Pipeline::Sugar pipeline => {
song_titles => [sub { $_->{title} }],
first_two => ['song_titles', take(2)],
};
say join ', ', first_two(@tracks); # Dancing Queen, Knowing Me, Knowing You
say scalar first_two(@tracks); # 2
say join ', ', first_two(@tracks, pipeline(sub { uc }));
# DANCING QUEEN, KNOWING ME, KNOWING YOU
Each function runs its values through the pipeline, and returns what calling the pipeline would: a list, or, asked for one value, how many came out, or a sink's result. Give it a pipeline last, and it runs the values through that too, so callers can add stages of their own.
A string names a pipeline, as in pipeline: one from the same use, in any order, or one defined before it. Each is named in your package, too, so pipeline('first_two')->iterator($source) pulls values through it. Give the definitions as an array of pairs, [...], to read them in the order you wrote them. A mistake, such as a name already taken, or two pipelines that name each other, stops the program as it compiles, at the use line.
8. Chaining with |
With Ekorn::Pipeline::Operators in scope, sources, stages and sinks join with |, as commands do in a Unix shell. It's a shorthand for the commonest shape, source to stages to sink, when that reads best as one line. Everything it does can be done with new and iterator, which can also do things a chain can't.
Chain a source, stages and a sink
use Ekorn::Pipeline::Operators puts | in force for the rest of the scope. A chain from a source to a sink runs at once, and gives the sink's result:
use Ekorn::Pipeline::Operators;
my $blur = stage { $_->{artist} eq 'Blur' ? $_ : () };
my $count = from_array(@tracks)
| $blur
| count_items;
say $count; # 3
my $first = from_array(@tracks)
| $blur
| stage { $_->{title} }
| keep_first;
say $first; # Girls & Boys
Get a list from a chain
| gives one value, so a chain ending in a sink of many, such as keep_all, gives its list as an array reference. Dereference it for a list:
use Ekorn::Pipeline::Operators;
my $titles = from_array(@tracks)
| stage { $_->{title} }
| take(2)
| keep_all;
say join ', ', @$titles; # Dancing Queen, Knowing Me, Knowing You
my @titles = @{
from_array(@tracks)
| stage { $_->{title} }
| keep_all
};
say scalar @titles; # 6
my @titles = ... | keep_all, without the @{ }, gets one element: the reference.
Keep a source with its stages
A source joined to stages, with no sink yet, is a new source: the values, through the stages. Keep it, and end it in different sinks; each chain from it is a run of its own:
use Ekorn::Pipeline::Operators;
my $hits = from_array(@tracks) | stage { $_->{plays} >= 6000 ? $_->{title} : () };
say $hits | count_items; # 3
say $hits | keep_first; # Dancing Queen
my $next = Ekorn::Pipeline->new->iterator($hits);
while (my($title) = $next->()) {
say "Now playing: $title";
}
# Now playing: Dancing Queen
# Now playing: Girls & Boys
# Now playing: Sunshine of Your Love
Build a chain of stages, and add to it
Stages joined with |, with no source, make a pipeline you can keep and use anywhere a pipeline goes. |= adds a stage to one:
use Ekorn::Pipeline::Operators;
my $steps = stage { $_->{title} }
| stage { uc };
$steps |= take(2);
say join ', ', tracks($steps); # DANCING QUEEN, KNOWING ME, KNOWING YOU
my $titles = from_array(@tracks)
| $steps
| keep_all;
say join ', ', @$titles; # DANCING QUEEN, KNOWING ME, KNOWING YOU
Read a source through with @{}
@{ $source } reads a source to its end, with no sink, and gives every value:
use Ekorn::Pipeline::Operators;
my $loud = from_file($setlist) | stage { uc };
my @all = @{$loud};
say $all[0]; # DANCING QUEEN
The catch: it looks like an array, but there isn't one. Each use runs the source from its start to its end, however little you take: $loud->[0] reads every line to give the first, and scalar @$loud keeps every line to count them. Use keep_first, count_items, take or an iterator instead when you want less than everything.
Mind Perl's own |
Two things to know, both from Perl, not Ekorn:
Start a chain with one of Ekorn's types: a source, a stage made with
stage, or a ready-made one such astake(2).sub { ... } | sub { ... }is Perl's bitwise OR of two addresses, which gives a meaningless number, with no error.**
|binds more loosely than.and+,** so put a chain inside a larger expression in parentheses:'Blur: ' . ($source | $blur | count_items).
And keep a chain's result, even one run only for what its sink does, as in my $copied = from_file($a) | to_file($b). Written as a statement on its own, it still runs, but Perl warns of a useless |.
Know when to use new instead
A chain is a shorthand, and some things need the long hand. Use new and iterator for a pipeline with stages after its sink, for a list result as a list rather than a reference, and for a pipeline pulled as a source of its sink's result. A pipeline made with new has no operators of its own.
9. Functions that take stages
This is what Ekorn::Pipeline is for. Every chapter so far has been about the pieces; this one is about putting them where they pay off most: in the signature of a function that hands out values.
A function that hands out values can't know everything its callers will want done with them. One wants only some, another wants them reshaped, a third wants the first ten. The usual answers are options, which grow for ever, or returning everything and leaving each caller to loop. A function that takes stages does neither: its caller says what they want, beside the call, and the function does only what it alone can.
Take stages from your caller
Take stages as your last arguments, and run what you'd return through them:
sub tracks_by ($artist, @stages) {
my @by = grep { $_->{artist} eq $artist } @tracks;
return Ekorn::Pipeline->new(@stages)->(@by);
}
say join ', ', tracks_by(
'Blur',
sub { $_->{title} },
take(2),
);
# Girls & Boys, Parklife
say scalar tracks_by('Abba'); # 2
A caller who wants nothing done gives no stages, and gets the values as they are.
Replace options with stages
Here's the difference, side by side. With options, every way of shaping the result is code the author writes, tests, documents and maintains, and the options interact: does limit count before or after min_plays?
# With options
sub hits_by_option (%opt) {
my @found = @tracks;
@found = grep { $_->{plays} >= $opt{min_plays} } @found if $opt{min_plays};
@found = map { $_->{title} } @found if $opt{titles};
@found = splice @found, 0, $opt{limit} if $opt{limit};
return @found;
}
# With stages
sub hits_by_stage (@stages) {
return Ekorn::Pipeline->new(@stages)->(@tracks);
}
my @a = hits_by_option(min_plays => 6000, titles => 1, limit => 2);
my @b = hits_by_stage(
at_least(6000),
value_of('title'),
take(2),
);
say "@a" eq "@b" ? 'the same' : 'different'; # the same
This uses at_least and value_of, from Build a vocabulary of stages. The second function does everything the first does, and everything it doesn't, without a line more; its stages run in the order the caller wrote them, so there's no question to answer about order.
The catch, to be fair: options are easier to discover, as a list in the documentation, and a caller of hits_by_stage has to know what a stage is. And an option the function can act on itself, better than a stage could, should stay an option: which file to read, or a filter a database can apply with an index, before any row is fetched. Keep those; let stages do the rest.
Return an iterator, so callers can stop your work
If your function makes its values one at a time, give iterator a source and return the iterator. Then a caller's stage that stops stops your work too. Here a function reads a log file, closing it however its caller stops:
my $log = "$dir/app.log";
open my $out, '>', $log or die "Can't write $log: $!\n";
print {$out} map { "$_\n" } 'INFO started', 'ERROR disk full', 'INFO retrying', 'ERROR disk still full';
close $out or die "Can't write $log: $!\n";
# The log's lines, through the caller's stages, read only as they're wanted.
sub log_lines ($file, @stages) {
return Ekorn::Pipeline->new(@stages)->iterator(from_file($file));
}
my $first_error = log_lines(
$log,
sub { m( \A ERROR \ (.*) )x ? $1 : () },
take(1),
);
say $first_error->all; # disk full
The caller got one error, the function read two lines, and the file was closed as take(1) stopped the run. The function didn't need to know any of that was going to happen.
Do some of the work yourself, then the caller's
Put stages of your own before your caller's, so their stages are given values in the shape you promise:
# Titles, through whatever stages the caller gives.
sub titles (@stages) {
return Ekorn::Pipeline->new(sub { $_->{title} }, @stages)->(@tracks);
}
say join ', ', titles(drop(4)); # Country House, Sunshine of Your Love
Say in your documentation what the caller's first stage is given.
Pass your caller's stages on
A function built on another that takes stages passes its caller's on, after its own:
sub hits (@stages) {
return tracks(at_least(6000), @stages);
}
say join ', ', hits(sub { $_->{artist} }); # Abba, Blur, Cream
Find stages after values of any kind
A function that takes a list of values of its own can't tell where the stages start by counting. is_stage, from Ekorn::Pipeline, tells a stage from a value, the same way new does, so it takes pipelines and ready-made stages too:
use Ekorn::Pipeline qw(is_stage);
# The values given, through the stages after them.
sub through (@args) {
my @stages;
unshift @stages, pop @args while @args && is_stage($args[-1]);
return Ekorn::Pipeline->new(@stages)->(@args);
}
say join ', ', through('Parklife', 'Country House', 'Tender', sub { length $_ < 10 ? $_ : () }); # Parklife, Tender
The catch: the values mustn't be code references or pipelines, or the last of them would be taken for stages. If they can be, use the next recipe. And a function that takes a hash reference of options first should test is_stage before is_hashref, which is true of a pipeline too.
Take a pipeline given last
If your values can be code references, such as callbacks, take the stages as one pipeline, given last. A pipeline is never one of the values, and the word pipeline makes one for the caller in a few characters:
# What each callback returns, through the stages given last, as a pipeline.
sub call_all (@callbacks) {
my $stages = @callbacks && $callbacks[-1] isa 'Ekorn::Pipeline' ? pop @callbacks : pipeline();
return $stages->(map { $_->() } @callbacks);
}
my @plays = map { my $track = $_; sub { $track->{plays} } } @tracks;
say join ', ', call_all(@plays, pipeline(
sub { $_ > 6000 ? $_ : () },
take(2),
));
# 9100, 6100
Report errors at your caller's line
When a stage dies, the error is reported at the line that ran the pipeline. In a module, that's a line inside your function, which isn't where the caller's mistake is. Add Ekorn::Pipeline's packages to your module's @CARP_NOT, and the error is reported at your caller's line, where they wrote the stage:
package My::Catalogue {
use Ekorn::Pipeline;
our @CARP_NOT = @Ekorn::Pipeline::Errors::INTERNAL;
sub titles (@stages) {
return Ekorn::Pipeline->new(sub { $_->{title} }, @stages)->(@tracks);
}
}
my $line = __LINE__ + 1;
eval { My::Catalogue::titles(sub { die "Not today\n" }) };
say $@ =~ m( line \ $line: )x ? 'reported at my line' : 'reported elsewhere'; # reported at my line
Give a class pipe and tap
A class that inherits from Ekorn::Pipeline::Pipes gets two methods that run the object through stages. pipe gives what comes out; tap gives the object back, so a chain of method calls can carry on:
package Playlist {
use parent 'Ekorn::Pipeline::Pipes';
sub new ($class, @tracks) { bless {tracks => [@tracks]}, $class }
sub tracks ($self) { @{ $self->{tracks} } }
sub size ($self) { scalar $self->tracks }
}
my $list = Playlist->new(@tracks[0 .. 2]);
my @artists = $list->pipe(
sub { $_->tracks },
sub { $_->{artist} },
);
say "@artists"; # Abba Abba Blur
say $list->tap(sub { warn "a long list\n" if $_->size > 10; $_ })->size; # 3
Compared with a callback
The other common way to let a caller act on each value is a callback: each_track(sub { ... }). Stages are a callback grown up:
Several, not one. A caller gives as many stages as they like, each doing one thing, instead of one callback doing everything.
They return what goes on. A callback's return is usually ignored; a stage's return is the result, so filtering and reshaping need no array outside the call.
They can stop. Stopping a callback early needs an exception or a flag that the function checks, written specially for that function. A stop works the same way in every function that takes stages.
They're reusable.
take(10)and your ownby('Blur')work with every function that takes stages, not just the one they were written for.
What callbacks have going for them: everyone knows them already, and a single callback call is cheaper than a pipeline run.
10. Building parts for others
Once a stage, a source or a sink is useful to you, it's usually useful to someone else. This chapter is about writing them so they work alongside Ekorn's own, whoever uses them: a module of stages, and your own classes as sources and sinks, as DBIx::Squirrel's database rows are.
Write a function that makes a stage
For a stage others will use, write a function that returns it, as take(10) does. The function can take options, and check them once, when it's called, rather than on every value. croakf, from Ekorn::Pipeline::Errors, reports a mistake at the line that called your function, with a message made by sprintf:
package My::Stages::Words {
use Ekorn::Pipeline::Errors qw(croakf);
sub longer_than ($length) {
croakf 'Expected a length of 0 or more, not %s', $length // 'undef'
unless defined $length && $length =~ m( \A \d+ \z )xa;
return sub ($word) { length $word > $length ? $word : () };
}
}
say join ', ', tracks(sub { $_->{title} }, My::Stages::Words::longer_than(14));
# Knowing Me, Knowing You, Sunshine of Your Love
eval { My::Stages::Words::longer_than(-1) };
print $@ =~ s( \ at \ .* )( at your line)xsr; # Expected a length of 0 or more, not -1 at your line
A real module would export longer_than; it's called by its full name here only so the example runs as it stands.
Keep a stage's state with gather
A stage that remembers anything must be made with gather, so it starts afresh on each run, as chapter 4 showed. A stage that drops repeats is a good example:
sub distinct () {
return gather {
my %seen; # this run's
return sub ($value) { $seen{$value}++ ? () : $value };
};
}
my $artists = Ekorn::Pipeline->new(sub { $_->{artist} }, distinct());
say join ' ', $artists->(@tracks); # Abba Blur Cream
say join ' ', $artists->(@tracks); # Abba Blur Cream
Made without gather, the second line would be empty: %seen would remember the first run.
Pass the last value on with the stop
A stage that knows a value is the last one wanted should send it with its stop, as take does. Then the run ends without asking its source for one more:
sub up_to ($wanted) {
return sub ($value) { $value eq $wanted ? Stop($value) : $value };
}
say join ', ', tracks(sub { $_->{title} }, up_to('Girls & Boys'));
# Dancing Queen, Knowing Me, Knowing You, Girls & Boys
up_to compares the whole value, so it's given titles here, not tracks. Say in a stage's documentation what it expects to be given.
Make your objects sources and sinks
Your own classes can be sources and sinks, with no need to load Ekorn or inherit from anything. An object says what it can be by its methods, named in capitals because Ekorn calls them, not your users:
EKORN_SOURCE, to be read from;EKORN_SINK_ONEorEKORN_SINK_MANY, to be written to, giving one value or a list;EKORN_STAGE, to be a stage with state of its own.
Each is called at the start of a run, and returns the parts, as a set-up does: a step, and optionally a flush, an end, and, for a source, a context. Here's a queue that can be both read from and written to:
package My::Queue {
sub new ($class, @items) { bless {items => [@items]}, $class }
# Read: each item, taken off the front of the queue.
sub EKORN_SOURCE ($self) {
return {step => sub () { @{ $self->{items} } ? shift @{ $self->{items} } : () }};
}
# Write: each item added to the back; the result, how many.
sub EKORN_SINK_ONE ($self) {
my $added = 0;
return {
step => sub ($item) { push @{ $self->{items} }, $item; $added++ },
flush => sub () { $added },
};
}
}
my $queue = My::Queue->new('Waterloo');
say scalar tracks(
sub { $_->{title} },
take(2),
$queue,
);
# 2
say join ', ', Ekorn::Pipeline->new(sub { uc })->iterator($queue)->all;
# WATERLOO, DANCING QUEEN, KNOWING ME, KNOWING YOU
Your users run your objects in Ekorn's pipelines, with Ekorn's stages and their own. Don't subclass Ekorn's classes or wrap them to do this: the methods are the whole contract, so your package and Ekorn can each change their insides freely. The reference's Making each kind of part says what each method is called with and must return, with a worked example of each.
Tell your stages about the run
A source can give its run a context, which stages ask for with Ekorn::Pipeline::run_context(), as the CSV source in chapter 5 did. That's how DBIx::Squirrel's pluck('name') finds the column called name in each row: the statement's source gives the column names as the context, and the stage looks them up. Stages like these go in your package, as only it knows what the context holds.
Let users chain your sources with |
To let your users start a chain with your source, overload | and @{} with Ekorn's own functions for them. Load Ekorn::Pipeline::Operators with an empty import list, so it doesn't put the operators in force in your own code:
package My::Chained::Queue {
our @ISA = ('My::Queue');
use Ekorn::Pipeline::Operators ();
use overload
'|' => \&Ekorn::Pipeline::Operators::chain,
'@{}' => \&Ekorn::Pipeline::Operators::drain;
}
use Ekorn::Pipeline::Operators;
my $queue = My::Chained::Queue->new(qw(one two three));
my $words = $queue
| stage { uc }
| take(2)
| keep_all;
say join ' ', @$words; # ONE TWO
This uses My::Queue, from the recipe before.
Decide where a stage belongs
Where a stage goes depends on what it needs to know. Ask: could it work on values from anywhere, or does it need something only one module or source knows?
Stages for any values, whatever they are and wherever they come from, such as one that drops repeats or keeps every tenth value, are general. They can go in a module of their own, or be offered to Ekorn::Pipeline itself.
Stages for one kind of value, or one source, go with the module that knows about it. DBIx::Squirrel's
pluckneeds a statement's column names, so it's in DBIx::Squirrel, not here.
What the values happen to be isn't enough to decide: a stage that works on hash references from a database works as well on hash references from a JSON file. Only what it has to know decides.
Test a stage
Calling a pipeline with values tests a stage, with no source to set up. Run it twice, to show it starts afresh, and with no values, to show what it does with none:
use Test2::V0;
my $p = Ekorn::Pipeline->new(distinct());
is [$p->(qw(a b a))], [qw(a b)], 'drops repeats';
is [$p->(qw(a))], [qw(a)], 'afresh on each run';
is [$p->()], [], 'nothing, from nothing';
done_testing;
# ok 1 - drops repeats
# ok 2 - afresh on each run
# ok 3 - nothing, from nothing
# 1..3
This uses distinct, from Keep a stage's state with gather. Test a stage that stops through an iterator too, counting the source's calls, to show it reads no more than it should.
Walk through a module of parts
A module of parts is a few functions, each checking its options once and returning a part made with Ekorn's words. Here's one for JSON Lines, built in four steps: a cut-down from_jsonl, which reads them, and to_jsonl, which writes them, as Ekorn::Pipeline::JSON's do. Each is a source or a sink given a file, and a stage given none, so the caller gets one name for each direction, whatever they're doing.
1. The package. It exports on request, with an :all tag, and reports its users' mistakes at their lines, by putting Ekorn's internal packages in its @CARP_NOT and raising errors with croakf. The options are checked once, as a function is called, not as each value goes by:
package My::JSONL {
use Exporter qw(import);
use JSON::PP qw();
use Ekorn::Pipeline::Errors qw(croakf);
use Ekorn::Pipeline::Factories qw(sink_one source stage);
our @EXPORT_OK = qw(from_jsonl to_jsonl);
our %EXPORT_TAGS = (all => \@EXPORT_OK);
our @CARP_NOT = @Ekorn::Pipeline::Errors::INTERNAL;
# The options given, checked: only file is known.
sub _options (@given) {
croakf 'Expected options as name => value pairs' if @given % 2;
my %option = @given;
my @unknown = sort grep { $_ ne 'file' } keys %option;
croakf 'Expected the option file, not %s', join ', ', @unknown if @unknown;
return %option;
}
}
2. Reading: a source, or a stage. Given a file, from_jsonl returns a source. Its block is the set-up, run as each run starts, so the file is opened for that run alone, and it returns the run's parts: a step, giving the next line's document, or nothing once the lines run out, and an end, which closes the file however the run ended. Given no file, it returns a stage decoding each value. A decoder that parses each line whole keeps nothing between values, so one can serve every run:
package My::JSONL {
# A source of the documents in a file of JSON Lines, given a file;
# a stage decoding each value, given none.
sub from_jsonl (@options) {
my %option = _options(@options);
my $json = JSON::PP->new->utf8->allow_nonref;
return stage { $json->decode($_) } unless defined $option{file};
my $path = $option{file};
return source {
# This run's: opened as it starts, and closed as it ends.
open my $fh, '<:raw', $path or croakf q(Can't read %s: %s), $path, $!;
return {
step => sub () {
my $line = readline $fh;
return unless defined $line; # nothing: the source is done
return $json->decode($line);
},
end => sub ($how) { close $fh },
};
};
}
}
3. Writing: a sink, or a stage. Given a file, to_jsonl returns a sink_one, a sink whose result is one value. Its set-up opens the file for the run; its step writes each value; its flush, called once the values have all come, gives the result, how many; and its end closes the file, croaking if it can't, as a full disk shows itself there. Given no file, it returns a stage encoding each value, which a caller can send on to to_file, or anywhere else:
package My::JSONL {
# A sink writing each value as a line of JSON Lines to a file, giving
# how many, given a file; a stage encoding each value, given none.
sub to_jsonl (@options) {
my %option = _options(@options);
my $json = JSON::PP->new->utf8->allow_nonref->canonical;
return stage { $json->encode($_) } unless defined $option{file};
my $path = $option{file};
return sink_one {
# This run's: emptied as it starts, and closed as it ends.
open my $fh, '>:raw', $path or croakf q(Can't write %s: %s), $path, $!;
my $n = 0;
return {
step => sub ($value) { print {$fh} $json->encode($value), "\n"; $n++ },
flush => sub () {$n},
end => sub ($how) { close $fh or croakf q(Can't write %s: %s), $path, $! },
};
};
}
}
4. Using it. Each part goes where Ekorn's own do, with Ekorn's stages and the caller's own. A real module would be loaded with use My::JSONL qw(:all); its functions are called by their full names here only so the example runs as it stands:
my $saved = "$dir/tracks.jsonl";
my $n = tracks(My::JSONL::to_jsonl(file => $saved));
say "$n written"; # 6 written
my $next = Ekorn::Pipeline->new(
take_if { $_->{plays} >= 7000 },
sub { $_->{title} },
)->iterator(My::JSONL::from_jsonl(file => $saved));
say join ', ', $next->all; # Dancing Queen, Sunshine of Your Love
say Ekorn::Pipeline->new(My::JSONL::to_jsonl())->({title => 'Waterloo', plays => 8000});
# {"plays":8000,"title":"Waterloo"}
eval { My::JSONL::from_jsonl(flie => $saved) };
print $@ =~ s( \ at \ .* )( at your line)xsr; # Expected the option file, not flie at your line
Ekorn::Pipeline::JSON is built the same way, with more to it:
A function for each format,
from_jsonandto_jsonfor one document, andfrom_jsonlandto_jsonlfor JSON Lines, so the name says what's read or written, with no option to set or guess.Its sources read blocks, not lines, through the decoder's incremental parser, so a document can span lines.
**
from_jsonl's stage is made withgather,** notstage, as the incremental parser holds what it has read: each run gets a parser of its own, made in the gather's block, so two runs never share one. State that lasts between values is the sign a part needs a set-up.**
to_json's stage needs a flush:** it holds every value until they've all come, and passes on one array then, which isgather's flush, the stage's counterpart of a sink's.Its encoder can be configured or replaced, and each option is checked as the function is called, with a message for each mistake, as
_optionsdoes here.
Read its source for the whole of it: it's one file, of under three hundred lines of code.
Publish a module of stages
Export the stages a module is mostly used for by default, and the rest on request, with an
:alltag, as Ekorn's own modules do.Choose names that don't clash with what common modules export. Ekorn has
drop, notskip, because Test2::V0 and Test::More both exportskip; andkeep_first, notfirst, because of List::Util.**Put
@Ekorn::Pipeline::Errors::INTERNALin your@CARP_NOT,** so your users' mistakes are reported at their lines.Require the versions of Ekorn::Pipeline you tested with, up to the next change of the middle number, which is when the way stages behave can change. In a
cpanfile:requires 'Ekorn::Pipeline', '>= v0.3.0, < v0.4.0';
11. When things go wrong
A pipeline's errors say which stage failed, and where. This chapter shows how to read them, and the stages that help when a pipeline gives the wrong answer rather than an error.
Find the stage that failed
A stage that dies is named by its place in the pipeline, with its own message below, indented:
eval { tracks(
sub { $_->{title} },
sub { die "No more Blur\n" if m( Boys )x; $_ },
) };
print $@ =~ s( \ at \ \S+ \ line \ \d+ )( at ... line ...)xr;
# Pipeline stage #2 failed at ... line ...:
# No more Blur
The line is the one that ran the pipeline: with an iterator, the line that asked for the result. A module can make it its caller's line, as Report errors at your caller's line shows.
Find a stage inside a pipeline inside a pipeline
A stage inside a pipeline used as a stage is named by its place in each: here the failing stage is the second of the second, #2.2. And a pipeline with a name is named too, which is easier to find than a number:
my $no_blur = Ekorn::Pipeline->new(
sub { $_->{title} },
sub { die "No more Blur\n" if m( Boys )x; $_ },
);
eval { tracks(
sub { $_ },
$no_blur,
) };
print $@ =~ s( \ at \ \S+ \ line \ \d+ )( at ... line ...)xr;
# Pipeline stage #2.2 failed at ... line ...:
# No more Blur
$no_blur->named('no_blur');
eval { tracks(
sub { $_ },
$no_blur,
) };
print $@ =~ s( \ at \ \S+ \ line \ \d+ )( at ... line ...)xr;
# Pipeline stage #2.2 (main::no_blur#2) failed at ... line ...:
# No more Blur
main::no_blur#2 is the second stage of the pipeline named main::no_blur, wherever it's used.
Find a flush or an end that failed
A flush that dies is named by its stage's place, as a stage is. An end that dies is named as the end of its stage, and the other ends are still called, told died:
my $fussy = gather {
return {step => sub ($v) { $v }, end => sub ($how) { die "Couldn't tidy up\n" }};
};
eval { tracks($fussy) };
print $@ =~ s( \ at \ \S+ \ line \ \d+ )( at ... line ...)xr;
# Pipeline end of stage #1 failed at ... line ...:
# Couldn't tidy up
Throw your own exception objects
An exception object is thrown on unchanged, not wrapped in a message, so a caller can catch it by its class:
package Not::Found { sub new ($class, %args) { bless {%args}, $class } }
eval { tracks(sub { die Not::Found->new(title => $_->{title}) }) };
say ref $@, ': ', $@->{title}; # Not::Found: Dancing Queen
See mistakes as the pipeline is made
A mistake in making a pipeline or a ready-made stage, such as a stage that isn't one, or take(-1), is reported at once, at the line that made it, before any value goes through:
eval { Ekorn::Pipeline->new(sub { $_ }, 'title') };
print $@ =~ s( \ at \ .* )()xsr, "\n";
# Expected stage #2 to be a stage: a code reference, a pipeline, or an object offering a stage's role method
eval { take(-1) };
print $@ =~ s( \ at \ .* )()xsr, "\n"; # Expected a count of 0 or more
Diagnostics, in the reference, lists every message and what it means.
Watch values go by
When a pipeline gives the wrong answer, look at what each stage is given. peek, from Ekorn::Pipeline::Stages::Taps, calls its block with each value, and passes the value on, whatever the block returns, so you can put it anywhere without changing what happens:
my @log;
my @titles = tracks(
peek { push @log, $_->{artist} },
sub { $_->{title} },
take(2),
);
say "@log"; # Abba Abba
take(2) stopped the run after two tracks, so peek only saw two.
Count values at any point
count_into counts the values going by into a variable, setting it to 0 at the start of each run:
my($read, $kept);
tracks(
count_into(\$read),
at_least(6000),
count_into(\$kept),
);
say "$kept of $read"; # 3 of 6
This uses at_least, from Build a vocabulary of stages.
Dump values while debugging
dump_to, imported on request, writes each value with Data::Dumper, keys sorted, to a filehandle, or to STDERR if you give it none:
open my $out, '>', \my $dumped or die "Can't write the dump: $!\n";
tracks(
take(1),
dump_to($out),
);
print $dumped;
# {
# 'artist' => 'Abba',
# 'plays' => 9100,
# 'title' => 'Dancing Queen'
# }
If Text::Redact::Secrets is installed, dump_to redacts the secrets it finds in what it writes, such as a password, a token or the password in a DSN, leaving the value passed on as it was:
my $dsn = join '', 'postgres://app:', 'Zq8vR2mX', 'kP4wL9nT', '@db/app';
open my $log, '>', \my $logged or die "Can't write the dump: $!\n";
my ($kept) = Ekorn::Pipeline->new(dump_to($log))->({dsn => $dsn});
print $logged;
# {
# 'dsn' => 'postgres://app:*REDACTED*@db/app'
# }
say $kept->{dsn} eq $dsn ? 'passed on as it came' : 'changed';
# passed on as it came
When you need to see a secret, while debugging, turn redaction off for as long as you need it, with local:
my $dsn = join '', 'postgres://app:', 'Zq8vR2mX', 'kP4wL9nT', '@db/app';
open my $debug, '>', \my $seen or die "Can't write the dump: $!\n";
{
local $Ekorn::Pipeline::Stages::Taps::DumpSecrets = 1;
Ekorn::Pipeline->new(dump_to($debug))->({dsn => $dsn});
}
say $seen =~ m( \Q$dsn\E )x ? 'written whole' : 'redacted';
# written whole
If the module is installed but broken, dump_to croaks rather than write a dump that may hold secrets, and says so. Redaction finds the secrets its rules know, not every secret, so dump_to is still for debugging only: take it out before the code goes live.
12. Questions you may have
How fast is it?
Slower than plain Perl, and how much depends on how much each stage does, and on how the pipeline is run. Through two trivial stages, a filter and a multiplication, on a recent laptop:
pushed, a pipeline runs about 2.3 million values a second, about twelve times slower than the same work with
mapandgrep;pulled through an iterator, about a million a second, about twenty-five times slower, as each value handed out is a call of its own.
A push takes a fast path, a plain loop with none of the machinery for flushes, ends and sinks, whenever its run has none of them: through plain subs, and through stages made for the run with gather that have no flush or end, such as take, drop and count_into. A run with a flush, an end or a sink, such as batch, take_last or keep_all, takes the full path, which costs about twice as much per value, and some 20 microseconds to set up each run.
That sounds a lot, and for a tight loop over numbers in memory, it is: use map and grep there. But each value costs well under a microsecond, and most pipelines spend far longer than that on the value itself: reading a line, fetching a row, calling a service. Against a database row, the pipeline's share is lost in the noise. And a pipeline that stops early does less work than code that can't: take(10) over a million rows reads ten.
Runs without flushes, ends or sinks, pushed or pulled, have been made faster so far. Runs with them haven't yet, so their figures describe the code as it is, not a limit of the design.
How much memory does it use?
Very little of its own. A value goes through every stage before the next starts, so nothing waits between stages, and a pulled pipeline over a million values holds about one at a time. Only stages that hold values on purpose, such as batch, collect and sorting sinks, hold more, and they hold what they say.
Does it read ahead?
No. Pushed, the values are a list you already made. Pulled, the source is asked for values only when a result is wanted, and only once the values from its last call have all gone through.
Does it run stages in parallel, or asynchronously?
No. A pipeline runs in the calling thread, one value at a time, from start to end, and a stage that blocks blocks the run. That keeps it simple and predictable, with no locks and no surprises in the order of results, but it isn't a tool for concurrency.
Does it need a database?
No. It began as the pipeline in DBIx::Squirrel, a database module (and ekorn is Norwegian for squirrel), but it doesn't use the DBI, or anything beyond a few small, common modules: Carp, Data::Dumper, Exporter, Ref::Util, Scalar::Util, experimental and namespace::clean. It works with values from anywhere.
Can a stage be a method?
Not directly, as a method needs its object, but a stage can call one: sub { $formatter->format($_) }. A class whose objects should work as stages, sources or sinks themselves can offer the methods in Make your objects sources and sinks.
Is it safe to keep a pipeline and use it again?
Yes. A pipeline is made once and run as often as you like, from as many places as you like, even from inside one of its own stages. Each run has its own state, as long as stages that keep state are made with gather, and sources and sinks are made for each run, as Ekorn's are.
Which Perl does it need, and will it change?
Perl 5.36 or later. Its version is 0.x, so its interface may still change before 1.0, but the version number says when: a release that only fixes or adds moves the last number, and one that changes how stages behave, or renames something, moves the middle one. Require >= v0.3.0, < v0.4.0, and you get fixes without surprises.
Where does it shine, and where does it fall short?
It shines:
in functions that hand out values, whose callers can then say what they want, beside the call, without the function growing an option for each;
when the work should stop early, as a stop ends the source's work too, and closes its file;
with values made one at a time, read from files, databases or services, where it holds almost nothing in memory, and every part is told when the run ends, however it ends;
in reuse, as a stage written once works with every function that takes stages, whoever wrote it.
It falls short:
on speed, for tight loops over values in memory, where
mapandgrepare much faster;on concurrency, which it doesn't do at all;
on discoverability, as a caller has to learn what a stage is, where an option would be listed in the documentation;
in a few sharp corners, most of them Perl's: a test isn't a filter (give it to
take_if), a plain closure keeps its state between runs,sub {...} | sub {...}is a bitwise OR,@{ $source }reads everything, and a generator runs its block to the end.
Is the author a Blur fan?
No. They're in the examples because only they have three tracks.
Where next?
The reference describes every module, function and method in full, with every message in Diagnostics. Each module's own documentation says the same more briefly, and is the authority where they differ.
SEE ALSO
Ekorn::Pipeline, Ekorn::Pipeline::Gather, Ekorn::Pipeline::Stop, Ekorn::Pipeline::Pipes
AUTHOR
Iain Campbell <cpanic@cpan.org>
LICENSE
Copyright (c) 2024-2026 by Iain Campbell.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.