Revision history for Ekorn-Pipeline
Unreleased
Pipelines can be made with pipeline(), and named, and a function
whose values can be code references can take its stages as a
pipeline given last. pipeline() is one of the words in
Ekorn::Pipeline::Factories, short functions that make Ekorn's
types: stage, gather, source, generator with yield, sink_one and
sink_many. The README's CPAN badge keeps up with releases, and the
how-to says where a stage belongs. Stages::Flow has take_first,
take_last and take_single, and take_if and drop_if, which filter.
Ekorn::Pipeline::Tooling exports every stage, source, sink and
word from one module, by tags. Ekorn::Pipeline::JSON reads and
writes JSON and JSON Lines, as sources, stages or sinks, and
Ekorn::Pipeline::Sugar names pipelines as the program compiles,
each with a function of its name.
A run's parts can be made afresh for it, and told how it ended:
sources that close what they open, however the run ends; sinks
that consume every value and give a result, committing only when
the run is done; and gathers with an end as well as a flush, what
v0.2.0 called the end function; Sources and Sinks have them
ready-made, and Operators joins them with |. A part's set-up, or
a package's role method, can return its parts by name, as a hash
reference, rather than as a list in a set order.
An iterator is an object, which
can read what's left in one call, end its run early, and say how
it ended, and which ends its run when it's let go.
Pushing values through plain subs is several times faster, and
reading an iterator's values in one call a tenth faster.
It ships eighteen roadmap items, 2026-0009, 2026-0010, 2026-0011,
2026-0013, 2026-0014, 2026-0016, 2026-0017, 2026-0018, 2026-0021,
2026-0024, 2026-0026, 2026-0027, 2026-0028, 2026-0029, 2026-0030,
2026-0031, 2026-0032 and 2026-0034. It changes what some code
sees, as the entries marked "Changed" say: a gather, an iterator,
every canned stage, and the stages a pipeline gives back are
objects, called as code references, not code references
themselves; a gather's message names the flush and the end; a
stage's error names the named pipeline it's in; the messages for
what isn't a stage or a source say what one is; and a pipeline
named alone, pipeline('clean'), is the pipeline itself, so its
stages are numbered where it's given. Otherwise,
existing code runs as before.
- Ekorn::Pipeline::Sugar names pipelines as the program compiles:
use Ekorn::Pipeline::Sugar pipeline => {name => [@stages]}, or an
array of pairs, names each in the package that uses it, and gives
that package a function of each name, which runs values through
the pipeline, and, given a pipeline last, through that too. A
definition can name any other in the same use, in either order.
Mistakes, such as a name taken already, or a circle of names,
croak as the use compiles. import_into does the same for a
package given, for other modules' sugar. Sub::Util, which comes
with Perl, is a new dependency. (roadmap 2026-0010)
- Ekorn::Pipeline::JSON makes from_json and to_json, for one JSON
document, and from_jsonl and to_jsonl, for JSON Lines: each a
source or a sink given a file or a filehandle, and a stage given
neither. from_json gives an array's elements, or the document
itself; from_jsonl gives each line's document, read a block at a
time, as values are pulled. to_json writes every value as one
array, and to_jsonl a value a line. As stages, each passes on
what a file would hold, so to_json and from_json undo each
other, as to_jsonl and from_jsonl do. The encoder is
JSON::MaybeXS's, configured by a json option, or replaced by an
object of the caller's. Tooling exports all four, by a :json
tag. JSON::MaybeXS is a new dependency. The user guide reads and
writes JSON in chapters 5 and 6, and walks through how a module
like it is put together in chapter 10. (roadmap 2026-0034)
- Ekorn::Pipeline::Tooling exports what callers use to do work,
from one module, so there's nothing to remember about which
module makes which: tags :core (Stop and gather), :flow, :batch
and :taps, :stages for the three, :sources, :sinks, :words (the
factories) and :all. Each tag holds every name its module
exports, as the same sub, and nothing is exported by default.
It leaves out is_stage, is_sink and croakf, which are for
extending the package, and the operators, which stay a pragma.
The user guide's examples, the README's and the reference's
At a glance import from it. (roadmap 2026-0031)
- The reference has a section, Making each kind of part, for
developers: a table of what each piece of a part, its set-up,
step, flush, end and context, is called with, returns, and
whether it may stop the run, and a worked example of each kind
of part, made with a word and as a class, from a stage with
state to an object that's both a source and a sink, committing
only a run that went through. Every example in the README, the
user guide, the reference and the modules' POD writes a
pipeline of two or more stages a stage to a line, and
paragraphs are filled evenly again.
- The role methods, by which any object can be a source, a stage
or a sink, are named in capitals, EKORN_SOURCE, EKORN_STAGE,
EKORN_SINK_ONE and EKORN_SINK_MANY, as Perl names subs that its
runtime calls, not their caller, such as DESTROY, and as CPAN
names hooks for another package, such as TO_JSON. No release had
them in lower case (roadmap 2026-0027).
- A set-up, a gather's, a source's or a sink's, and a role method,
EKORN_SOURCE, EKORN_STAGE, EKORN_SINK_ONE or EKORN_SINK_MANY,
can return its parts by name, as a hash reference with the keys
step, flush, end and context, giving only those it has, in place
of a list in a set order with undef for one left out before one
given: gather { {step => $step, end => $end} }. Other keys are
ignored, and a hash without what its part needs croaks, as a
list does. The list still works. Ekorn's own sources, sinks and
stages, and an iterator's EKORN_SOURCE, return theirs by name
(roadmap 2026-0026).
- Ekorn::Pipeline::Sources has ready-made sources, needing only
Perl, none exported by default: from_array(@values); from_fh($fh),
a filehandle's records, as $/ divides them, chomped, left open;
from_file($path, $layers), a file's records, opened through the
I/O layers given, if any, for each run, and closed however it
ends; and counter($start, $step), without end. With $/
undefined, from_fh and from_file slurp, an empty file giving
none. (roadmap 2026-0016)
- Ekorn::Pipeline::Sinks has ready-made sinks, needing only Perl,
none exported by default: keep_all, every item; keep($n), the
first $n, stopping there; keep_first, stopping at it;
keep_single, warning at a second; keep_last; count_items; and
to_fh($fh) and to_file($path, $layers), writing each item
followed by $/, as from_fh and from_file take it off, and giving
how many. to_file opens its file for each run, through the I/O
layers given, if any, and closes it however the run ends.
(roadmap 2026-0017)
- Ekorn::Pipeline::Operators, used in a scope, puts |, |= and @{}
in force there for Ekorn's types: $source | $stage | $sink gives
the sink's result, $stage | $stage a chain of stages, $source |
$stage a source, and @{ $source } every value, read through. A
chain starts with a source or a stage made with a word, never a
plain code reference, whose | is Perl's bitwise OR. Out of
scope, they croak "Expected use Ekorn::Pipeline::Operators in
scope, for %s". A package's own source class chains by
overloading | and @{} with Operators' chain and drain. (roadmap
2026-0018)
- Changed: a stage that isn't one croaks "Expected stage #%d to be
a stage: a code reference, a pipeline, or an object offering a
stage's role method", where it said "Expected stage #%d to be a
code reference or a pipeline", and, given to pipeline or named,
"... a pipeline or its name, or an object offering a stage's role
method"; both left out the parts made for a run. A pipeline whose
overridden stages method returns something that isn't a stage has
a message of its own, "Expected stage #%d, a pipeline, to give
only stages from its stages method", in place of one saying it
should be a pipeline. Stage->new, given one of Ekorn's types,
croaks "Expected a code reference, not one of Ekorn's types".
- Changed: iterator, given something other than a source, croaks
"Expected a source: a code reference, or an object offering
EKORN_SOURCE", where it said "Expected a code reference", which
was accurate only until sources could be objects.
- Changed: a stage that dies within a named pipeline is named with
it: after its place, the innermost named pipeline and the
stage's place there, "Pipeline stage #1.2 (main::clean#2)
failed at ...", and an end, "Pipeline end of stage #1.2
(main::clean#2) failed at ...". A stage of no named pipeline is
named as before. A pipeline's name method gives its first name,
qualified, or undef. (roadmap 2026-0013)
- Ekorn::Pipeline::Stages::Flow exports two more stages on request:
take_if { ... }, passing on each value the block is true of, and
drop_if { ... }, each it's false of, testing every value, as grep
does, where take_while and drop_while stop testing at the first
value that fails. A stage ending in a test passes on true or
false; take_if passes on the value. (roadmap 2026-0030)
- Faster: a pipeline whose stages are all plain subs, called with
values, runs them in a plain loop, with none of the machinery
for flushes, ends and sinks: a call of a few values is about
twelve times faster, and a call of many through one stage three
and a half times. Every other run is a little faster too: a push
hands its values over at once, and less is done for each value.
Nothing else changes. (roadmap 2026-0028)
- Faster: pulled runs, as DBIx-Squirrel measured them. An
iterator's all collects every value in one pass of the run,
rather than a call for each, about a tenth faster through one
plain stage, and each pull copies less. Nothing else changes.
(roadmap 2026-0032)
- Changed: pipeline_in, and so the word pipeline, given one name
and nothing else, returns the pipeline named itself, not a new
pipeline holding it, if it's of the invocant's class or a
subclass. So a stage of a named pipeline given second is
"Pipeline stage #2.2 (main::clean#2)", not "#2.1.2". named on a
class still makes a new pipeline, even of one name. (roadmap
2026-0029)
- The message for a gather's block that returns the wrong things
now ends "and optionally a flush and an end", where it ended
"and optionally an end function": what v0.2.0 called the end
function is now the flush, and an end, told how the run ended,
may follow it. Code matching the old wording must change.
(roadmap 2026-0014)
- A run's parts can be made for it. Ekorn::Pipeline::Source makes
a source from a set-up, called at the start of each run, which
returns its step and, optionally, its end and a context;
iterator takes one, or anything offering EKORN_SOURCE. A source
given to new as a stage croaks. (roadmap 2026-0014)
- Ekorn::Pipeline::Sink makes sinks, with new_sink_one, giving one
value, and new_sink_many, giving a list: a set-up returns the
step, the flush, giving the result, and optionally an end. A
pipeline ending in a sink gives its result: in scalar context, a
sink of one's value, or a sink of many's list as an array
reference. is_sink, exported on request, says "one", "many" or
false. (roadmap 2026-0014)
- Each end, a source's, a gather's or a sink's, is called once,
when the run has ended, however it ended, told "done", "stopped"
or "died", the source's first. A run that dies calls every end,
told "died", before throwing its error; an end that dies is
reported as "Pipeline end of %s failed at %s line %d:", or, in a
run already dying, warns "The end of %s died while the run was
already dying: %s". A gather's block may return an end, third,
after its flush. (roadmap 2026-0014)
- new takes, as a stage, any object offering EKORN_STAGE,
EKORN_SINK_ONE or EKORN_SINK_MANY, called at the start of each
run, so a package can deliver parts for Ekorn without using its
classes; one offering more than one croaks. is_stage is true for
them. run_context gives a stage the context its run's source
gave. (roadmap 2026-0014)
- Changed: iterator returns an Ekorn::Pipeline::Iterator, an object
called as a code reference, as before, with all, first, single,
finish and ended. Let go before its values run out, it ends its
run, so a source closes what it opened. An iterator can be a
source for another pipeline, once. (roadmap 2026-0014)
- Changed: a gather is an Ekorn::Pipeline::Gather object, called
as a code reference, not a blessed code reference, and stages
returns a pipeline's stages as objects: a code reference given
to new comes back as an Ekorn::Pipeline::Stage of it. Code
testing them with is_coderef or ref sees the change. (roadmap
2026-0014)
- New messages, for sources, sinks, parts and ends, are listed in
the POD's DIAGNOSTICS and the reference's Diagnostics. (roadmap
2026-0014)
- The how-to, Ekorn::Pipeline::Manual::HowTo, is rewritten as a
user guide. It tours the package from the bottom up, a layer at
a time: stages, pipelines pushed and pulled, stopping, parts made
for each run, sources, sinks, composition, the operators,
functions that take stages, and parts for other packages; then
errors and debugging, and the questions people ask first, such
as how fast it is, how it compares with map, grep, options and
callbacks, and where it falls short. Each of its examples runs,
and prints what it says.
- The README is rewritten for this release. It shows the shape of
a run, from a source through stages to a sink, with sections on
parts made for each run, sources, sinks, joining them with |,
named pipelines, and parts for other packages, and lists the
modules that come with it.
- The manuals' tables show a | written in them as |, not \|.
- The reference has a section, Runs and how they end, and another,
Delivering parts for Ekorn; the how-to has recipes for sources
and sinks that close and commit however a run ends, and for
reading an iterator's values in one call. (roadmap 2026-0014)
- pipeline(), exported on request by Ekorn::Pipeline::Factories,
makes a new pipeline from stages, as new does, but a stage can
also be the name of a pipeline: pipeline('clean', take(5)). It's
Ekorn::Pipeline->pipeline_in(scalar caller, @stages), the one
method that reads names, making its invocant's class. (roadmap
2026-0011, 2026-0021)
- Ekorn::Pipeline::Factories has the words that make Ekorn's
types, each one line over a class method, none exported by
default: pipeline, stage, gather (Gather's own), source,
generator and yield, sink_one and sink_many. generator gives a
source of what its block passes to yield, as a Python generator
does, collecting it all in one step; yield outside one croaks
"Expected yield to be called inside a generator". (roadmap
2026-0021)
- Ekorn::Pipeline->named(clean => @stages) defines a named
pipeline, and $p->named('clean') names one you have. A name is
kept in the caller's package unless it's qualified, whatever
class defined it, is resolved when a pipeline is made, and can't
be defined twice, or removed. Only pipeline() and named() read names: new
doesn't, and is_stage stays false for a string. (roadmap
2026-0011)
- Five new messages, for pipeline() and named(): "Expected stage
#%d to be a code reference, a pipeline or the name of one", "No
pipeline named %s", "A pipeline named %s is already defined",
"Expected a name, made of words joined by ::" and "Expected only
a name, naming a pipeline you have". (roadmap 2026-0011)
- The reference, the how-to and the README show a pipeline given
last as the way to take stages after values that can be code
references, and how to name a pipeline you use often. (roadmap
2026-0011)
- The reference has a section on the words, and its examples, and
the how-to's, make sources and sinks with them. (roadmap
2026-0021)
- The README's CPAN badge comes from shields.io, which keeps up
with a release within minutes, in place of badge.fury.io's, which
GitHub showed out of date for hours. (roadmap 2026-0009)
- The how-to has a recipe, "Decide where a stage belongs": under
Ekorn::Pipeline::Stages, or with the module whose values it
needs to know about. Its cpanfile example asks for v0.2.0.
- Ekorn::Pipeline::Stages::Flow exports three more stages on
request: take_first, which is take(1); take_last, passing on the
last value once there are no more; and take_single, passing on
the one value, and, at a second, warning "Expected only one item,
but there were more" and stopping. Each, with take($n), is the
stage of a keep sink, passing on what the sink would give.
(roadmap 2026-0024)
- Changed: every canned stage, in Stages::Flow, Batch and Taps, is
an object called as a code reference: a Gather, or, for
take_while, peek and dump_to, an Ekorn::Pipeline::Stage. Code
testing one with ref or is_coderef sees the change. (roadmap
2026-0024)
v0.2.0 2026-11-09
Messages say a stage can be a code reference or a pipeline.
It ships one roadmap item, 2026-0008. Two messages change, so it's
v0.2.0: code that matches the old wording, as a string or an
anchored pattern, needs updating. Nothing else changes in
behaviour. DBIx::Squirrel signed this release off before it went
out.
- "Expected stage #%d to be a code reference" is now "Expected
stage #%d to be a code reference or a pipeline". (roadmap
2026-0008)
- "Expected stage #%s, made with gather, to return one or two code
references" is now "Expected stage #%s, made with gather, to
return a code reference or a pipeline, and optionally an end
function". (roadmap 2026-0008)
- The DIAGNOSTICS, the reference, the how-to and the README say the
same, and no longer explain the old wording. (roadmap 2026-0008)
v0.1.3 2026-11-09
A pipeline can be used as a stage, and the docs are plainer.
Pipelines can be built from other pipelines without ->stages, and a
gather block can return one. The documentation is rewritten in
plainer English.
It ships three roadmap items, 2026-0004, 2026-0006 and 2026-0007.
Existing code runs as before, and every message a pipeline gives
today reads as before. The one exception is a subclass of
Ekorn::Pipeline used with an import list, which now croaks, as
below. DBIx::Squirrel signed this release off before it went out.
- Ekorn::Pipeline->new accepts a pipeline as a stage, and splices
its stages in where it stands. They run in the new pipeline's
run, so a take inside limits the whole run. The stages come from
the pipeline's stages method, so a subclass can override it.
(roadmap 2026-0004)
- In error messages, a stage inside a pipeline used as a stage is
named by its position in each pipeline: "Pipeline stage #2.3
failed", or "#1.2.3" three levels down. DIAGNOSTICS shows "#%s"
for these messages. Other stages are named as before.
(roadmap 2026-0004)
- is_stage, exported by Ekorn::Pipeline on request, is true for
anything new accepts as a stage: a code reference or a pipeline.
Use it in a function that takes its stages from the end of its
arguments, so it accepts pipelines too. Ask for v0.1.3 on the use
line, as v0.1.2 lets "use Ekorn::Pipeline qw(is_stage)" compile.
(roadmap 2026-0004)
- Ekorn::Pipeline now has an import method, from Exporter, for
is_stage. A subclass inherits it, so "use My::Subclass qw(name)"
croaks unless My::Subclass exports that name; before, the list
was ignored. (roadmap 2026-0004)
- A gather block can return a pipeline as its stage. Its stages are
made for each run in its place, gather stages inside it afresh,
and the block's end function, if any, is called after theirs. A
stage inside it is named as "#2.3" in messages. The message for a
bad gather block is unchanged. (roadmap 2026-0007)
- The README, the POD, the reference and the how-to are rewritten
in shorter sentences, saying the same. Two out-of-date claims in
the README are fixed: Ekorn::Pipeline is on CPAN, and
DBIx::Squirrel uses it. (roadmap 2026-0006)
- The README has MetaCPAN and Kwalitee badges.
v0.1.2 2026-11-08
The reference and the how-to on MetaCPAN, as POD manuals.
It adds documentation only: the code is as in v0.1.1, and it ships
no roadmap items.
- The reference and the how-to ship as POD,
Ekorn::Pipeline::Manual::Reference and
Ekorn::Pipeline::Manual::HowTo, made from their Markdown, so
MetaCPAN and perldoc have them with the modules.
v0.1.1 2026-11-08
The pipeline from DBIx-Squirrel v2.0.2, on its own, so that any
function can take stages from its caller.
It ships three roadmap items, settled after v0.1.0's trial: a
pipeline's stages can be spliced into another, with the stages
method (2026-0001); an iterator ends once a stage dies (2026-0002);
and a pipeline's count in scalar context is promised (2026-0003).
- Requires Perl 5.36 or later.
- Ekorn::Pipeline runs values through a list of stages, each a code
reference given one value, in $_ and as its argument, and
returning nothing (a filter), one value (a map) or several (a
flatMap). A pipeline is called with values, or pulls them from a
source with an iterator, calling it only as results are wanted.
- Once a stage dies, an iterator has ended, as a pushed run has:
every call after returns an empty list, and neither the source
nor any end function is called again. In v0.1.0's trial it
carried on, without the value that died. (roadmap 2026-0002)
- Called in scalar context, a pipeline returns how many results
came out, as an array would; the POD now promises it.
(roadmap 2026-0003)
- A pipeline's stages method returns its stages, so they can be
spliced into another pipeline, or given to any function that
takes stages, running as if listed there. (roadmap 2026-0001)
- SECURITY.md says how to report a vulnerability, privately, and
CONTRIBUTING.md how to report a bug or ask for a feature; the
distribution takes issues, not pull requests.
- Ekorn::Pipeline::Stop ends a pipeline early, passing on any
values it carries, after which the source isn't called again. A
stop not made by throw, and so not an array, carries nothing.
- Ekorn::Pipeline::Gather makes a stage afresh for each run, with
an optional end function, called once the values run out.
- Ekorn::Pipeline::Pipes gives a class pipe and tap methods, which
run the object through stages.
- Canned stages: take, drop, take_while and drop_while in
Ekorn::Pipeline::Stages::Flow; batch and collect in
Ekorn::Pipeline::Stages::Batch; peek, count_into and dump_to in
Ekorn::Pipeline::Stages::Taps.
- A stage that dies is named by its place, at the caller's line, and
so are mistakes in making a pipeline or a canned stage.
- Modules of stages can live under any namespace: general stages
under Ekorn::Pipeline::Stages, and those for one kind of value or
source with the module that knows about it, built from Gather,
Stop and croakf, which reports at the caller's line.
- Versions: a v0.1.x only fixes or adds. A change to how stages
behave, or a public name dropped or renamed, waits for v0.2.0.
- A how-to, docs/HOWTO.md, with recipes for callers, function
authors and stage authors, and a reference, docs/REFERENCE.md,
describing every module and how a pipeline behaves, with notes
for stage authors. Both are on GitHub, linked from the README.
Keyboard Shortcuts
Global
s
Focus search bar
?
Bring up this help dialog
GitHub
gp
Go to pull requests
gi
Go to GitHub issues (only if GitHub is preferred repository)