=head1 NAME
PDL::Dataflow -- description of the dataflow philosophy
=head1 SYNOPSIS
pdl>
$x
= zeroes(10);
pdl>
$y
=
$x
->slice(
"2:4:2"
);
pdl>
$y
++;
pdl>
print
$x
;
[0 0 1 0 1 0 0 0 0 0]
=head1 WARNING
Dataflow is very experimental. Many features of it are disabled
for
2.0, particularly families
for
one-directional
dataflow. If you wish to
use
one-directional dataflow
for
something, please contact the author first and we'll work out
how to make it functional again.
Two-directional dataflow (which implements ->slice() etc.)
is fully functional, however. Just about any function which
returns some subset of the
values
in some ndarray will make a binding
so that
$x
= some ndarray
$y
=
$x
->slice(
"some parts"
);
$y
->set(3,3,10);
also changes the corresponding element in
$x
.
$y
has
become effectively
a window to some
sub
-elements of
$x
. You can also define your own routines
that
do
different types of subsets. If you don't want
$y
to be a window
to
$x
, you must
do
$y
=
$x
->slice(
"some parts"
)->copy;
The copying turns off all dataflow between the two ndarrays.
The difficulties
with
one-directional
dataflow are related to sequences like
$y
=
$x
+ 1;
$y
++;
where there are several possible outcomes and the semantics get a little
murky.
=head1 DESCRIPTION
Dataflow is new to PDL2.0. The basic philosophy
behind dataflow is that
>
$x
= pdl 2,3,4;
>
$y
=
$x
* 2;
>
print
$y
[2 3 4]
>
$x
->set(0,5);
>
print
$y
;
[10 3 4]
should work. It doesn't. It was considered that doing this
might be too confusing
for
novices and occasional users of the language.
Therefore, you need to explicitly turn on dataflow, so
>
$x
= pdl 2,3,4;
>
$x
->doflow();
>
$y
=
$x
* 2;
...
produces the unexpected result. The rest of this documents
explains various features and details of the dataflow implementation.
=head1 Lazy evaluation
When you calculate something like the above
>
$x
= pdl 2,3,4;
>
$x
->doflow();
>
$y
=
$x
* 2;
nothing will have been calculated at this point. Even the memory
for
the contents of
$y
has
not been allocated. Only the command
>
print
$y
will actually cause
$y
to be calculated. This is important to bear
in mind
when
doing performance measurements and benchmarks as well
as
when
tracking errors.
There is an explanation
for
this behaviour: it may save cycles
but more importantly, imagine the following:
>
$x
= pdl 2,3,4;
>
$y
= pdl 5,6,7;
>
$c
=
$x
+
$y
;
...
>
$x
->resize(4);
>
$y
->resize(4);
>
print
$c
;
Now,
if
$c
were evaluated between the two resizes, an error condition
of incompatible sizes would occur.
What happens in the current version is that resizing
$x
raises
a flag in
$c
:
"PDL_PARENTDIMSCHANGED"
and
$y
just raises the same flag
again. When
$c
is
next
evaluated, the flags are checked and it is found
that a recalculation is needed.
Of course, lazy evaluation can sometimes make debugging more painful
because errors may occur somewhere where you'd not expect them.
A better stack trace
for
errors is in the works
for
PDL, probably
so that you can toggle a switch
$PDL::traceevals
and get a good trace
of where the error actually was.
=head1 Families
This is one of the more intricate concepts of one-directional dataflow.
Consider the following code (
$x
and
$y
are pdls that have dataflow enabled):
$w
=
$u
+
$v
;
$y
=
$w
+ 1;
$x
=
$w
->diagonal();
$x
++;
$z
=
$w
+ 1;
What should
$y
and
$z
contain now? What about
when
$u
is changed
and a recalculation is triggered.
In order to make dataflow work like you'd expect, a rather strange
concept must be introduced: families. Let us make a diagram:
u v
\ /
w
/|
/ |
y x
This is what PDL actually
has
in memory
after
the first three lines.
When
$x
is changed, we want
$w
to change but we don't want
$y
to change
because it already is on the graph. It may not be clear now why you don't
want it to change but
if
there were 40 lines of code between the 2nd
and 4th lines, you would. So we need to make a copy of
$w
and
$x
:
u v
\ /
w' . . . w
/| |\
/ | | \
y x' . . . x z
Notice that we primed the original w and x, because they
do
not correspond
to the objects in
$w
and
$x
any more. Also, notice the dotted lines
between the two objects:
when
$u
is changed and this diagram is re-evaluated,
$w
really does get the value of w'
with
the diagonal incremented.
To generalize on the above, whenever an ndarray is mutated i.e.
when
its actual
*value
* is forcibly changed (not just the reference):
$x
=
$x
+ 1
would produce a completely different result (
$w
and
$x
would not be bound
any more whereas
$x
.=
$x
+ 1
would yield the same as
$x
++), a
"family"
consisting of all other ndarrays
joined to the mutated ndarray by a two-way transformation is created
and all those are copied.
All slices or transformations that simply
select
a subset of the original
pdl are two-way. Matrix inverse should be. No arithmetic
operators are.
=head1 Sources
What you were told in the previous section is not quite true:
the behaviour described is not
*always
* what you want. Sometimes you
would probably like to have a data
"source"
:
$x
= pdl 2,3,4;
$y
= pdl 5,6,7;
$c
=
$x
+
$y
;
line(
$c
);
Now,
if
you know that
$x
is going to change and that you want
its children to change
with
it, you can declare it into a data source
(XXX unimplemented in current version):
$x
->datasource(1);
After this,
$x
++ or
$x
.= something will not create a new family
but will alter
$x
and cut its relation
with
its previous parents.
All its children will follow its current value.
So
if
$c
in the previous section had been declared as a source,
$e
and
$f
would remain equal.
=head1 Binding
A dataflow mechanism would not be very useful without the ability
to
bind
events onto changed data. Therefore, we provide such a mechanism:
>
$x
= pdl 2,3,4
>
$y
=
$x
+ 1;
>
$c
=
$y
* 2;
>
$c
->
bind
(
sub
{
print
"A now: $x, C now: $c\n"
} )
> PDL::dowhenidle();
A now: [2,3,4], C now: [6 8 10]
>
$x
->set(0,1);
>
$x
->set(1,1);
> PDL::dowhenidle();
A now: [1,1,4], C now: [4 4 10]
Notice how the callbacks only get called during PDL::dowhenidle.
An easy way to interface this to Perl event loop mechanisms
(such as Tk) is being planned.
There are many kinds of uses
for
this feature: self-updating graphs,
for
instance.
Blah blah blah XXX more explanation
=head1 Limitations
Dataflow as such is a fairly limited addition on top of Perl.
To get a more refined addition, the internals of Perl need to be
hacked a little. A true implementation would enable flow of everything,
including
=over 12
=item data
=item data size
=item datatype
=item operations
=back
At the moment we only have the first two (hey, 50% in a couple of months
is not bad ;) but even this is useful by itself. However, especially
the
last
one is desirable since it would add the possibility
of flowing closures from place to place and would make many things
more flexible.
To get the rest working, the internals of dataflow probably need to
be changed to be a more general framework.
Additionally, it would be nice to be able to flow data in
time
,
lucid-like (so you could easily define all kinds of signal processing
things).
=head1 AUTHOR
Copyright(C) 1997 Tuomas J. Lukka (lukka
@fas
.harvard.edu).
Redistribution in the same form is allowed provided that the copyright
notice stays intact but reprinting requires
a permission from the author.