/ 
Scalar-ValueTags-LineageTracking-0.004
/ Scalar::ValueTags::LineageTracking

NAME

Scalar::ValueTags::LineageTracking - track data lineages using ValueTags

SYNOPIS

# setup: choose type of data to store in tags
my $lt = Scalar::ValueTags::LineageTracking->new( tag_type => 'string' );
my $lt = Scalar::ValueTags::LineageTracking->new( tag_type => 'ref' );

# default tag_type: string
my $lt = Scalar::ValueTags::LineageTracking->new();

### append tracking data sources when data is received from external sources
my $names = $dbh->selectcol_arrayref('select name from user where user_id = %s", undef, $id);
my $name = $names->[0];

# using 'ref' tag_type
$lt->add_data_sources(\$name,
    [
        { table => 'user', column => 'name', id => $id, file => __FILE__, line => __LINE__ },
        ...,
    ]
);

# using 'string' tag_type
my $json = JSON->new->utf8->canonical->pretty(0);
$lt->add_data_sources( \$name,
    [
        $json->encode( { table => 'user', column => 'name', id => $id, file => __FILE__, line => __LINE__ } ),
        ...,
    ],
);

### clear previous data sources and set data source
# 'ref' tag_type
$lt->set_data_sources( \$name,
    [
        { table => 'user', column => 'name' },
        ...,
    ]
);
# 'string' tag_type
$lt->set_data_sources( \$name,
    [
        $json->encode( { table => 'user', column => 'name' } ),
        ...,
    ]
);

### data sources are propagated any time data is used
$lt->add_data_source(\$name, 'tag-one');
$lt->add_data_source(\$honorif, 'tag-two');
my $salutation = "Hello, $honorif $name";

# returns data sources set on both $honorif and $name:
my $sources = $lt->get_data_sources(\$salutation);

### retrieve and clear data sources, for reporting
my $sources = $lt->get_and_clear_data_sources(\$salutation);
send_lineage( { sources => $sources, var => 'salutation', file => __FILE__, line => __LINE__ } );

DESCRIPTION

This module uses Scalar::ValueTags (propagated value magic) to implement data flow tracking that can be used to populate data lineage systems such as OpenLineage.

LineageTracking allows arbitrary tracking data to be attached to data values when the values are received from an external system, propagates the tracking data any time that other data is derived from the original data, and captures the tracking data from the value when it is sent to an external sink.

The tracking data may be either an unblessed Perl reference or a serialized data string, depending on the tag_type parameter.

When the tracking items are propagated, they are de-duplicated using either the string or the refaddr of the data structure that was set in set_data_sources.

The tracking data may be formatted in any way, such as OpenLineage.

Implementation

For the "hash" tag_type, the SVTAGS_UNIQUE_HASH behavior is used, allowing string-serialized data to be used as tags. The tags are de-duplicated by the string value, so the serialization must generate canonical strings.

For the "ref" tag_type, the SVTAGS_UNIQUE_REF_ARRAY behavior is used, allowing any arbitray Perl reference to be used as tags. The tags are de-duplicated by the refaddr of the reference rather than the contents, so identical tags must use the same Perl reference.

PARAMETERS

tag_type

The tag_type determines the type of the tags passed to Scalar::ValueTags.

  • "string"

    Each value tag must be a string suitable for use as a hash key. De-duplication is done automatically since the string is used as a hash key.

  • "ref"

    Each value tag must be a Perl reference. De-duplication is done by the refaddr of the reference.

METHODS

add_data_sources

# using 'string' tag type
$lt->add_data_sources( \$var, [ $string1, $string2, ... ] );

# using 'ref' tag type
$lt->add_data_sources( \$var, [ { this => 1 }, { that => 2 } ] );

Append the given data sources to the value of the given $var. Each data source must match the tag_type set upon instantiation.

Does not return anything.

clear_data_sources

my $data_sources = $lt->clear_data_sources(\$var);
for my $data_source (@$data_sources) { ... }

Clears all data sources on the value in given $var.

Returns an arrayref of all of the data sources that previously existed on the value in $var.

get_data_sources

my $data_sources = $lt->get_data_sources(\$var);
for my $data_source (@$data_sources) { ... }

Returns an arrayref of all data sources set for the value in the given $var. Includes all propagated tags from all data values that were used to calculate the value of $var.

set_data_sources

# using 'string' tag type
my $prev_sources = $lt->set_data_sources( \$var, [ $string1, $string2, ... ] );

# using 'ref' tag type
my $prev_sources = $lt->set_data_sources( \$var, [ { this => 1 }, { that => 2 } ] );

Set the data sources for the value of the given $var to be the given data sources, clearing any pervious data sources. Each data source must match the tag_type set upon instantiation.

Returns an arrayref of the previous data sources that were set on the value in $var.

LICENSE

This module is released under the same terms as Perl itself.

AUTHORS

  • Noel Maddy <zhtwnpanta@gmail.com>