/ 
MCE-1.699_006
⭐ Starred 45 / MCE::Shared

NAME

MCE::Shared - MCE extension for sharing data between workers

VERSION

This document describes MCE::Shared version 1.699_006

SYNOPSIS

# OO construction

use MCE::Shared Sereal => 1;

my $ar = MCE::Shared->array( @list );
my $cv = MCE::Shared->condvar( 0 );
my $fh = MCE::Shared->handle( '>>', \*STDOUT );
my $ha = MCE::Shared->hash( @pairs );
my $oh = MCE::Shared->ordhash( @pairs );
my $qu = MCE::Shared->queue( await => 1, fast => 0 );
my $va = MCE::Shared->scalar( $value );
my $nu = MCE::Shared->sequence( $begin, $end, $step, $fmt );
my $ob = MCE::Shared->share( $blessed_object );

# Tie construction

use MCE::Flow;
use MCE::Shared Sereal => 1;
use feature 'say';

tie my $var, 'MCE::Shared', 'initial value';
tie my @ary, 'MCE::Shared', qw(a list of values);
tie my %has, 'MCE::Shared', (key1 => 'value', key2 => 'value');

tie my $cnt, 'MCE::Shared', 0;
tie my @foo, 'MCE::Shared';
tie my %bar, 'MCE::Shared';

my $m1 = MCE::Mutex->new;

mce_flow {
   max_workers => 4
},
sub {
   my ($mce) = @_;
   my ($pid, $wid) = (MCE->pid, MCE->wid);

   ## Locking is required when multiple workers update the same element.
   ## This requires 2 trips to the manager process (fetch and store).

   $m1->synchronize( sub {
      $cnt += 1;
   });

   ## Locking is not necessary when updating unique elements.

   $foo[ $wid - 1 ] = $pid;
   $bar{ $pid }     = $wid;

   return;
};

say "scalar : $cnt";
say " array : $_" for (@foo);
say "  hash : $_ => $bar{$_}" for (sort keys %bar);

-- Output

scalar : 4
 array : 37847
 array : 37848
 array : 37849
 array : 37850
  hash : 37847 => 1
  hash : 37848 => 2
  hash : 37849 => 3
  hash : 37850 => 4

DESCRIPTION

This module provides data sharing for MCE supporting threads and processes.

MCE::Shared enables extra functionality on systems with IO::FDPass. Without it, MCE::Shared is unable to send file descriptors to the shared-manager process for <queue>, condvar, and possibly handle.

As of this writing, the IO::FDPass module is not a requirement for running MCE::Shared nor is the check made during installation. The reason is that IO::FDPass is not possible on Cygwin and not sure about AIX.

The following is a suggestion for systems without IO::FDPass. This restriction applies to queue, condvar, and handle only.

use MCE::Shared;

# Construct shared queue(s) and condvar(s) first.
# These contain GLOB handles - freezing not allowed.

my $q1  = MCE::Shared->queue();
my $q2  = MCE::Shared->queue();

my $cv1 = MCE::Shared->condvar();
my $cv2 = MCE::Shared->condvar();

# Start the shared-manager manually.

MCE::Shared->start();

# The shared-manager process knows of STDOUT, STDERR, STDIN

my $fh1 = MCE::Shared->handle(">>", \*STDOUT);  # ok
my $fh2 = MCE::Shared->handle("<", "/path/to/sequence.fasta");  # ok
my $h1  = MCE::Shared->hash();

Otherwise, sharing is immediate and not delayed with IO::FDPass. It is not necessary to share queue and condvar first or worry about starting the shared-manager process.

use MCE::Shared;

my $h1 = MCE::Shared->hash();    # shares immediately
my $q1 = MCE::Shared->queue();   # IO::FDPass sends file descriptors
my $cv = MCE::Shared->condvar(); # IO::FDPass sends file descriptors
my $h2 = MCE::Shared->ordhash();

DATA SHARING

array
condvar
handle
hash
ordhash
queue
scalar
sequence

array, condvar, handle, hash, ordhash, queue, scalar, and sequence are sugar syntax for constructing a shared object.

# long form

use MCE::Shared;
use MCE::Shared::Array;
use MCE::Shared::Hash;

my $ar = MCE::Shared->share( MCE::Shared::Array->new() );
my $ha = MCE::Shared->share( MCE::Shared::Hash->new() );

# short form

use MCE::Shared;

my $ar = MCE::Shared->array( @list );
my $cv = MCE::Shared->condvar( 0 );
my $fh = MCE::Shared->handle( '>>', \*STDOUT );
my $ha = MCE::Shared->hash( @pairs );
my $oh = MCE::Shared->ordhash( @pairs );
my $qu = MCE::Shared->queue( await => 1, fast => 0 );
my $va = MCE::Shared->scalar( $value );
my $nu = MCE::Shared->sequence( $begin, $end, $step, $fmt );
num_sequence

num_sequence is an alias for sequence.

OBJECT SHARING

share

This class method transfers the blessed-object to the shared-manager process and returns a MCE::Shared::Object containing the SHARED_ID. The object must not contain any GLOB's or CODE_REF's or the transfer will fail.

Unlike threads::shared, objects are not deeply shared. The shared object is accessable only through the underlying OO interface.

use MCE::Shared;
use Hash::Ordered;

my ($ho_shared, $ho_unshared);

$ho_shared = MCE::Shared->share( Hash::Ordered->new() );

$ho_shared->push( @pairs );            # OO interface only
$ho_shared->merge( @pairs );

$ho_unshared = $ho_shared->export();   # back to unshared
$ho_unshared = $ho_shared->destroy();  # including destruction

The following provide long and short forms for constructing a shared array, hash, or scalar object.

use MCE::Shared;

use MCE::Shared::Array;    # Loading helper classes is not necessary
use MCE::Shared::Hash;     # when using the shorter form.
use MCE::Shared::Scalar;

my $a1 = MCE::Shared->share( MCE::Shared::Array->new( @list ) );
my $a3 = MCE::Shared->share( [ @list ] );  # sugar syntax
my $a2 = MCE::Shared->array( @list );

my $h1 = MCE::Shared->share( MCE::Shared::Hash->new( @pairs ) );
my $h3 = MCE::Shared->share( { @pairs } ); # sugar syntax
my $h2 = MCE::Shared->hash( @pairs );

my $s1 = MCE::Shared->share( MCE::Shared::Scalar->new( 20 ) );
my $s2 = MCE::Shared->share( \do{ my $o = 20 } );
my $s4 = MCE::Shared->scalar( 20 );

PDL SHARING

pdl_byte
pdl_short
pdl_ushort
pdl_long
pdl_longlong
pdl_float
pdl_double
pdl_ones
pdl_sequence
pdl_zeroes
pdl_indx
pdl

pdl_byte, pdl_short, pdl_ushort, pdl_long, pdl_longlong, pdl_float, pdl_double, pdl_ones, pdl_sequence, pdl_zeroes, pdl_indx, and pdl are sugar syntax for PDL construction take place under the shared-manager process.

use PDL;
use PDL::IO::Storable;   # must load for freezing/thawing

use MCE::Shared;         # must load MCE::Shared after PDL

# not efficient from memory copy/transfer and unnecessary destruction
my $ob1 = MCE::Shared->share( zeroes( 256, 256 ) );

# efficient
my $ob1 = MCE::Shared->zeroes( 256, 256 );
ins_inplace

The ins_inplace method applies to shared PDL objects. It supports two forms for writing bits back into the PDL object residing under the shared-manager process.

# --- action taken by the shared-manager process
# ins_inplace(  2 args ):   $this->slice( $arg1 ) .= $arg2;
# ins_inplace( >2 args ):   ins( inplace( $this ), $what, @coords );

# --- use case
$o->ins_inplace( ":,$start:$stop", $result );  #  2 args
$o->ins_inplace( $result, 0, $seq_n );         # >2 args

The MCE-Cookbook on Github provides a couple working PDL demonstrations for further reading.

https://github.com/marioroy/mce-cookbook

COMMON API

blessed

Returns the real blessed name, provided by the shared-manager process.

use Scalar::Util qw(blessed);
use MCE::Shared;

use MCE::Shared::Ordhash;
use Hash::Ordered;

my $oh1 = MCE::Shared->share( MCE::Shared::Ordhash->new() );
my $oh2 = MCE::Shared->share( Hash::Ordered->new() );

print blessed($oh1), "\n";    # MCE::Shared::Object
print blessed($oh2), "\n";    # MCE::Shared::Object

print $oh1->blessed(), "\n";  # MCE::Shared::Ordhash
print $oh2->blessed(), "\n";  # Hash::Ordered
destroy

Exports optionally, but destroys the shared object entirely from the shared-manager process.

my $exported_ob = $shared_ob->destroy();

$shared_ob; # becomes undef
export

Exports the shared object into a non-shared object. One must export when passing the shared object into any dump routine. Otherwise, the data ${ SHARED_ID } is all one will see.

use MCE::Shared;
use MCE::Shared::Ordhash;

sub _dump {
   require Data::Dumper unless $INC{'Data/Dumper.pm'};
   no warnings 'once';

   local $Data::Dumper::Varname  = 'VAR';
   local $Data::Dumper::Deepcopy = 1;
   local $Data::Dumper::Indent   = 1;
   local $Data::Dumper::Purity   = 1;
   local $Data::Dumper::Sortkeys = 0;
   local $Data::Dumper::Terse    = 0;

   print Data::Dumper::Dumper($_[0]) . "\n";
}

# these do the same thing
my $oh1 = MCE::Shared->share( MCE::Shared::Ordhash->new() );
my $oh2 = MCE::Shared->ordhash();

_dump($oh1);  # ${ 1 }  # SHARED_ID value
_dump($oh2);  # ${ 2 }

_dump($oh1->export());  # actual structure and content
_dump($oh2->export());

export can optionally take a list of indices/keys for what to export. This applies to shared array, hash, and ordhash.

use MCE::Shared;

my $h1 = MCE::Shared->hash(           # shared hash
   qw/ I Heard The Bluebirds Sing by Marty Robbins /
     # k v     k   v         k    v  k     v
);

my $h2 = $h1->export( qw/ I The / );  # non-shared hash

_dump($h2);

__END__

$VAR1 = bless( {
  'I' => 'Heard',
  'The' => 'Bluebirds'
}, 'MCE::Shared::Hash' );
next
prev
reset

next, prev, and reset enables parallel iteration between workers for shared array, hash, ordhash, and sequence. Call reset after running to start over. Workers may iterate either direction next or prev.

use MCE::Hobo;
use MCE::Shared;

my $ob = MCE::Shared->array( 'a' .. 'j' );

sub parallel {
   my ($id) = @_;
   while (defined (my $item = $ob->next)) {
      print "$id: $item\n";
      sleep 1;
   }
}

MCE::Hobo->new( \&parallel, $_ ) for 1 .. 3;

# ... do other work ...

$_->join() for MCE::Hobo->list();

-- Output

1: a
2: b
3: c
2: f
1: d
3: e
2: g
3: i
1: h
2: j

There are two forms for iterating through a shared hash or ordhash object.

use MCE::Hobo;
use MCE::Shared;

my $ob = MCE::Shared->ordhash(
   map {( "key_$_" => "val_$_" )} "a" .. "j"
);

sub iter1 {
   my ($id) = @_;
   while ( my ($key, $val) = $ob->next ) {
      print "$id: $key => $val\n";
      sleep 1;
   }
}

sub iter2 {
   my ($id) = @_;
   while ( defined (my $val = $ob->prev) ) {
      print "$id: $val\n";
      sleep 1;
   }
}

MCE::Hobo->new(\&iter1, $_) for 1 .. 3;
$_->join() for MCE::Hobo->list();

$ob->reset();

MCE::Hobo->new(\&iter2, $_) for 1 .. 3;
$_->join() for MCE::Hobo->list();

The shared-manager process will iterate orderly, but there is no guarantee for the amount of time required by workers. Thus, output may not be ordered.

-- Output

1: key_a => val_a
2: key_b => val_b
3: key_c => val_c
1: key_d => val_d
2: key_f => val_f
3: key_e => val_e
3: key_i => val_i
2: key_g => val_g
1: key_h => val_h
3: key_j => val_j
1: val_j
2: val_i
3: val_h
1: val_g
3: val_e
2: val_f
3: val_d
2: val_b
1: val_c
2: val_a
dset
dmerge
dpush
dunshift

Deep-sharing non-blessed structure(s) is possible with dset, dmerge, dpush, and dunshift. These do the same thing as their counterparts set, merge, push, and unshift. These methods traverse the list or hash recursively and covert non-blessed deeply-structures to shared objects.

SERVER API

start

Starts the shared-manager process. This is done automatically.

MCE::Shared->start();
stop

Stops the shared-manager process wiping all shared data content. This is not typically done by the user, but rather by END automatically when the script terminates.

MCE::Shared->stop();
init

This is called automatically by each MCE/Hobo worker immediately after being spawned for selection of 1 of 8 data channels. The effect is extra parallelism during inter-process communication. The ID (optionally - must be an integer) is modded with 8 in a round-robin fashion.

MCE::Shared->init();
MCE::Shared->init( ID );

INDEX

MCE, MCE::Core

AUTHOR

Mario E. Roy, <marioeroy AT gmail DOT com>