/ 
MCE-1.699_008
River stage three • 22 direct dependents • 287 total dependents
/ MCE::Shared

NAME

MCE::Shared - MCE extension for sharing data between workers

VERSION

This document describes MCE::Shared version 1.699_008

SYNOPSIS

# OO construction

use MCE::Shared Sereal => 1;

my $ar = MCE::Shared->array( @list );
my $cv = MCE::Shared->condvar( 0 );
my $fh = MCE::Shared->handle( '>>', \*STDOUT );
my $ha = MCE::Shared->hash( @pairs );
my $db = MCE::Shared->minidb();
my $oh = MCE::Shared->ordhash( @pairs );
my $qu = MCE::Shared->queue( await => 1, fast => 0 );
my $va = MCE::Shared->scalar( $value );
my $nu = MCE::Shared->sequence( $begin, $end, $step, $fmt );
my $ob = MCE::Shared->share( $blessed_object );

# Tie construction

use MCE::Flow;
use MCE::Shared Sereal => 1;
use feature 'say';

tie my $var, 'MCE::Shared', 'initial value';
tie my @ary, 'MCE::Shared', qw(a list of values);
tie my %has, 'MCE::Shared', (key1 => 'value', key2 => 'value');

tie my $cnt, 'MCE::Shared', 0;
tie my @foo, 'MCE::Shared';
tie my %bar, 'MCE::Shared';

my $m1 = MCE::Mutex->new;

mce_flow {
   max_workers => 4
},
sub {
   my ($mce) = @_;
   my ($pid, $wid) = (MCE->pid, MCE->wid);

   ## Locking is required when multiple workers update the same element.
   ## This requires 2 trips to the manager process (fetch and store).

   $m1->synchronize( sub {
      $cnt += 1;
   });

   ## Locking is not necessary when updating unique elements.

   $foo[ $wid - 1 ] = $pid;
   $bar{ $pid }     = $wid;

   return;
};

say "scalar : $cnt";
say " array : $_" for (@foo);
say "  hash : $_ => $bar{$_}" for (sort keys %bar);

-- Output

scalar : 4
 array : 37847
 array : 37848
 array : 37849
 array : 37850
  hash : 37847 => 1
  hash : 37848 => 2
  hash : 37849 => 3
  hash : 37850 => 4

DESCRIPTION

This module provides data sharing for MCE supporting threads and processes.

MCE::Shared enables extra functionality on systems with IO::FDPass. Without it, MCE::Shared is unable to send file descriptors to the shared-manager process for queue, condvar, and possibly handle.

As of this writing, the IO::FDPass module is not a requirement for running MCE::Shared nor is the check made during installation. The reason is that IO::FDPass is not possible on Cygwin and not sure about AIX.

The following is a suggestion for systems without IO::FDPass. This restriction applies to queue, condvar, and handle only.

use MCE::Shared;

# Construct shared queue(s) and condvar(s) first.
# These contain GLOB handles - freezing not allowed.

my $q1  = MCE::Shared->queue();
my $q2  = MCE::Shared->queue();

my $cv1 = MCE::Shared->condvar();
my $cv2 = MCE::Shared->condvar();

# Start the shared-manager manually.

MCE::Shared->start();

# The shared-manager process knows of STDOUT, STDERR, STDIN

my $fh1 = MCE::Shared->handle(">>", \*STDOUT);  # ok
my $fh2 = MCE::Shared->handle("<", "/path/to/sequence.fasta");  # ok
my $h1  = MCE::Shared->hash();

Otherwise, sharing is immediate and not delayed with IO::FDPass. It is not necessary to share queue and condvar first or worry about starting the shared-manager process.

use MCE::Shared;

my $h1 = MCE::Shared->hash();    # shares immediately
my $q1 = MCE::Shared->queue();   # IO::FDPass sends file descriptors
my $cv = MCE::Shared->condvar(); # IO::FDPass sends file descriptors
my $h2 = MCE::Shared->ordhash();

DATA SHARING

array
condvar
handle
hash
minidb
ordhash
queue
scalar
sequence

array, condvar, handle, hash, minidb, ordhash, queue, scalar, and sequence are sugar syntax for constructing a shared object.

# long form

use MCE::Shared;
use MCE::Shared::Array;
use MCE::Shared::Hash;

my $ar = MCE::Shared->share( MCE::Shared::Array->new() );
my $ha = MCE::Shared->share( MCE::Shared::Hash->new() );

# short form

use MCE::Shared;

my $ar = MCE::Shared->array( @list );
my $cv = MCE::Shared->condvar( 0 );
my $fh = MCE::Shared->handle( '>>', \*STDOUT );
my $ha = MCE::Shared->hash( @pairs );
my $db = MCE::Shared->minidb();
my $oh = MCE::Shared->ordhash( @pairs );
my $qu = MCE::Shared->queue( await => 1, fast => 0 );
my $va = MCE::Shared->scalar( $value );
my $nu = MCE::Shared->sequence( $begin, $end, $step, $fmt );
num_sequence

num_sequence is an alias for sequence.

OBJECT SHARING

share

This class method transfers the blessed-object to the shared-manager process and returns a MCE::Shared::Object containing the SHARED_ID. The object must not contain any GLOB's or CODE_REF's or the transfer will fail.

Unlike threads::shared, objects are not deeply shared. The shared object is accessable only through the underlying OO interface.

use MCE::Shared;
use Hash::Ordered;

my ($ho_shared, $ho_unshared);

$ho_shared = MCE::Shared->share( Hash::Ordered->new() );

$ho_shared->push( @pairs );            # OO interface only
$ho_shared->mset( @pairs );

$ho_unshared = $ho_shared->export();   # back to unshared
$ho_unshared = $ho_shared->destroy();  # including destruction

The following provide long and short forms for constructing a shared array, hash, or scalar object.

use MCE::Shared;

use MCE::Shared::Array;    # Loading helper classes is not necessary
use MCE::Shared::Hash;     # when using the shorter form.
use MCE::Shared::Scalar;

my $a1 = MCE::Shared->share( MCE::Shared::Array->new( @list ) );
my $a3 = MCE::Shared->share( [ @list ] );  # sugar syntax
my $a2 = MCE::Shared->array( @list );

my $h1 = MCE::Shared->share( MCE::Shared::Hash->new( @pairs ) );
my $h3 = MCE::Shared->share( { @pairs } ); # sugar syntax
my $h2 = MCE::Shared->hash( @pairs );

my $s1 = MCE::Shared->share( MCE::Shared::Scalar->new( 20 ) );
my $s2 = MCE::Shared->share( \do{ my $o = 20 } );
my $s4 = MCE::Shared->scalar( 20 );

PDL SHARING

pdl_byte
pdl_short
pdl_ushort
pdl_long
pdl_longlong
pdl_float
pdl_double
pdl_ones
pdl_sequence
pdl_zeroes
pdl_indx
pdl

pdl_byte, pdl_short, pdl_ushort, pdl_long, pdl_longlong, pdl_float, pdl_double, pdl_ones, pdl_sequence, pdl_zeroes, pdl_indx, and pdl are sugar syntax for PDL construction take place under the shared-manager process.

use PDL;
use PDL::IO::Storable;   # must load for freezing/thawing

use MCE::Shared;         # must load MCE::Shared after PDL

# not efficient from memory copy/transfer and unnecessary destruction
my $ob1 = MCE::Shared->share( zeroes( 256, 256 ) );

# efficient
my $ob1 = MCE::Shared->zeroes( 256, 256 );
ins_inplace

The ins_inplace method applies to shared PDL objects. It supports two forms for writing bits back into the PDL object residing under the shared-manager process.

# --- action taken by the shared-manager process
# ins_inplace(  2 args ):   $this->slice( $arg1 ) .= $arg2;
# ins_inplace( >2 args ):   ins( inplace( $this ), $what, @coords );

# --- use case
$o->ins_inplace( ":,$start:$stop", $result );  #  2 args
$o->ins_inplace( $result, 0, $seq_n );         # >2 args

The MCE-Cookbook on Github provides a couple working PDL demonstrations for further reading.

https://github.com/marioroy/mce-cookbook

COMMON API

blessed

Returns the real blessed name, provided by the shared-manager process.

use Scalar::Util qw(blessed);
use MCE::Shared;

use MCE::Shared::Ordhash;
use Hash::Ordered;

my $oh1 = MCE::Shared->share( MCE::Shared::Ordhash->new() );
my $oh2 = MCE::Shared->share( Hash::Ordered->new() );

print blessed($oh1), "\n";    # MCE::Shared::Object
print blessed($oh2), "\n";    # MCE::Shared::Object

print $oh1->blessed(), "\n";  # MCE::Shared::Ordhash
print $oh2->blessed(), "\n";  # Hash::Ordered
destroy

Exports optionally, but destroys the shared object entirely from the shared-manager process.

my $exported_ob = $shared_ob->destroy();

$shared_ob; # becomes undef
export
export ( keys )

Exports the shared object into a non-shared object. One must export when passing the shared object into any dump routine. Otherwise, the data ${ SHARED_ID } is all one will see.

use MCE::Shared;
use MCE::Shared::Ordhash;

sub _dump {
   require Data::Dumper unless $INC{'Data/Dumper.pm'};
   no warnings 'once';

   local $Data::Dumper::Varname  = 'VAR';
   local $Data::Dumper::Deepcopy = 1;
   local $Data::Dumper::Indent   = 1;
   local $Data::Dumper::Purity   = 1;
   local $Data::Dumper::Sortkeys = 0;
   local $Data::Dumper::Terse    = 0;

   print Data::Dumper::Dumper($_[0]) . "\n";
}

# these do the same thing
my $oh1 = MCE::Shared->share( MCE::Shared::Ordhash->new() );
my $oh2 = MCE::Shared->ordhash();

_dump($oh1);  # ${ 1 }  # SHARED_ID value
_dump($oh2);  # ${ 2 }

_dump($oh1->export());  # actual structure and content
_dump($oh2->export());

export can optionally take a list of indices/keys for what to export. This applies to shared array, hash, and ordhash.

use MCE::Shared;

my $h1 = MCE::Shared->hash(           # shared hash
   qw/ I Heard The Bluebirds Sing by Marty Robbins /
     # k v     k   v         k    v  k     v
);

my $h2 = $h1->export( qw/ I The / );  # non-shared hash

_dump($h2);

__END__

$VAR1 = bless( {
  'I' => 'Heard',
  'The' => 'Bluebirds'
}, 'MCE::Shared::Hash' );
next
rewind
rewind ( "query string" )
rewind ( begin, end, [ step, format ] )

next and rewind enable parallel iteration between workers for shared array, hash, ordhash, and sequence. Call rewind after running to reset the pointer.

use MCE::Hobo;
use MCE::Shared;

my $ob = MCE::Shared->array( 'a' .. 'j' );

sub parallel {
   my ($id) = @_;
   while (defined (my $item = $ob->next)) {
      print "$id: $item\n";
      sleep 1;
   }
}

MCE::Hobo->new( \&parallel, $_ ) for 1 .. 3;

# ... do other work ...

$_->join() for MCE::Hobo->list();

-- Output

1: a
2: b
3: c
2: f
1: d
3: e
2: g
3: i
1: h
2: j

There are two forms for iterating through a shared hash or ordhash object. The next method is wantarray-aware providing key and value in list context and value only in scalar context.

use MCE::Hobo;
use MCE::Shared;

my $ob = MCE::Shared->ordhash(
   map {( "key_$_" => "val_$_" )} "a" .. "j"
);

sub iter1 {
   my ($id) = @_;
   while ( my ($key, $val) = $ob->next ) {
      print "$id: $key => $val\n";
      sleep 1;
   }
}

sub iter2 {
   my ($id) = @_;
   while ( defined (my $val = $ob->next) ) {
      print "$id: $val\n";
      sleep 1;
   }
}

MCE::Hobo->new(\&iter1, $_) for 1 .. 3;
$_->join() for MCE::Hobo->list();

$ob->rewind();

MCE::Hobo->new(\&iter2, $_) for 1 .. 3;
$_->join() for MCE::Hobo->list();

Although the shared-manager process iterates orderly, there is no guarantee for the amount of time required by workers. Thus, output may not be ordered.

-- Output

1: key_a => val_a
2: key_b => val_b
3: key_c => val_c
1: key_d => val_d
3: key_f => val_f
2: key_e => val_e
1: key_g => val_g
3: key_i => val_i
2: key_h => val_h
1: key_j => val_j
1: val_a
2: val_b
3: val_c
3: val_f
1: val_d
2: val_e
3: val_h
1: val_g
2: val_i
3: val_j
store ( key, value )

Deep-sharing non-blessed structure(s) is possible with store only. store, an alias to STORE, converts non-blessed deeply-structures to shared objects recursively.

use MCE::Shared;

my $h1 = MCE::Shared->hash();
my $h2 = MCE::Shared->hash();

# auto-shares deeply
$h1->store( 'key', [ 0, 2, 5, { 'foo' => 'bar' } ] );
$h2->{key}[3]{foo} = 'baz';   # via auto-vivification

my $v1 = $h1->get('key')->get(3)->get('foo');  # bar
my $v2 = $h2->get('key')->get(3)->get('foo');  # baz
my $v3 = $h2->{key}[3]{foo};                   # baz

Each level in a deeply structure requires a separate trip to the shared-manager processs. There is a faster way if the app calls for just HoH and/or HoA. The included MCE::Shared::Minidb module provides optimized methods for working with HoH and HoA structures.

See MCE::Shared::Minidb.

SERVER API

start

Starts the shared-manager process. This is done automatically.

MCE::Shared->start();
stop

Stops the shared-manager process wiping all shared data content. This is not typically done by the user, but rather by END automatically when the script terminates.

MCE::Shared->stop();
init

This is called automatically by each MCE/Hobo worker immediately after being spawned. The effect is extra parallelism during inter-process communication. The optional ID (an integer) is modded in a round-robin fashion.

MCE::Shared->init( ID );
MCE::Shared->init();

INDEX

MCE, MCE::Core

AUTHOR

Mario E. Roy, <marioeroy AT gmail DOT com>