NAME

MCE::Examples - A list of examples demonstrating Many-core Engine

VERSION

This document describes MCE::Examples version 1.499_002

DESCRIPTION

MCE comes with various examples showing real-world use case scenarios on parallelizing something as small as cat (try with -n) to searching for patterns and word count aggregation.

INCLUDED with DISTRIBUTION


            
              
              barrier_sync.pl
          A barrier sync demonstration.
cat.pl    Concatenation script, similar to the cat binary.
egrep.pl  Egrep script, similar to the egrep binary.
wc.pl     Word count script, similar to the wc binary.
findnull.pl
          A parallel driven script to report lines containing
          null fields. It's many times faster than the binary
          egrep command. Try against a large file containing
          very long lines.
foreach.pl, forseq.pl, forchunk.pl
          These take the same sqrt example from Parallel::Loops
          and measures the overhead of the engine. The number
          indicates the size of @input which can be submitted
          and results displayed in under 1 second.
          Parallel::Loops is based on Parallel::ForkManager.
          MCE utilizes a pool of workers.
          Parallel::Loops:     600  Forking each @input is expensive
          MCE foreach....:  34,000  Sends result after each @input
          MCE forseq.....:  70,000  Loops through sequence of numbers
          MCE forchunk...: 465,000  Chunking reduces overhead
interval.pl
          Demonstration of the interval option appearing in MCE 1.5.
matmult/matmult_base.pl, matmult_mce.pl, strassen_mce.pl
          Various matrix multiplication demonstrations benchmarking
          PDL, PDL + MCE, as well as parallelizing Strassen's
          divide-and-conquer algorithm. Also included are 2 plain
          Perl examples.
scaling_pings.pl
          Perform ping test and report back failing IPs to
          standard output.
seq_demo.pl
          A demonstration of the new sequence option appearing
          in MCE 1.3. Run with seq_demo.pl | sort
tbray/wf_mce1.pl, wf_mce2.pl, wf_mce3.pl
          An implementation of wide finder utilizing MCE.
          As fast as MMAP IO when file resides in OS FS cache.
          2x ~ 3x faster when reading directly from disk.

CHUNK_SIZE => 1 (in essence, wanting no chunking on input data)

Imagine a long running process and wanting to parallelize an array against a pool of workers. Note: The sequence option can be used if simply wanting to loop through a sequence of numbers in parallel one number at a time.

Below, a callback function for displaying results is used. The logic shows how one can display results immediately while still preserving the output order as if processing serially. The %result hash is a temporary cache to store results for out-of-order replies.


            
              
              my @input_data  = (0 .. 18000 - 1);
my $max_workers = 3;
my $order_id    = 1;
my %result;
## Callback function for displaying results.
sub display_result {
   my ($wk_result, $chunk_id) = @_;
   $result{$chunk_id} = $wk_result;
   while (1) {
      last unless (exists $result{$order_id});
      printf "i: %d sqrt(i): %f\n",
         $input_data[$order_id - 1], $result{$order_id};
      delete $result{$order_id};
      $order_id++;
   }
}
## Compute via MCE.
my $mce = MCE->new(
   input_data  => \@input_data,
   max_workers => $max_workers,
   chunk_size  => 1,
   user_func => sub {
      my ($self, $chunk_ref, $chunk_id) = @_;
      my $wk_result = sqrt($chunk_ref->[0]);
      MCE->do('display_result', $wk_result, $chunk_id);
   }
);
MCE->run();

FOREACH sugar METHOD


            
              
              ## Compute via MCE. Foreach implies chunk_size => 1.
my $mce = MCE->new(
   max_workers => $max_workers
);
## Worker calls code block passing a reference to an array containing
## one item. Use $chunk_ref->[0] to retrieve the single element.
MCE->foreach(\@input_data, sub {
   my ($self, $chunk_ref, $chunk_id) = @_;
   my $wk_result = sqrt($chunk_ref->[0]);
   MCE->do('display_result', $wk_result, $chunk_id);
});

CHUNKING INPUT_DATA

Chunking reduces overhead many folds. Instead of passing a single item from @input_data, a chunk of $chunk_size is sent to the next available worker. The sequence option can be used as well if simply wanting to loop through a sequence of numbers with chunking applied in parallel.


            
              
              my @input_data  = (0 .. 385000 - 1);
my $max_workers = 3;
my $chunk_size  = 500;
my $order_id    = 1;
my %result;
## Callback function for displaying results.
sub display_result {
   my ($wk_result, $chunk_id) = @_;
   $result{$chunk_id} = $wk_result;
   while (1) {
      last unless (exists $result{$order_id});
      my $i = ($order_id - 1) * $chunk_size;
      foreach ( @{ $result{$order_id} } ) {
         printf "i: %d sqrt(i): %f\n", $input_data[$i++], $_;
      }
      delete $result{$order_id};
      $order_id++;
   }
}
## Compute via MCE.
my $mce = MCE->new(
   input_data  => \@input_data,
   max_workers => $max_workers,
   chunk_size  => $chunk_size,
   user_func => sub {
      my ($self, $chunk_ref, $chunk_id) = @_;
      my @wk_result;
      foreach ( @{ $chunk_ref } ) {
         push @wk_result, sqrt($_);
      }
      MCE->do('display_result', \@wk_result, $chunk_id);
   }
);
MCE->run();

FORCHUNK sugar METHOD


            
              
              ## Compute via MCE.
my $mce = MCE->new(
   max_workers => $max_workers,
   chunk_size  => $chunk_size
);
## Below, $chunk_ref is a reference to an array containing the next
## $chunk_size items from @input_data.
MCE->forchunk(\@input_data, sub {
   my ($self, $chunk_ref, $chunk_id) = @_;
   my @wk_result;
   foreach ( @{ $chunk_ref } ) {
      push @wk_result, sqrt($_);
   }
   MCE->do('display_result', \@wk_result, $chunk_id);
});

DEMO APPLYING SEQUENCES WITH USER_TASKS

One may specify the sequence option per each task. The following is taken directly from the seq_demo.pl example. Think of the following demonstration as having 3 mini-MCEs running simultaneously in parallel. Chunking can also be configured independently as well.


            
              
              use MCE;
## Run with seq_demo.pl | sort
sub user_func {
   my ($self, $seq_n, $chunk_id) = @_;
   my $wid      = MCE->wid();
   my $task_id  = MCE->task_id();
   my $task_wid = MCE->task_wid();
   if (ref $seq_n eq 'ARRAY') {
      ## Received the next "chunked" sequence of numbers
      ## e.g. when chunk_size > 1, $seq_n will be an array ref above
      foreach (@{ $seq_n }) {
         printf(
            "task_id %d: seq_n %s: chunk_id %d: wid %d: task_wid %d\n",
            $task_id,    $_,       $chunk_id,   $wid,   $task_wid
         );
      }
   }
   else {
      printf(
         "task_id %d: seq_n %s: chunk_id %d: wid %d: task_wid %d\n",
         $task_id,    $seq_n,   $chunk_id,   $wid,   $task_wid
      );
   }
}
## Each task can be configured independently.
my $mce = MCE->new(
   user_tasks => [{
      max_workers => 2,
      chunk_size  => 1,
      sequence    => { begin => 11, end => 19, step => 1 },
      user_func   => \&user_func
   },{
      max_workers => 2,
      chunk_size  => 5,
      sequence    => { begin => 21, end => 29, step => 1 },
      user_func   => \&user_func
   },{
      max_workers => 2,
      chunk_size  => 3,
      sequence    => { begin => 31, end => 39, step => 1 },
      user_func   => \&user_func
   }]
);
MCE->run();
-- Output
task_id 0: seq_n 11: chunk_id 1: wid 1: task_wid 1
task_id 0: seq_n 12: chunk_id 2: wid 2: task_wid 2
task_id 0: seq_n 13: chunk_id 3: wid 1: task_wid 1
task_id 0: seq_n 14: chunk_id 4: wid 2: task_wid 2
task_id 0: seq_n 15: chunk_id 5: wid 1: task_wid 1
task_id 0: seq_n 16: chunk_id 6: wid 2: task_wid 2
task_id 0: seq_n 17: chunk_id 7: wid 1: task_wid 1
task_id 0: seq_n 18: chunk_id 8: wid 2: task_wid 2
task_id 0: seq_n 19: chunk_id 9: wid 1: task_wid 1
task_id 1: seq_n 21: chunk_id 1: wid 3: task_wid 1
task_id 1: seq_n 22: chunk_id 1: wid 3: task_wid 1
task_id 1: seq_n 23: chunk_id 1: wid 3: task_wid 1
task_id 1: seq_n 24: chunk_id 1: wid 3: task_wid 1
task_id 1: seq_n 25: chunk_id 1: wid 3: task_wid 1
task_id 1: seq_n 26: chunk_id 2: wid 4: task_wid 2
task_id 1: seq_n 27: chunk_id 2: wid 4: task_wid 2
task_id 1: seq_n 28: chunk_id 2: wid 4: task_wid 2
task_id 1: seq_n 29: chunk_id 2: wid 4: task_wid 2
task_id 2: seq_n 31: chunk_id 1: wid 5: task_wid 1
task_id 2: seq_n 32: chunk_id 1: wid 5: task_wid 1
task_id 2: seq_n 33: chunk_id 1: wid 5: task_wid 1
task_id 2: seq_n 34: chunk_id 2: wid 6: task_wid 2
task_id 2: seq_n 35: chunk_id 2: wid 6: task_wid 2
task_id 2: seq_n 36: chunk_id 2: wid 6: task_wid 2
task_id 2: seq_n 37: chunk_id 3: wid 5: task_wid 1
task_id 2: seq_n 38: chunk_id 3: wid 5: task_wid 1
task_id 2: seq_n 39: chunk_id 3: wid 5: task_wid 1

MULTIPLE WORKERS RUNNING IN PARALLEL

Both input_data and sequence options are optional in MCE. One can simply use MCE to parallelize multiple workers. The "do" & "sendto" methods can be used to pass data back to the manager process. One doesn't have to wait until the worker has completed processing to pass data back. Both "do" & "sendto" methods are processed serially by the main process on a first come, first serve basis. All 4 workers run in parallel for the demonstration below.


            
              
              use MCE;
sub report_stats {
   my ($wid, $msg, $hash_ref) = @_;
   print "Worker $wid says $msg: ", $hash_ref->{'counter'}, "\n";
}
my $mce = MCE->new(
   max_workers => 4,
   user_func => sub {
      my ($self) = @_;
      my $wid = MCE->wid();
      if ($wid == 1) {
         my %hash = ('counter' => 0);
         while (1) {
            $hash{'counter'} += 1;
            MCE->do('report_stats', $wid, 'Hello there', \%hash);
            last if ($hash{'counter'} == 4);
            sleep 2;
         }
      }
      else {
         my %hash = ('counter' => 0);
         while (1) {
            $hash{'counter'} += 1;
            MCE->do('report_stats', $wid, 'Welcome ...', \%hash);
            last if ($hash{'counter'} == 2);
            sleep 4;
         }
      }
      MCE->sendto('stdout', "Worker $wid is exiting\n");
   }
);
MCE->run;
Worker 2 gets there first in 2nd output below.
$ ./demo.pl
Worker 1 says Hello there: 1
Worker 2 says Welcome ...: 1
Worker 3 says Welcome ...: 1
Worker 4 says Welcome ...: 1
Worker 1 says Hello there: 2
Worker 2 says Welcome ...: 2
Worker 3 says Welcome ...: 2
Worker 1 says Hello there: 3
Worker 2 is exiting
Worker 3 is exiting
Worker 4 says Welcome ...: 2
Worker 4 is exiting
Worker 1 says Hello there: 4
Worker 1 is exiting
$ ./demo.pl
Worker 2 says Welcome ...: 1
Worker 1 says Hello there: 1
Worker 4 says Welcome ...: 1
Worker 3 says Welcome ...: 1
Worker 1 says Hello there: 2
Worker 2 says Welcome ...: 2
Worker 4 says Welcome ...: 2
Worker 3 says Welcome ...: 2
Worker 2 is exiting
Worker 4 is exiting
Worker 1 says Hello there: 3
Worker 3 is exiting
Worker 1 says Hello there: 4
Worker 1 is exiting

INDEX

MCE

AUTHOR

Mario E. Roy, <marioeroy AT gmail DOT com>

LICENSE

This program is free software; you can redistribute it and/or modify it under the terms of either: the GNU General Public License as published by the Free Software Foundation; or the Artistic License.

See http://dev.perl.org/licenses/ for more information.

To install MCE, copy and paste the appropriate command in to your terminal.

cpanm

cpanm MCE

CPAN shell

perl -MCPAN -e shell
install MCE

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)