—

              
HideShow 291 lines of Pod
=head1 NAME
MCE - Many-Core Engine for Perl providing parallel processing capabilities
=head1 VERSION
This document describes MCE version 1.699_001
Many-Core Engine (MCE) for Perl helps enable a new level of performance by
maximizing all available cores.
=begin html
<p><img src="https://raw.githubusercontent.com/marioroy/mce-assets/master/images_README/MCE.png" width="612" height="444" alt="MCE" /></p>
=end html
=head1 DESCRIPTION
MCE spawns a pool of workers and therefore does not fork a new process per
each element of data. Instead, MCE follows a bank queuing model. Imagine the
line being the data and bank-tellers the parallel workers. MCE enhances that
model by adding the ability to chunk the next n elements from the input
stream to the next available worker.
=begin html
<p><img src="https://raw.githubusercontent.com/marioroy/mce-assets/master/images_README/Bank_Queuing_Model.png" width="664" height="446" alt="Bank Queuing Model" /></p>
=end html
=head1 SYNOPSIS
This is a simplistic use case of MCE running with 5 workers.
   # Construction using the Core API
   use MCE;
   my $mce = MCE->new(
      max_workers => 5,
      user_func => sub {
         my ($mce) = @_;
         $mce->say("Hello from " . $mce->wid);
      }
   );
   $mce->run;
   # Construction using a MCE model
   use MCE::Flow max_workers => 5;
   mce_flow sub {
      my ($mce) = @_;
      MCE->say("Hello from " . MCE->wid);
   };
Parsing a huge log file.
   use MCE::Loop;
   MCE::Loop::init { max_workers => 8, use_slurpio => 1 };
   my $pattern  = 'karl';
   my $hugefile = 'very_huge.file';
   my @result = mce_loop_f {
      my ($mce, $slurp_ref, $chunk_id) = @_;
      # Quickly determine if a match is found.
      # Process slurped chunk only if true.
      if ($$slurp_ref =~ /$pattern/m) {
         my @matches;
         # The following is fast on Unix, but performance degrades
         # drastically on Windows beyond 4 workers.
         open my $MEM_FH, '<', $slurp_ref;
         binmode $MEM_FH, ':raw';
         while (<$MEM_FH>) { push @matches, $_ if (/$pattern/); }
         close   $MEM_FH;
         # Therefore, use the following construction on Windows.
         while ( $$slurp_ref =~ /([^\n]+\n)/mg ) {
            my $line = $1; # save $1 to not lose the value
            push @matches, $line if ($line =~ /$pattern/);
         }
         # Gather matched lines.
         MCE->gather(@matches);
      }
   } $hugefile;
   print join('', @result);
Looping through a sequence of numbers.
   use MCE::Flow;
   use MCE::Shared;
   my $pi = MCE::Shared->scalar( 0.0 );
   my $N  = shift || 4_000_000;
   sub compute_pi {
      my ( $begin_seq, $end_seq ) = @_;
      my ( $_pi, $t ) = ( 0.0 );
      foreach my $i ( $begin_seq .. $end_seq ) {
         $t = ( $i + 0.5 ) / $N;
         $_pi += 4.0 / ( 1.0 + $t * $t );
      }
      $pi->incrby( $_pi );
   }
   # Compute bounds only, workers receive [ begin, end ] values
   MCE::Flow::init { bounds_only => 1 };
   mce_flow_s sub { compute_pi( $_->[0], $_->[1] ) }, 0, $N - 1;
   printf "pi = %0.13f\n", $pi->get / $N;  # 3.1415926535898
=head1 CORE MODULES
Three modules make up the core engine for MCE.
=over 3
=item L<MCE::Core|MCE::Core>
Provides the Core API for Many-Core Engine.
=item L<MCE::Signal|MCE::Signal>
Temporary directory creation/cleanup and signal handling.
=item L<MCE::Util|MCE::Util>
Utility functions for Many-Core Engine.
=back
=head1 MCE EXTRAS
There are 6 add-on modules for use with MCE.
=over 3
=item L<MCE::Candy|MCE::Candy>
Provides a collection of sugar methods and output iterators for preserving
output order.
=item L<MCE::Hobo|MCE::Hobo>
Provides threads-like capability for executing code asynchronously via child
processes.
=item L<MCE::Mutex|MCE::Mutex>
Provides a simple semaphore implementation supporting threads and processes.
=item L<MCE::Queue|MCE::Queue>
Provides a hybrid queuing implementation for MCE supporting normal queues and
priority queues from a single module. MCE::Queue exchanges data via the core
engine to enable queueing to work for both children (spawned from fork) and
threads.
=item L<MCE::Relay|MCE::Relay>
Enables workers to receive and pass on information orderly with zero
involvement by the manager process while running.
=item L<MCE::Shared|MCE::Shared>
An extension for sharing data or objects supporting threads and processes.
Eight classes are included for use with MCE::Shared.
L<MCE::Shared::Array|MCE::Shared::Array>
L<MCE::Shared::Condvar|MCE::Shared::Condvar>
L<MCE::Shared::Handle|MCE::Shared::Handle>
L<MCE::Shared::Hash|MCE::Shared::Hash>
L<MCE::Shared::Ordhash|MCE::Shared::Ordhash>
L<MCE::Shared::Queue|MCE::Shared::Queue>
L<MCE::Shared::Scalar|MCE::Shared::Scalar>
L<MCE::Shared::Sequence|MCE::Shared::Sequence>
=back
=head1 MCE MODELS
The models take Many-Core Engine to a new level for ease of use. Two options
(chunk_size and max_workers) are configured automatically as well as spawning
and shutdown.
=over 3
=item L<MCE::Loop|MCE::Loop>
Provides a parallel loop utilizing MCE for building creative loops.
=item L<MCE::Flow|MCE::Flow>
A parallel flow model for building creative applications. This makes use of
user_tasks in MCE. The author has full control when utilizing this model.
MCE::Flow is similar to MCE::Loop, but allows for multiple code blocks to
run in parallel with a slight change to syntax.
=item L<MCE::Grep|MCE::Grep>
Provides a parallel grep implementation similar to the native grep function.
=item L<MCE::Map|MCE::Map>
Provides a parallel map model similar to the native map function.
=item L<MCE::Step|MCE::Step>
Provides a parallel step implementation utilizing MCE::Queue between user
tasks. MCE::Step is a spin off from MCE::Flow with a touch of MCE::Stream.
This model, introduced in 1.506, allows one to pass data from one sub-task
into the next transparently.
=item L<MCE::Stream|MCE::Stream>
Provides an efficient parallel implementation for chaining multiple maps
and greps together through user_tasks and MCE::Queue. Like with MCE::Flow,
MCE::Stream can run multiple code blocks in parallel with a slight change
to syntax from MCE::Map and MCE::Grep.
=back
=head1 MISCELLANEOUS
Miscellaneous additions included with the distribution.
=over 3
=item L<MCE::Examples|MCE::Examples>
Describes various demonstrations for MCE including a Monte Carlo simulation.
=item L<MCE::Subs|MCE::Subs>
Exports functions mapped directly to MCE methods; e.g. mce_wid. The module
allows 3 options; :manager, :worker, and :getter.
=back
=head1 REQUIREMENTS
Perl 5.8.0 or later. PDL::IO::Storable is required in scripts running PDL.
=head1 SOURCE AND FURTHER READING
The source, examples, and cookbook are hosted at GitHub.
L<https://github.com/marioroy/mce-perl>
L<https://github.com/marioroy/mce-examples>
L<https://github.com/marioroy/mce-cookbook>
=head1 AUTHOR
Mario E. Roy, S<E<lt>marioeroy AT gmail DOT comE<gt>>
=head1 COPYRIGHT AND LICENSE
Copyright (C) 2012-2015 by Mario E. Roy
MCE is released under the same license as Perl.
See L<http://dev.perl.org/licenses/> for more information.
=cut