NAME
PDL::Parallel::MPI
Routines to allow PDL objects to be moved around on parallel systems using the MPI library.
SYNOPSIS
use PDL;
use PDL::Parallel::MPI;
mpirun(2);
MPI_Init();
$rank = get_rank();
$a=$rank * ones(2);
print "my rank is $rank and \$a is $a\n";
$a->move( 1 => 0);
print "my rank is $rank and \$a is $a\n";
MPI_Finalize();MPI STANDARD CALLS
Most of the functions from the MPI standard may be used from this module on regular perl data. This is functionallity inherited from the Parallel::MPI module. Read the documentation for Parallel::MPI to see how to use.
One may mix mpi calls on perl built-in-datatypes and mpi calls on piddles.
use PDL;
use PDL::Parallel::MPI;
mpirun(2);
MPI_Init();
$rank = get_rank();
$pi = 3.1;
if ($rank == 0) {
MPI_Send(\$pi,1,MPI_DOUBLE,1,0,MPI_COMM_WORLD);
} else {
$message = zeroes(1);
$message->receive(0);
print "pi is $message\n";
}
MPI_Finalize();MPI GENERIC CALLS
MPI_Init
Call this before you pass around any piddles or make any mpi calls.
# usage:
MPI_Init();MPI_Finalize
Call this before your mpi program exits or you may get zombies.
# usage:
MPI_Finalize();get_rank
Returns an integer specifying who this process is. Starts at 0. Optional communicator argument.
# usage
get_rank($comm); # comm is optional and defaults to MPI_COMM_WORLDcomm_size
Returns an integer specifying how many processes there are. Optional communicator argument.
# usage
comm_size($comm); # comm is optional and defaults to MPI_COMM_WORLDmpirun
Typically one would invoke a mpi program using mpirun, which comes with your mpi compiler. This function simply a dirty hack which makes a script to invoke itself using that program.
mpirun($number_of_processes);PDL SPECIFIC MPI CALLS
move
move is a piddle method. It copies a piddle from one processes onto another processes. The first arguement is the rank of the source processes, and the second argument is the rank of the receiving processes. The piddle should be the allocated to be same size and datatype on both machines (this is not checked). The method does nothing if executed on a process which is neither the source or the destination. => may be used in place of "," for readability.
# usage
$piddle->move($source_processor , $dest_processor);# example
$a = $rank * ones(4);
$a->move( 0 => 1);send / receive
You can use send and receive to move piddles around by yourself, although I really recommend using move instead.
$piddle->send($dest,$tag,$comm); # $tag and $comm are optional
$piddle->receive($source,$tag,$comm); # didobroadcast
Piddle method which copies the value at the root process to all of the other processes in the communicator. The root defaults to 0 if not specified and the communicator to MPI_COMM_WORLD. Piddles should be pre-allocated to be the same size and datatype on all processes.
# usage
$piddle->broadcast($root,$comm); # $root and $comm optional# example
$a=$rank * ones(4);
$a->broadcast(3);send_nonblocking / receive_nonblocking
These piddle methods initiate communication and return before that communication is completed. They return a request object which can be checked for completion or waited on. Data at source and dest should be pre-allocated to have the same size and datatype.
# $tag and $comm are optional arguments.
$request = $piddle->send_nonblocking($dest_proc,$tag,$comm);
$request = $piddle->receive_nonblocking($dest_proc,$tag,$comm);
...
$request->wait(); # blocks until the communication is completed.
or
$request->test(); # returns true if the communication is completed.
# $request is deallocated after a wait or test returns true.# this example is similar to how mpi_rotate is implemented.
$r_send = $source->send_nonblocking(($rank+1) % $population);
$r_receive = $dest->receive_nonblocking(($rank-1) % $population);
$r_receive->wait();
$r_send->wait();get_status / print_status
get_status returns a hashref which contains the status of the last receive. The fields are count, source, tag, and error. print_status simply prints out the status nicely. Note that if there is an error in a receive and exception will be thrown.
print_status();
print ${get_status()}{count};mpi_rotate
mpi_rotate is a piddle method which should be executed at the same time on all processors. For each process, it moves the entire piddle to the next process. This movement is (inefficently) done in place by default, or you can specify a destination.
$piddle->mpi_rotate(
dest => $dest_piddle, # optional
offset => $offset, # optional, defaults to +1
);scatter
Takes a piddle and splits its data onto all of the processors. This would take an n-dimensional piddle on the root and turn it into an n-1 dimensional piddle on all processors. It may be called as a piddle method, which is equivilant to simply specifing the 'source' argument. On the root, one must specify the source. On all other procs, one may also pass a 'source' argument to allow scatter to grok the size of the destination piddle to allocate. Alternatively on the non-root procs one may specify the dest piddle explicitly, or simply specify the dimensions of the destionation piddle.
# usage (all arguments are optional, but see above).
# may be used as a piddle method, which simply sets the
# source argument.
$dest_piddle = scatter(
source => $src_piddle,
dest => $dest_piddle,
dims => $array_ref,
root => $root_proc, # root defaults to 0.
comm => $comm, # defaults to MPI_COMM_WORLD
);# with 4 processes
$a = sequence(4,4);
$b = $a->scatter;gather
gather is the opposite of scatter. Using it as a piddle method simply specifies the source. If called on an n dimensional piddle on all procs, the root will contain an n+1 dimensional piddle on completion.
memory =>
+------------+ +-------------+
^ |a0 | -----> | a0 a1 a2 a3 |
procs | |a1 | gather | |
| |a2 | | |
|a3 | <----- | |
+------------+ scatter +-------------+# usage
gather(
source => $src_piddle,
dest => $dest_piddle, # only used at root, extrapolated from source if not specified.
root => $root_proc, # defaults to 0
comm => $comm, # defaults to MPI_COMM_WORLD
);# example. assume nprocs == 4.
$a = ones(4);
$b = $a->gather;
# $b->dims now is (4,4) on proc 0.allgather
allgather does the same thing as gather except that the result is placed on all processors rather than just the root.
memory =>
+------------+ +-------------+
^ |a0 | | a0 a1 a2 a3 |
procs | |a1 | -----> | a0 a1 a2 a3 |
| |a2 | allgather | a0 a1 a2 a3 |
|a3 | | a0 a1 a2 a3 |
+------------+ +-------------+alltoall
memory =>
+-------------+ +-------------+
^ | a0 a1 a2 a3 | | a0 b0 c0 d0 |
procs | | b0 b1 b2 b3 | -----> | a1 b1 c1 d1 |
| | c0 c1 c2 c3 | alltoall | a2 b2 c2 d2 |
| d0 d1 d2 d3 | | a3 b3 c3 d3 |
+-------------+ +-------------+# usage
# calling as piddle method simply sets the source argument.
$dest_piddle = alltoall(
source => $src_piddle,
dest => $dest_piddle, # created for you if not passed.
comm => $comm, # defaults to MPI_COMM_WORLD.
);# example: assume comm_size is 4.
$a = $rank * sequence(4);
$b = $a->alltoall;reduce
+-------------+ +----------------------------------+
| a0 a1 a2 a3 | reduce | a0+b0+c0+d0 , a1+b1+c1+d1, .... |
| b0 b1 b2 b3 | -----> | |
| c0 c1 c2 c3 | | |
| d0 d1 d2 d3 | | |
+-------------+ +----------------------------------+Allowed operations are: + * max min & | ^ and or xor.
# usage (also as piddle method; source is set)
$dest_piddle = reduce(
source => $src_piddle,
dest => $dest_piddle, # signifigant only at root & created for you if not specified
root => $root, # defaults to 0
op => $op, # defaults to '+'
comm => $comm, # defaults to MPI_COMM_WORLD
);# example
$a=$rank * (sequence(4)+1);
$b=$a->reduce; allreduce
Just like reduce except that the result is put on all the processes.
# usage (also as piddle method; source is set)
$dest_piddle = allreduce(
source => $src_piddle,
dest => $dest_piddle, # created for you if not specified
root => $root, # defaults to 0
op => $op, # defaults to '+'
comm => $comm, # defaults to MPI_COMM_WORLD
);# example
$a=$rank * (sequence(4)+1);
$b=$a->allreduce; scan
+-------------+ +----------------------------------+
| a0 a1 a2 a3 | scan | a0 , a1 , a2 , a3 |
| b0 b1 b2 b3 | -----> | a0+b0 , a1+b1 , a2+b2, a3+b3 |
| c0 c1 c2 c3 | | |
| d0 d1 d2 d3 | | ... |
+-------------+ +----------------------------------+Allowed operations are: + * max min & | ^ and or xor.
# usage (also as piddle method; source is set)
$dest_piddle = scan(
source => $src_piddle,
dest => $dest_piddle, # created for you if not specified
root => $root, # defaults to 0
op => $op, # defaults to '+'
comm => $comm, # defaults to MPI_COMM_WORLD
);# example
$a=$rank * (sequence(4)+1);
$b=$a->scan; reduce_and_scatter
Does a reduce followed by a scatter. A regular scatter distributes the data evenly over all processes, but with reduce_and_scatter you get to specify the distribution (if you want; defaults to uniform).
# usage (also as piddle method; source is set)
$dest_piddle = reduce_and_scatter(
source => $src_piddle,
dest => $dest_piddle, # created for you if not specified.
recv_count => $recv_count_piddle # 1D int piddle. put $r[$i] elements on proc $i.
op => $op, # defaults to '+'
comm => $comm, # defaults to MPI_COMM_WORLD
);# example, taken from t/20_reduce_and_scatter.t
mpirun(4); MPI_Init(); $rank = get_rank();
$a=$rank * (sequence(4)+1);
$b=$a->reduce_and_scatter;
print "rank = $rank, b=$b\n" ;WARNINGS
This module is still under development. Signifigant changes are expected. As things are *expected* to break somehow or another, there is no waranty, and the author does not take responsibility for the damage/distruction of your data/computer/sanity.
PLANS
indexing
Currently there is no support for any sort of indexing/dataflow/child piddles. Ideally one would like to say:
$piddle->diagonal(0,1)->move(0 => 1);But currently one must say:
$tmp = $piddle->diagonal(0,1)->copy;
$tmp->move( 0 => 1);
$piddle->diagonal(0,1) .= $tmp;I believe the former behavior to be possible to implement. I plan to do so once I reach a sufficent degree of enlightenment.
distributed data
One might wish to use their own personal massively parallel supercomputer interactively with the pdl shell (perldl). This would require master/slave interactions and a distributed data model. Such a project won't be started until after I finish PDL::Parallel::OpenMP.
AUTHOR
Darin McGill darin@ocf.berkeley.edu
If you find this module useful, please let me know. I very much appreciate bug reports, suggestions and other feedback.
ACKNOWLEDGEMENTS
This module is an extension of Parallel::MPI written by Josh Wilmes and Chris Stevens. Signifigant portions of code has been copied from Parallel::MPI verbatim. Used with permission. Josh and Chris did most of the work to make perl's built in datatypes (scalars, arrays) work with MPI. I rely heavily on their work for MPI intitialization and error handling.
The diagrams in this document were inspired by and are similar to diagrams found in MPI-The Complete Reference.
Sections of code from the main PDL distribution (such as header files, and code from PDL::CallExt.xs) were used extensively in development. Many thanks to the PDL developers for their help, and of course, for creating the PDL system in the first place.
SEE ALSO
The PDL::Parallel homepage: http://www.ocf.berkeley.edu/~darin/projects/superperl
The PDL home page: http://pdl.perl.org
The Perl module Parallel::MPI.
The Message Passing Interface: http://www.mpi-forum.org/
PDL::Parallel::OpenMP (under development).
COPYING
This module is free software. It may be modified and/or redistributed under the same terms as perl itself.