This directory contains a minimal port of MPI to the QCDOC using my
qmemcpy routines.  Since they are in C++, things should be compiled as
if they were C++.  This can be done by giving source files a .C
extension.

This directory can be found as

http://thy.phy.bnl.gov/~creutz/qcdoc/mpi

The mpitest program is adapted from various mpi tutorials.  The
Makefile is for use on the qcdoc host machines.  To compile the
examples for linux, Makefile.linux sets useful flags.

For simple Send/Recv pairs: the send buffer is immediately copied out,
so many sends can be stacked up.  In contrast, there is currently only
one receive buffer, so don't send more than one message to a given
processor without reading the earlier ones.  To receive messages with
Send/Recv pairs EVERY processor must call MPI_Wait() the same number
of times or the machine will lock up.  This is because there are
sync's in there.  The receive must be separated from the matching send
by the MPI_Wait().

For Isend/Irecv: the MPI_Wait() call follows the pair, and the
limitation of a single receive is eliminated via tables.  Currently 4
receives/sends can be stacked on any given processor; to change this,
before including mpi.h, define MAXPENDING to be the number you need.

The only blocking functions are MPI_Wait, MPI_Reduce, MPI_Allreduce,
MPI_Bcast, MPI_Barrier, MPI_Init, and MPI_Finalize.

Not all the esoteric reductions have been adequately tested.

Principle files:
  mpi.h         copied from the MPICH distribution
  mpi.C         the primary source

The "include" directory contains

  binding.h     copied from the MPICH distribution
  mpi_errno.h   copied from the MPICH distribution
  mpidefs.h     copied from the MPICH distribution
  mpio.h        copied from the MPICH distribution
  qmemcpyoc.C   the underlying data moving routines
  dummy.h       dummy qos functions for compiling on linux

Functions implemented so far:

   int MPI_Init (int * argc, char *** argv);
   int MPI_Finalize ();
   int MPI_Comm_rank (MPI_Comm comm, int *rank);
   int MPI_Comm_size(MPI_Comm comm, int *size);
   int MPI_Get_processor_name( char * name, int * resultlen);
   int MPI_Barrier(MPI_Comm comm);
   int MPI_Abort(MPI_Comm comm, int errorcode);
   int MPI_Bcast(void* buffer, int count, MPI_Datatype datatype,
        int root, MPI_Comm comm); double MPI_Wtime();
   int MPI_Reduce(void * sendbuf, void * recvbuf, int count, 
        MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm);
   int MPI_Allreduce(void * sendbuf, void * recvbuf, int count, 
        MPI_Datatype datatype, MPI_Op op, MPI_Comm comm);
   int MPI_Recv( void * buf, int count, MPI_Datatype datatype, 
        int source, int tag, MPI_Comm comm, MPI_Status * status )
   int MPI_Send( void * buf, int count, MPI_Datatype datatype, 
        int dest, int tag, MPI_Comm comm )
   int MPI_Wait (MPI_Request * request, MPI_Status * status );
   int MPI_Isend( void * buf, int count, MPI_Datatype datatype, 
        int dest, int tag, MPI_Comm comm, MPI_Request * request )
   int MPI_Irecv( void * buf, int count, MPI_Datatype datatype, 
        int source, int tag, MPI_Comm comm, MPI_Request * request )
   int MPI_Issend( void * buf, int count, MPI_Datatype datatype, 
	int dest, int tag, MPI_Comm comm, MPI_Request * request )
   int MPI_Attr_get (MPI_Comm comm, int keyval, 
        void * attr_value, int *flag )

-----------------------------

Mike Creutz
creutz@bnl.gov