Subject: point-to-point proposal consistent with collective proposal

From: [email-masked] (Al Geist)
Date: Fri, 25 Dec 92 19:14:59 -0500

Point-to-point Communication Proposal
-------------------------------------
After reading Marc Snir's Message Passing Interface Outline
and the other point-to-point comments,
I would like to propose the following minimum set of point-to-point routines
for discussion at the January meeting in Dallas.

Stated Goals:
1. Design an application programming interface.

2. Design an interface that is not too different from current practice

3. Define an interface that can be quickly implemented on many vendor platforms.

4. Focus on a proposal that can be agreed upon in 6 months.

5. Provide a reliable communication interface.

6. Allow efficient communication.

7. Interface should not contain more functions than are really necessary.

-----------------------------------------------------------------------------
General Comments:

The collective communication interface should be an extension of the
point-to-point interface.
a. As Marc points out on page 2
   "SEND and RECV are a particular case of broadcast in a group of size 2;
   this observation can be used to check if the definition of collective
   communication semantics are consistent with the definition of
   point-to-point communication."

b. Collective routines like broadcast should provide the same
   message data format (contiguous, buffer with stride, etc.)
   as point-to-point routines.

c. By using a structured name space (described on pages 7-8 of [Snir])
   where all processes are identified by a (group,rank) pair
   for both the point-to-point and collective routines,
   then the users will have a consistent naming scheme
   across all the MPI communication routines.
   And those desiring a flat name space can have it by
   using the default group "ALL".

d. Syntax guidelines - short, mnemonic names;
                       systematic ordering of parameters;
                       systematic naming convention.
-------------------------------------------------------------------------------
Proposed point-to-point routines:

1. info = MPI_SEND( buf, bytes, type, gid, dest )

   Function:
   Sends a contiguous message of length "bytes" pointed to by "buf" to the
   process identified by ("gid","dest") with a message tag of "type".
   On return "info" contains the error code.

   Conceptually the user can view this function as
   "I want this message delivered to the specified process
	in the fastest reliable manner."

   Implementation comments:
   On return user can consider buf is safe for reuse.
   MPI_SEND need not be strictly blocking.
   Users do not want to have to dance around the send buffer
   with STAUS and WAIT checking for when the send buffer is safe to change.
   If a compiler can determine that (or rearrange code so that) 
   the send buffer is not touched before the send completes,
   then the implementation could be non-blocking.
   If a multiprocessor requires synchronous communication
   or routing along a particular topology, then this could
   be set in a global routine that affects all point-to-point
   and collective routines.

2. info = MPI_RECV( buf, bytes, type, gid, source )

   Function:
   Receive the message specified by "type", "gid", and "source"
   into "buf". The message must be <= "bytes" in size or "info" returns error.
   Routine waits until specified message has arrived [like blocking recv].
   On return "info" contains the error code.
   A minus one (-1) in "type", "gid", or "source" means don't care
   and that argument will match any message.

   Conceptually the user can view this function as
   "I must have the specified message before I can do anything else.
    I will wait until it arrives."

3. info = MPI_NRECV( buf, bytes, type, gid, source )

   Function:
   Checks to see if the specified message has arrived [like probe].
   A minus one (-1) in "type", "gid", or "source" means don't care
   and that argument will match any message.
   If the message has arrived it is returned in "buf" and "info" = 1.
   The message must be <= "bytes" in size or "info" returns error.
   If the message has not arrived, then the routine returns with "info" = 0.
   Values less than 0 returned by "info" indicate an error.
   MPI_NRECV can be called multiple times to check on the arrival of a message, 
   and the user can later call MPI_RECV to wait on the same message.

   Conceptually the user can view this function as
   "I would like to receive the specified message now,
    but if it is not here, I will not wait for it.
    I will go do something else."

   Implementation is not specified here, but for efficiency, implementors
   may wish to post a receive on the first call to MPI_NRECV and
   just do checks [like status] on repeat calls. If implemented like this,
   MPI_RECV must be able to check both the system queue and the posted queue.

4. type   = MPI_TYPE()
5. bytes  = MPI_BYTES()
6. gid    = MPI_GID()
7. source = MPI_SOURCE()

   Function:
   These routines return the actual values for the last message
   returned by MPI_RECV or MPI_NRECV. Values < 0 indicate an error.
   Allows the user to determine the size of the received message
   and the values of "don't care" arguments.

8. bytes = MPI_PACK(   data_type, nitems, stride, inbuf, outbuf )
9. bytes = MPI_UNPACK( data_type, nitems, stride, inbuf, outbuf )

   Function:
   General purpose routines for packing and unpacking an arbitrarily
   complex message buffer. Allows for non-contiguous messages,
   mixed data types in a single message, and heterogeneity between machines.
   Same routines can be used to build collective communication buffers,
   for example for MPI_BCAST.
   On return "outbuf" contains the (potentially) encoded data,
   and "bytes" is the size in bytes of the encoded data.
   "bytes" < 0 indicates an error.

   The following data types would be supported:
   BYTE   INTEGER*2   INTEGER*4   REAL*4   REAL*8   COMPLEX*8   COMPLEX*16
   I would also like to see the following data types considered:
   CHARACTER_STRING   INTEGER*8   REAL*16   COMPLEX*32

   "nitems" is the total number of items of this data type to pack 
   or unpack from "inbuf". "stride" can be used for packing 
   rows of a matrix in Fortran or columns of a matrix in C.

   MPI_PACK can be called  multiple times to construct a single message buffer.
   MPI_UNPACKs must be called by the user to decode such a buffer on receipt.
   Neither of these routines need be called if the user does not require
   non-contiguous messages, mixed data types, or heterogeneity.

   Having separate packing and unpacking routines
   allows send and receive to be optimized for a contiguous stream of bytes,
   and simplifies future extensions of MPI into areas such as heterogeneity.

===============================================================================
For completeness here is the present proposal from the
collective communication subcommittee.

1. info = MPI_BCAST( buf, bytes, type, gid, root )

   Function:
   Called by all members of the group "gid" 
   using the same arguments for "bytes", "type", "gid", and "root".
   On return the contents of "buf" on "root" is contained in "buf"
   on all group members.
   On return "info" contains the error code.

2. info = MPI_GATHER( buf, bytes, type, gid, root )

   Function:
   Called by all members of the group "gid" 
   using the same arguments for "bytes", "type", "gid", and "root".
   On return all the individual "buf" are concatenated into the "root" buf,
   which must be of size at least gsize*bytes where gsize is group size.
   The data is laid in the "root" buf in rank order that is
   | gid,0 data | gid,1 data | ...| gid, root data | ...| gid, gsize-1 data |
   Other member's "buf" are unchanged on return.
   On return "info" contains the error code.

3. info = MPI_GLOBAL_OP( inbuf, bytes, type, gid, op, outbuf )

   Function:
   Called by all members of the group "gid"
   using the same arguments for "bytes", "type", "gid", and "op".
   On return the "outbuf" of all group members contains the 
   result of the global operation "op" applied pointwise to
   the collective "inbuf". For example, if the op is max and
   inbuf contains two float point numbers then 
     outbuf(1) = global max( inbuf(1)) and 
     outbuf(2) = global max( inbuf(2)) 
   A set of standard operations are supplied with MPI including:
     global max - for each data type
     global min - for each data type
     global sum - for each data type
     global mult- for each data type
     global AND - for integer and logical type
     global OR  - for integer and logical type
     global XOR - for integer and logical type
   Optionally the users may define their own global functions for this routine.
   On return "info" contains the error code.

4. info = MPI_SYNCH( gid )

   Function:
   Called by all members of the group "gid"
   Returns only when all members have called this function.
   On return "info" contains the error code.

5. gid = MPI_MKGROUP( list_of_processes )

   Function:
   Called by all processes in the list.
   Forms a logical group containing the listed processes
   and assigns each process a unique rank in the group.
   The ranks are consecutively numbered from 0 to gsize-1.
   On return "gid" is an MPI assigned group ID (or error code if < 0)

6. gsize = MPI_GROUPSIZE( gid )

   Function:
   Can be called by any process.
   On return "gsize" is the number of members in the group "gid"
   (or error code if < 0).

7. rank = MPI_MYRANK( gid )

   Function:
   Can be called only by members of group "gid".
   On return "rank" is the rank of the calling process in group "gid"
   (an integer between 0 and gsize-1) or error code if < 0.

===========================================================================