Subject: Proposal for explicit datatype packing and unpacking

From: "Nathan E. Doss" <[email-masked]>
Date: Thu, 20 Jan 1994 10:37:54 -0600 (CST)

Dear All,

  The remainder of this message proposes a small addition to the 
  the datatype section of the point-to-point chapter.

Nathan, Tony, Puri, and Kishore

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Proposal for explicit packing and unpacking of datatypes
--------------------------------------------------------

Current Status

   MPI does not provide for the explicit "packing" and "unpacking" of
messages.  MPI provides an interface where the user can say what needs
to be sent or received, but does not have to specify how it is packed
or unpacked.  This is a good thing, but we lacks the ability for the
user to specify ``when'' a message should be packed or unpacked (i.e.
data conversions and scatter/gather).  Currently, MPI must gather the
data every time a send is performed and scatter the data on every
receive (and in heterogeneous environments, perform any neccessary
data conversions on every send and receive).

   MPI provides the ability to pack by sending a message to yourself
and receiving it using MPI_BYTE, however, in heterogeneous
environments, the MPI_TYPE_SIZE function may not give the correct size
of a message receieved from a remote node (in MPI_BYTE format) since
the size is given in terms of the local data representation.  As an
example, assume we have a datatype that consists of every other
integer from an array of 20 integers and wish to send this information
to some number of other processors.  We want to avoid doing any 
unnecessary data conversions and gathering.  The following code
should work in homogeneous environments but is not guaranteed to 
work in heterogeneous environments:

   int          a[20];
   MPI_Datatype type;
   int          size;
   int          rank;
   char        *buffer;
   MPI_Status   status;

   MPI_Comm_rank   ( MPI_COMM_WORLD, &rank );
   MPI_Type_vector ( 10, 1, 2, MPI_INT, &type );
   MPI_Type_size   ( type, &size );
   buffer          = (char *) malloc ( size );
   MPI_Sendrecv    ( a,      1,    type,     rank, 0, 
                     buffer, size, MPI_BYTE, rank, 0,
                     MPI_COMM_WORLD, &status );

   for (each dest)
      MPI_Send     ( buffer, size, MPI_BYTE, dest, 0, MPI_COMM_WORLD );

This does not necessarily work since the size returned by
MPI_TYPE_SIZE may not be big enough to hold the datatype elements
after data conversions have been applied to it.  Another problem
with this example is that in implementations that check to see that
the sent type map is equivalent to the received type map, the MPI_SENDRECV
call would generate an error or warning (depending on the implementation)
that the types do not match.

   In the latest point to point draft (dated Jan 4, 1994), an MPI_MOVE
function is proposed which would be used in place of the MPI_SENDRECV
call used in the previous example.  The draft states that this
function is "[e]ssentially equivalent to MPI_SENDRECV with source =
dest, but could be defined to be lenient w.r.t. type matching.  It
could then be used for packing (with totype = MPI_BYTE) or unpacking
(with fromtype = MPI_BYTE)."  This does remove the problem caused by
MPI_SENDRECV performing type checking, however MPI_MOVE as proposed
could not take the place of MPI_SENDRECV in a heterogeneous
implementation.  MPI_MOVE as proposed appears to only convert between
types as represented on the local machine.  This function may be
convenient for converting/copying locally, but would not be useful for
packing/unpacking in a heterogeneous environment.

Proposal

   We propose a function that gives the size of the "packed" data in
relationship to a communicator, MPI_TYPE_COMM_SIZE.  In a
heterogeneous implementation, the decision to apply data conversions
to messages may depend on the members of the communicator.  For
example, one communicator may contain only homogeneous nodes with no
conversions necessary while another communicator may consist of
heterogeneous nodes.  The size of the packed data in these two cases
may be different if the determination of whether or not to pack data
is based on the type of nodes in a communicator.  This allows the user
to allocate a contiguous buffer into which a message may be received
with no data conversions.

MPI_TYPE_COMM_SIZE ( datatype, comm, size )
IN  datatype               datatype (handle)
IN  comm                   communicator (handle)
OUT size                   type size after conversions (integer)

      The following MPI_TYPE_PACK and MPI_TYPE_UNPACK functions can be
used to pack (unpack) buffers with respect to the given communicator.
Like the proposed MPI_MOVE, these functions are lenient with respect to
type checking.  Unlike MPI_MOVE, they are specifically for packing and
unpacking messages and perform the necessary data conversions.

MPI_TYPE_PACK ( comm, oldbuffer, count, oldtype, newbuffer)
IN  comm                   communicator (handle)
IN  oldbuffer              buffer to convert from 
IN  count                  count of old buffer items
IN  oldtype                MPI type of oldbuffer
OUT newbuffer              packed buffer after gather & data conversions

MPI_TYPE_UNPACK ( comm, oldbuffer, newbuffer, count, newtype)
IN  comm                   communicator (handle)
IN  oldbuffer              packed buffer
IN  newbuffer              unpacked buffer after scatter & data conversions
IN  count                  count of buffer items
IN  newtype                MPI type of newbuffer

The messages to be packed or unpacked are assumed to be of type MPI_BYTE.

   The following illustrates how the previously given example can be
written using these proposed functions:

   int          a[20];
   MPI_Datatype type;
   int          size;
   int          rank;
   char        *buffer;
   MPI_Status   status;

   MPI_Comm_rank      ( MPI_COMM_WORLD, &rank );
   MPI_Type_vector    ( 10, 1, 2, MPI_INT, &type );
   MPI_Type_comm_size ( type, MPI_COMM_WORLD, &size );
   buffer             = (char *) malloc ( size );
   MPI_Type_pack ( MPI_COMM_WORLD, a, 1, type, buffer ); 

   for (each dest)
      MPI_Send   ( buffer, size, MPI_BYTE, dest, 0, MPI_COMM_WORLD );

If type matching is performed, then the a type map consisting entirely
of MPI_BYTEs must be used by each receive.

Note
   The draft may want to specify that if type matching is done, that
the MPI_BYTE type can match any other type.  If this is so, then the
argument for MPI_TYPE_PACK and MPI_TYPE_UNPACK is weakened.

Advantages 
   o These functions provide a clear way for programmers to
     separate the gather (scatter) and data conversion operations
     from the communication operation.  For example, when forwarding
     messages, tere is no need to unpack (scatter and apply data 
     conversions) and then repack (gather and apply data 
     conversions).

   o There is no way in MPI to find the size of a datatype
     after data conversions are applied.  MPI_TYPE_COMM_SIZE
     provides this.

   o By providing MPI_TYPE_PACK and MPI_TYPE_UNPACK, MPI can
     provide an explicit means for users to pack and unpack
     buffers in a heterogeneously clean manner.

Disadvantages
   o On some platforms it may be more efficient to do the gather
     repetitively than to pack and then perform multiple sends.
     Thus, these use of these functions may decrease rather than
     increase 

   o The MPI_TYPE_PACK and MPI_TYPE_UNPACK functions are not
     strictly necessary since MPI_SENDRECV can be used in a
     similar fashion.