Subject: various

From: L J Clarke <[email-masked]>
Date: Mon, 15 Feb 93 11:32:26 GMT

Dear colleagues

I circulated the documents prepared by Marc, and Bill & Rusty, around
interested people locally, for discussion.  Here are some resulting
considerations of the point-to-point component of MPI. 

a) Synchronous messages and blocking/nonblocking communication calls:
The properties of end to end synchronisation, and blocking/nonblocking
communication calls are "orthogonal" in the sense described by Bill and
Rusty.  Thus, another "axis" in the procedure naming (section 3.3)
should be added, which perhaps is either nothing or 's'.  The approach
taken by Marc is a little, different and appears to implicitly recognise
this. 

This doesnt seem (to me) to fit in with the framework which Marc uses,
but it might be appropriate to consider synchronous and ready-receiver
as points on the same axis.  They both refer to the knowledge which the
sender has about receipt of the message.  In the synchronous case, the
sender absolutely knows that the receiver has got the message; in the
ready-receiver case the sender (presumably) has no knowledge of whether
the receiver has or has not got the message.  This approach would also
remove the curious "synchronous ready receiver" from Marc's document. 

b) Cancellation of nonblocking communications: This subject has appeared
before.  We strongly support the points made by Jim Cownie regarding the
requirement for cancellation of nonblocking communications.  

An example of its usefullness arises when writing libraries
encapsulating common communication patterns, we view it as desirable
that the communication call should be able to return with an error code
if something is wrong.  A routine XXX may not be able to detect this
until after some nonblocking communications have been started, and some
of those which have been started have also been completed and the
information therein examined.  At this point it becomes known that the
operation cannot succeed, however there are these nonblockinf
communications outstanding.  It may not be possible to wait for them to
complete, as they may never complete since the matching operation may
never be started, so th natural thing to do is cancel the outstanding
communications and return cleanly with an error code.  The importance of
the clean return is apparent when one considers the possibility that the
memory into which pending nonblocking receives may write (or may not,
unknown) is a local variable of the caller of XXX, which will itself
then wish to return to its caller, which will in turn call some user
error procedure.  In this case the stack can become corrupted by data
from the uncancelled nonblocking receive started in XXX and the whole
program barfs up.  Not nice. 

c) Matching of buffer structures: The proposals contain three provisions
for buffer structures: contiguous, strided and iovec (ignoring
heterogeneous issues for the moment, see below).  Each of these should
be defined to provide a byte stream (which may not map to a contiguous
set of bytes in memory), and the programmer should thus be allowed to
mix'n'match sends and receives of different types freely. This provides 
a handy way of performing scatter and gather type operations.

d) Heterogeneous issues and data conversion: Provision of data type
conversion in MPI is gaining support.  This makes sense to me, since a
lot of PVM users are (presumably) taking advantage of such provisions in
PVM. 

The proposal of Bill and Rusty contains a set of routines which provide
data type conversion.  Unfortunately the capabilities provided are
weaker than PVM, since they allow data conversion in messages of the
form "array of datatype" only. 

This raises the issue of user defined data types in C/Fortran90 (of
course there is no issue in Fortran77). At least with the PVM interface
the programmer can portably communicate such data types, albeit with a
little effort and overhead. 

e) Completion/checking functions: In sections 1.5.5 and 1.5.6 of Marc's
document, the START and COMPLETE operations are described.  I cannot
agree that there should be seperate MPI_SEND_XXX and MPI_RECV_XXX
routines (XXX is one of {START,COMPLETE,CHECK}).  

The handle contains all of the information regarding the kind of
operation which has been started, so it is not necessary to have
different routines.  It can be most convenient to be able to wait for a
mixture of sends and receives to complete, more specifically to wait for
any of such a mixture.  In order to do this, you must have an interface
which allows send and receive handles to be passed to the same wait
routine.  Another advatage, perhaps, is that htis wait function does not
have to raise an error if the wrong kind of handle is supplied, although
this advantage is minimal since it will have to check validity of
handles in a more general sense.  There has been an argument of
readability for having seperate routines for send and receive.  We could
resolve this by defining MPI_SEND_XXX and MPI_RECV_XXX to be wrappers
around MPI_XXX, which check that all handles are of the appropriate
nature. 

Best Wishes
Lyndon

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/

---

From: [email-masked] (William Gropp)
Date: Mon, 15 Feb 93 09:08:24 CST

X-Resent-To: [email-masked] ; Mon, 15 Feb 1993 06:32:39 EST
   Date: Mon, 15 Feb 93 11:32:26 GMT
   From: L J Clarke <[email-masked]>
   Reply-To: [email-masked]

... 
Just one clarification:
...

Actually, our proposal is somewhat stronger than this, at least at the
"expert" level (level 1): The user can provide an array of structures,
each of which is essentially an "iovec" (that is, pointer and length)
with an additional field for the datatype.  Thus, a message consisting of
a pair of integers and an array of doubles could be sent as (please
excuse the syntax; this can be cleaned up in any number of ways):
   mvec[0].len = 2 * sizeof(int);
   mvec[0].ptr = &i[0];
   mvec[0].datatype = MPI_INT;
   mvec[1].len = 347 * sizeof(double);
   mvec[1].ptr = &array[0];
   mvec[1].datatype = MPI_DOUBLE;
Alternate forms of this structure can basically define a "channel program"
that allows for things like non-unit-stride access and scatter/gathers.
Thus, just as with PVM, a single message may portably contain multiple
data types; the difference is in when the data is presented to the
interface (one datatype at a time, as in PVM, or all at once, as in our
proposal).  

This is our intent as well.  Note that it would be easy to write a tool
that would take a structure/derived type definition and generate the
appropriate setup/calls in either the PVM case or our proposal (A compiler
could do it to, but I suspect that initially many would be happy not
to have the compiler tinkered with, particularly since such enhancements
would not be portable).

Bill and Rusty