Subject: Multifarious

From: [email-masked] (James Cownie)
Date: Mon, 4 Jan 93 13:49:35 GMT

Gentlepeople,

Firstly a general point
=======================

I'd like to add an additional criterion to the goals of the message
passing interface. 

8) The implementation and semantics of the interface should remain
   simple and "obvious" when multiple "lightweight" threads share the
   same address space.

I'd like this because 

1) Lightweight thread libraries on the Posix model are becoming
   available on most workstations (networks of which are one of our
   targets).

2) There is a clear industry trend to having a few processors on a
   logically (and often physically) shared memory both in workstations
   (e.g. SparcStation 10), and servers (SparcServer 690, RS6000 ??).
   It would be foolish to go down a path which made exploitation of
   such machines hard. Undoubtedly this trend will also occur in
   future MPP machines (it would "only" take a software change to the
   code on the Paragon to make it be like this. After all it has all
   the hardware there already).

As a consequence of this I'd like to remove as much hidden state as
possible from the message passing system, so

1) I dislike the functions which return information about "the last
   message", since in a single process with multiple threads there can
   be many "last" messages. The suggestion that these operations be
   performed on descriptors returned from the probe is much preferable
   (and semantically cleaner).

2) I dislike the idea of setting options in a global routine, because
   there may be other threads performing operations at the same time
   as the options are changing, and the semantics becomes very hard to
   pin down.

Secondly A few comments on Al Geist's proposal of Christmas Day 
===============================================================

1. MPI_SEND
I liked the separation in the original draft between the concepts of
blocking (the buffer is now free) and synchronous (the message has
been received at the other end) communications. I'd like to preserve
both possibilities.

3. MPI_NRECV
I don't understand how this is intended to work. Consider the 
following code.

	  char buffer[100];
	  int i;

	  for (i=0; i<10; i++)
	  {
	      if (&MPI_NRECV(buffer[10*i],10,1,1,1))
	         break;
	  }

Questions :
 1) Into which elements of buffer will the receive occur ?
    Al's implementation note suggests just the first, so what then do
    the other (up to) 9 calls achieve ? Will they receive the next
    nine matching messages ? (this doesn't appear very intuitive for a
    "polling" routine)

 2) Should I now do a blocking MPI_RECV as well ? Or does this only
    make sense if the MPI_NRECVs all fail ? But if they do fail, is
    the receive buffer still queued to receive a matching message when
    it becomes available ? (and if so how do I cancel the outstanding
    receive and get back ownership of my buffer ?)

4,5,6,7
    See comments above about this style of working.

8,9
    If we're only having one form of packing, then it should probably
    be the more general one with arrays of pointers and counts, since
    it can subsume (admittedly at some cost) the strided style.

Conclusion
==========

I prefer the original draft to Al's version. I'd like to change the
mode argument in the original draft (see my mail to mpi-lang). 

Removing internal state
=======================

I'd also like to remove the hidden state ("last message"), there are
various ways of achieving this, none of which is exceptionally
attractive.

1) Return a structure from MPI_RECV (and friends)
   This is fine in C, but not good in Fortran 77 (and may be more
   expensive than we want when you don't need any of the information).

   e.g.
   typedef struct MPI_REPLY
   {
      int status;
      int nbytes;
      int whoFrom;
      int tag;
      /* More ? */
   } MPI_REPLY; 

   { 
      MPI_REPLY res;

      res = MPI_RECV(...);

      if (res.whoFrom == 0) 

      ... etc ...
   }

2) Return results indirect through individual parameters (arguments for the
   Fortran inclined)

   This works, but is very error prone, particularly in Fortran when
   the arguments are INOUT. (We have an interface like this at the
   moment, and it does cause problems).

3) Return a tag on which queries can be made.

   This means that you must then have another call to the system to
   dispose of the tag (and associated system storage). This is
   unpleasant on a simple blocking receive when you don't want any of
   the information (since it's another call to the library).

4) Have additional (possibly optional) OUT arguments through which the
   results are returned if required.

   Using optional keyword arguments in F90 this works fine.
   C can be made to work more or less.
   F77 can't (within the standard).

5) Have one required argument which is a pointer to a reply structure
   like that used in 1) which is then filled in by the function.

   This can work in all the languages.
   In C you could also allow NULL to mean no such results are
   required.

IMHO Solution 5 is best. 

If we wanted to (and we may) then we could disallow direct access to
this structure and provide functions to pick it apart (these could be
macros in C, inline functions in C++). This would avoid having to
specify the contents of the structure and allow for future expansion.

Comments on Steve Lederman's note
=================================
> At the meeting in Minneapolis, several people, including myself,
> discussed the advantage of having a send where the sending process
> knows the receiving process has already posted a receive.  

We have this in our current message passing model. It has proved to be
an awful mistake. 98% of user's don't know when to use it, and then
complain that their code no longer works when they've used it
incorrectly. Please let's not do this. (I'm speaking from experience
here).

See you all in Dallas.

-- Jim
James Cownie 
Meiko Limited
650 Aztec West
Bristol BS12 4SD
England

Phone : +44 454 616171
FAX   : +44 454 618188
E-Mail: [email-masked] or [email-masked]