Subject: buffered/unbuffered comms (was compiler target (was Be brave. Be sure.))

From: L J Clarke <[email-masked]>
Date: Mon, 8 Feb 93 13:52:17 GMT

Hi 

Here comes my 5p worth on the subject of buffered and unbuffered comms. 

For completeness: by unbuffered I mean that the sender blocks until the
message has been (or certainly will be) copied into the space of the
receiver, a la occam for example; by buffered comms I mean that the
message is copied away into a system buffer somewhere and the sender
returns, a la PVM for example. 

It seems to me that there are a few different issues in this subject
which discussions may have touched on. 

a) Ease of programming.  

I don't think anyone can really disagree that many programmers will
report that they find buffered comms easier to use.  

These users have been fortunate enough not to have come up against the
boundedness of buffering provided with the system.  With high
probability they are programming (or adapting) applications by inserting
message passing calls directly into the application source. 

b) Portability of programs using MPI between different machines.

[Here I give away my bias.]

It is very difficult to make statements about portability (and
reliability/correctness) of programs which use buffered messages and
rely on system storage capacity and structure thereof. 

This becomes particularly difficult when the application makes use of
substantive libraries, which themselves are written using buffered comms
and place requirements on system buffering.  It can just become to
difficult to work out how much buffering you need.  We played with a
system which allowed the programmer to configure the system buffering,
and this feature was only used (properly) by high priests. 

In my work I will only be able to use MPI if I can write substantive
libraries in the confidence that they will not be subject to failure due
to running out of such buffer space.  Therefore we use only unbuffered
message passing with a mixture of blocking/nonblocking (like Intel NX
isend/irecv/msgwait) calls. Therefore my bias :-)

c) Portability of existing applications (using existing message passing
interfaces) to MPI.

This is a different subject to (c).  Following the discussion it seems
that an argument in favour of adopting buffered comms is the number of
existing programs (eg, lotsa programs written using Express) which use
unbuffered comms. 

It has been my experience that migration of applications between
different, broadly similar, message passing interfaces is not so
difficult, except for this point.  In a nutshell, the surgery you have
to perform to move programs/libraries between an interface that forces
buffered comms and one which forces unbuffered comms and
blocking/nonblocking (isend/irecv/msgwait) is grevious and error prone. 

Given the above, I come to the conclusions:

i)  MPI must contain unbuffered communications with blocking/nonblocking
(irecv/isend/msgwait kinda thing) calls, for reliability and
portability. 

ii) If a goal of MPI is that existing applications using message passing
interfaces (eg Express, PVM) should easily port to MPI, then MPI must
also contain buffered comms. This seems to be a matter for the full
committee, hence I have crossposted.

Incidentally, to pitch in 2ps on some other lines of discussion:

* I can see no particular benefit in allowing a buffered snd/rcv match
an unbuffered rcv/snd.  I can see considerable inconvenience in
forbidding a blocking send/recv to match with a nonblocking recv/send. 

* Yes, we do try to overlap communications with calculation.  (Sometimes
it doesnt buy you, sometimes it does.  Sometimes we have specifically
needed to avoid such overlap in order to maximise performance, but that
was a weird one :-) Just as important, we frequently "overlap"
communication with communication (ie, use nonblocking calls) to avoid
deadlock. 

* Please, oh please, let MPI decide that communications should be
adressed using a rank relative to a group/context (0 ...  GroupSize - 1)
or (1 ...  GroupSize).  We do this all the time, and its very very
convenient.  In fact, when we can't do this we end up having to create
arrays of the task identifiers - we end up doing it ourselves anyway. 

Best Wishes
Lyndon

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/