Subject: ready receive

From: [email-masked] (Steve Huss-Lederman)
Date: Thu, 8 Apr 93 14:53:37 EDT

I am glad to see that ready receive is being discussed.  I agree with
the comments that if a lot of machines don't implement a separate
ready receive then there is the chance that users will write programs
assuming it is there and then possible get slower results.  I can see
this especially happening if ones adds synchronizations that would not
otherwise be present.  Lyndon raised the issue of the extra expense in
the example of fan-in/fan-out if one uses forced type semantics but
only gets a regular receive.  I wrote a program to test this on the
Delta since one can get ready receive (forced type) on that machine.

All tests had these things in common.  They did 10 trials to reduce
the noise.  The program was run twice to make sure it was
reproducible.  I gsync() before each trial and test to make sure each
node starts at about the same time.  After that I also send another
long blocking message to really make sure they started about the same
time.  I think the code is correct :-)

There were only 2 nodes to keep the case simple and clean.  Node 0 can
be thought of as the leaf and node 1 as the parent in Lyndon's
example.  Node 0 sends its information to node 1 who then returns it
to node 0.  With more nodes, node 1 would pass the data on and receive
it again before it sends it back to node 0.  The difference in the 3
cases is whether you use ready receive semantics and if you actually
do the special type send.  For the pseudo code below irecv is
non-blocking, crecv is blocking and csend is blocking.

forced (ready receive):

node 0			node 1
------			------
irecv (tag=2)		crecv(tag=1)
csend(tag=1, reg)	csend(tag=2, forced)
wait(tag=2)

regular:

node 0			node 1
------			------
csend(tag=1, reg)	crecv(tag=1)
crecv (tag=2)		csend(tag=2, regular)

forced/regular (ready receive semantics without the special send):

node 0			node 1
------			------
irecv (tag=2)		crecv(tag=1)
csend(tag=1, reg)	csend(tag=2, regular)
wait(tag=2)

The times in seconds for various message sizes is:

node    # bytes forced          regular         reg/forced
1       100     2.946200e-04    2.283100e-04    2.955100e-04
0       100     4.382000e-04    3.820900e-04    4.548500e-04

1       500     4.092900e-04    4.174500e-04    4.086200e-04
0       500     6.818600e-04    6.904600e-04    6.711400e-04

1       2500    8.961900e-04    8.948600e-04    8.995300e-04
0       2500    1.155420e-03    1.288320e-03    1.223410e-03

1       12500   3.110890e-03    3.712720e-03    3.702740e-03
0       12500   3.494100e-03    4.035370e-03    3.953160e-03

1       62500   1.482880e-02    1.787127e-02    1.755866e-02
0       62500   1.542180e-02    1.819615e-02    1.786163e-02

1       312500  7.446586e-02    8.802909e-02    8.647874e-02
0       312500  7.508341e-02    8.834550e-02    8.674206e-02

1       1562500 3.713712e-01    4.369691e-01    4.320925e-01
0       1562500 3.719815e-01    4.372853e-01    4.323565e-01

I note a few general things about these results.  Node 0 is generally
slower because it is receiving the second message.  There is a greater
difference in time between the two nodes for smaller messages.

Several general conclusions can be drawn:

1) forced type is faster than regular once the message is several
thousand bytes long.  The times are about 17% more for regular than
forced for long messages.  This is as advertised (remember that only
one of the two messages was forced type).

2) Doing the irecv/wait is the same or faster than the "regular" send
semantics even without doing a forced type send.  This is the opposite
of what was probably expected.

With the caveat that these results are for the Delta, it seems that
ready receive (forced type) semantics are better in general.  Thus, if
you don't have to add synchronization to your code and you are sending
messages of sufficient length, then using ready receive is a win.  If
you add a synchronization to achieve ready receive then it could
clearly slow your code down.

I hope this answers the question Lyndon raised.  There is no penalty
(on the Delta) for solely using ready receive semantics.  I clearly
read all of his messages carefully and even spent several hours to
answer one of his many questions. I wonder if I really get the beer :-)