Subject: receiver-ready send, performance data

From: [email-masked]
Date: Thu, 8 Apr 93 16:25:00 PDT

Steve Lederman writes:

> Several general conclusions can be drawn:
> 
> 1) forced type is faster than regular once the message is several
> thousand bytes long.  The times are about 17% more for regular than
> forced for long messages.  This is as advertised (remember that only
> one of the two messages was forced type).
> 
> 2) Doing the irecv/wait is the same or faster than the "regular" send
> semantics even without doing a forced type send.  This is the opposite
> of what was probably expected.

I think it's a bit optimistic to say that these conclusions are
"general".

The situation that Steve outlines is highly asymmetric -- process 0
does basically all the receiving while 1 does the sending.

I've run literally hundreds of tests under symmetric conditions, both
two-party and three-party transfers (i.e., exchange and shift), and
I get results that are different from Steve's.

I agree that forced types have higher bandwidth.  However, the amount
of the increase depends on whether you're doing two- or three-party
transfers, as well as what machine you're on.  On the Delta, 20% is a
typical number; in some situations it's closer to 2X.  On the
iPSC/860, things are even more confusing -- overall, using forced
types can be about the same or 2X faster than unforced, or something
in between on average, and it's not even deterministic.  Oh, and on
the iPSC/860, breakeven for forced versus unforced is at 100 bytes
even with the extra sync, versus several thousand bytes on the Delta.

(A quick summary and discussion of my data is available in 
"Characterising and Tuning Communications Performance on the
Touchstone DELTA and iPSC/860 (extended abstract)", available via
anonymous ftp from delilah.ccsf.caltech.edu as file
delta/documents/PNL_papers/IUGpaper.ps)

I disagree with Steve's conclusion about irecv/wait being the same or
faster than crecv.  While that's usually true in his tests (but not
always -- see bytes=100), I have consistently found the opposite.

Steve continues:

> ... If
> you add a synchronization to achieve ready receive then it could
> clearly slow your code down.
> 
> ... There is no penalty
> (on the Delta) for solely using ready receive semantics.

These two statements seem contradictory.  In the general case, you
do have to add synchronization to use receiver-ready semantics, and
that can slow you down if the messages are short.  There ain't no
such thing as a free lunch.

My specific conclusion about timing is that

  On Intel systems, long messages are faster if forced and
  short messages aren't, and if you really care, your code
  had better be prepared to adapt.

My general conclusion about timing is that

  General conclusions about timing are really dangerous to make.

--Rik
----------------------------------------------------------------------
[email-masked] (alias 'd39135')   Rik Littlefield
Tel: 509-375-3927                         Pacific Northwest Lab, MS K1-87
Fax: 509-375-6631                         P.O.Box 999, Richland, WA  99352