We have assessed the costs of the different PVM routines in terms of memory-to-memory copy and network communication. By network communication, we mean one sending of one message over the network. This message is possibly fragmented by the system. These costs are given for one send-receive operation during our ``ping-pong" test, that is with the assumption that the receive is always posted when the send is done. By memory-to-memory copy, we mean the copy of one message from one local buffer to another local buffer. The number of memory-to-memory copies includes the extra system buffering.
The following table gives these costs
as illustrated in Figure 
.
The gap between pvm_send()-pvm_recv() and pvm_psend()-pvm_precv() seems surprising. To see how much of that can be attributed to the extra data copies, we measure the costs of memory-to-memory copies on all three systems, the results are shown in the following table. The time in the bcopy column is how long it takes to copy the message. The rest is the message roundtrip time for the three encoding methods, divided by two. The message size is 800 Kbytes, and the time is in microseconds.
For the CM5, the bcopy time accounted for most of the difference between pvm_send()-pvm_recv() and pvm_psend()-pvm_precv(). For the SP2 and Paragon, it accounted for most of the difference between pvm_psend()-pvm_precv() and pvm_send()-pvm_recv() PvmDataInPlace.