Sending a message requires three steps. First, a PVM buffer must be initialized by a call to pvm_initsend(). Second, the message must be ``packed" from the user data space into the PVM buffer by using any combination of the pvm_pk*() routines. PVM takes care of any data encoding and fragmentation. Third, the complete message is sent to another process with pvm_send().
Receiving a message involves two steps. First, the incoming message must be accepted by pvm_recv(), the blocking receive, or by pvm_nrecv(), the non-blocking receive. Second, once the message has arrived, it must be ``unpacked" into the user data space with a combination of the pvm_upk*() functions.
During the initialization of the PVM buffer,
the user can chose between three different ways of packing
the data in this buffer, depending on the parameter passed to pvm_initsend().
The default packing mode is PvmDataDefault. The data is packed
from the user space into the PVM-buffer and is encoded according to the XDR format.
This mode allows communication over a heterogeneous network (by heterogeneous, we mean a set
of computers at least two of which do not have the same data format).
A second mode is PvmDataRaw.
This mode is similar to the default, but the encoding step is skipped.
Thus, PvmDataRaw can be used only between hosts of compatible data formats. It is always
more efficient to use PvmDataRaw when running PVM on a single MPP. In the experiments described in
this paper, we used PvmDataDefault once on the CM-5, only to show that it is always
highly inefficient to use this data format (see Figure 
(b)).
A third option, PvmDataInPlace, leaves the data ``in place" in the user data space.
During the packing step, PVM simply keeps track of where and how much data is specified.
When pvm_send() is called, the data is fetched from the user space and
sent over the network (the data is in fact never packed).
Using PvmDataInPlace reduces the pack time dramatically and reduces memory requirements.
However,
care must be taken when using this method as the data
should not be modified between the pack call and
the send call. Indeed, since PVM keeps only pointers to the data, the
data can be modified
any time before the send.
This situation cannot occur
with PvmDataDefault or PvmDataInPlace.