Collective Communications

Collective communications transmit data among all the processes specified by a communicator object. One function, the barrier, serves to synchronize processes without passing data. Briefly, MPI provides the following collective communication functions.

• barrier synchronization across all processes
• broadcast from one process to all
• gather data from all to one
• scatter data from one to all
• allgather: like a gather, followed by a broadcast of the gather output
• alltoall: like a set of gathers in which each process receives a distinct result
• global reduction operations such as sum, max, min, and user-defined functions
• scan (or prefix) across processes

The syntax and semantics of the MPI collective functions was designed to be consistent with point to point communications. However, to keep the number of functions and their argument lists to a reasonable level of complexity, the MPI committee made collective functions more restrictive than the point to point functions, in several ways. One restriction is that, in contrast to point to point communication, the amount of data sent must exactly match the amount of data specified by the receiver. This was done to avoid the need for an array of status variables as an argument to the functions, which would otherwise be necessary for the receiver to discover the amount of data actually received.

A major simplification is that collective functions come in blocking versions only. Though a standing joke at committee meetings concerned the ``non-blocking barrier,'' such functions can be quite useful and may be included in a future version of MPI.

A final simplification of collective functions concerns modes. Collective functions come in only one mode, and this mode may be regarded as analogous to the standard mode of point to point. Specifically, the semantics are as follows. A collective function (on a given process) can return as soon as its participation in the overall communication is complete. As usual, the completion indicates that the caller is now free to access and modify locations in the communication buffer. It does not indicate that other processes have completed, or even started, the operation. Thus, a collective communication may, or may not, have the effect of synchronizing all calling processes. The barrier, of course, is the exception to this statement.

The choice of semantics was done so as to allow a variety of implementations.

The user of MPI must keep these issues in mind. For example, even though a particular implementation of MPI may provide a broadcast with the side-effect of synchronization (the standard allows this), the standard does not require this, and hence, any program that relies on the synchronization will be non-portable. On the other hand, a correct and portable program must allow a collective function to be synchronizing. Though one should not rely on synchronization side-effects, one must program so as to allow for it.

Though these issues and statements may seem unusually obscure, they are merely a consequence of the desire of MPI to:

• allow efficient implementations on a variety of architectures; and,
• be clear about exactly what is, and what is not, guaranteed by the standard.