This section illustrates the design of parallel libraries, and the use of communicators to ensure the safety of internal library communications.
Assume that a new parallel library function is needed that is similar to the MPI broadcast function, except that it is not required that all processes provide the rank of the root process. Instead of the root argument of MPI_BCAST, the function takes a Boolean flag input that is true if the calling process is the root, false, otherwise. To simplify the example we make another assumption: namely that the datatype of the send buffer is identical to the datatype of the receive buffer, so that only one datatype argument is needed. A possible code for such a modified broadcast function is shown below.
A (correct) execution of this code is illustrated in
Figure 
, with arrows used to indicate communications.
Figure: Correct invocation of mcast
Since the invocation of mcast at the three processes is not
simultaneous, it may actually happen that mcast is invoked at process
0 before process 1 executed the receive in the caller code.
This receive, rather than being matched by the caller code send at
process 2, might be matched by
the first send of process 0 within mcast. The erroneous execution
illustrated in Figure results.
Figure: Erroneous invocation of mcast
How can such erroneous execution be prevented? One option is to enforce synchronization at the entry to mcast, and, for symmetric reasons, at the exit from mcast. E.g., the first and last executable statements within the code of mcast would be a call to MPI_Barrier(comm). This, however, introduces two superfluous synchronizations that will slow down execution. Furthermore, this synchronization works only if the caller code obeys the convention that messages sent before a collective invocation should also be received at their destination before the matching invocation. Consider an invocation to mcast() in a context that does not obey this restriction, as shown in the example below.
The desired execution of the code in this example is illustrated
in Figure .
Figure: Correct invocation of mcast
However, a more likely matching of sends with receives will lead to the
erroneous execution is illustrated in
Figure .
Figure: Erroneous invocation of mcast
Erroneous results may also occur if a process that is not in the group of comm and does not participate in the collective invocation of mcast sends a message to processes one or two in the group of comm.
A more robust solution to this problem is to use a distinct communication domain for communication within the library, which is not used by the caller code. This will ensure that messages sent by the library are not received outside the library, and vice-versa. The modified code of the function mcast is shown below.
This code suffers the penalty of one communicator allocation and deallocation at
each invocation. We show in the next section, in
Example ,
how to avoid this overhead, by using a preallocated communicator.