Communication Completion

The functions MPI_WAIT and MPI_TEST are used to complete a nonblocking communication. The completion of a send operation indicates that the sender is now free to update the locations in the send buffer (the send operation itself leaves the content of the send buffer unchanged). It does not indicate that the message has been received, rather, it may have been buffered by the communication subsystem. However, if a synchronous mode send was used, the completion of the send operation indicates that a matching receive was initiated, and that the message will eventually be received by this matching receive.

The completion of a receive operation indicates that the receive buffer contains the received message, the receiver is now free to access it, and that the status object is set. It does not indicate that the matching send operation has completed (but indicates, of course, that the send was initiated).

We shall use the following terminology: A null handle is a handle with value
MPI_REQUEST_NULL. A persistent request and the handle to it are inactive if the request is not associated with any ongoing communication (see Section Persistent communication requests ). A handle is active if it is neither null nor inactive. A empty status is a status which is set to return tag = MPI_ANY_TAG, source = MPI_ANY_SOURCE, and is also internally configured so that calls to MPI_GET_COUNT and MPI_GET_ELEMENTS return count = 0. We set a status variable to empty when the value returned by it is not significant. Status is set in this way so as to prevent errors due to accesses of stale information.

MPI_WAIT(request, status)
[ INOUT request] request (handle)
[ OUT status] status object (Status)

int MPI_Wait(MPI_Request *request, MPI_Status *status)

MPI_WAIT(REQUEST, STATUS, IERROR)
INTEGER REQUEST, STATUS(MPI_STATUS_SIZE), IERROR

A call to MPI_WAIT returns when the operation identified by request is complete. If the communication object associated with this request was created by a nonblocking send or receive call, then the object is deallocated by the call to MPI_WAIT and the request handle is set to MPI_REQUEST_NULL. MPI_WAIT is a non-local operation.

The call returns, in status, information on the completed operation. The content of the status object for a receive operation can be accessed as described in section Return status . The status object for a send operation may be queried by a call to MPI_TEST_CANCELLED (see Section Probe and Cancel ).

One is allowed to call MPI_WAIT with a null or inactive request argument. In this case the operation returns immediately with empty status.

[] Advice to users.

Successful return of MPI_WAIT after a MPI_IBSEND implies that the user send buffer can be reused --- i.e., data has been sent out or copied into a buffer attached with MPI_BUFFER_ATTACH. Note that, at this point, we can no longer cancel the send (see Sec. Probe and Cancel ). If a matching receive is never posted, then the buffer cannot be freed. This runs somewhat counter to the stated goal of MPI_CANCEL (always being able to free program space that was committed to the communication subsystem). ( End of advice to users.)

[] Advice to implementors.

In a multi-threaded environment, a call to MPI_WAIT should block only the calling thread, allowing the thread scheduler to schedule another thread for execution. ( End of advice to implementors.)
MPI_TEST(request, flag, status)
[ INOUT request] communication request (handle)
[ OUT flag] true if operation completed (logical)
[ OUT status] status object (Status)

int MPI_Test(MPI_Request *request, int *flag, MPI_Status *status)

MPI_TEST(REQUEST, FLAG, STATUS, IERROR)
LOGICAL FLAG
INTEGER REQUEST, STATUS(MPI_STATUS_SIZE), IERROR

A call to MPI_TEST returns flag = true if the operation identified by request is complete. In such a case, the status object is set to contain information on the completed operation; if the communication object was created by a nonblocking send or receive, then it is deallocated and the request handle is set to MPI_REQUEST_NULL. The call returns flag = false, otherwise. In this case, the value of the status object is undefined. MPI_TEST is a local operation.

The return status object for a receive operation carries information that can be accessed as described in section Return status . The status object for a send operation carries information that can be accessed by a call to MPI_TEST_CANCELLED (see Section Probe and Cancel ).

One is allowed to call MPI_TEST with a null or inactive request argument. In such a case the operation returns with flag = true and empty status.

The functions MPI_WAIT and MPI_TEST can be used to complete both sends and receives.

[] Advice to users.

The use of the nonblocking MPI_TEST call allows the user to schedule alternative activities within a single thread of execution. An event-driven thread scheduler can be emulated with periodic calls to MPI_TEST. ( End of advice to users.)

[] Rationale.

The function MPI_TEST returns with flag = true exactly in those situations where the function MPI_WAIT returns; both functions return in such case the same value in status. Thus, a blocking Wait can be easily replaced by a nonblocking Test. ( End of rationale.)

  
 
Simple usage of nonblocking operations and  MPI_WAIT. 
 
CALL MPI_COMM_RANK(comm, rank, ierr) 
IF(rank.EQ.0) THEN 
    CALL MPI_ISEND(a(1), 10, MPI_REAL, 1, tag, comm, request, ierr) 
    **** do some computation to mask latency **** 
    CALL MPI_WAIT(request, status, ierr) 
ELSE 
    CALL MPI_IRECV(a(1), 15, MPI_REAL, 0, tag, comm, request, ierr) 
    **** do some computation to mask latency **** 
    CALL MPI_WAIT(request, status, ierr) 
END IF 
 
 

MPI_REQUEST_FREE(request)
[ INOUT request] communication request (handle)

int MPI_Request_free(MPI_Request *request)

MPI_REQUEST_FREE(REQUEST, IERROR)
INTEGER REQUEST, IERROR

Mark the request object for deallocation and set request to MPI_REQUEST_NULL. An ongoing communication that is associated with the request will be allowed to complete. The request will be deallocated only after its completion.

[] Rationale.

The MPI_REQUEST_FREE mechanism is provided for reasons of performance and convenience on the sending side. ( End of rationale.)

[] Advice to users.

Once a request is freed by a call to MPI_REQUEST_FREE, it is not possible to check for the successful completion of the associated communication with calls to MPI_WAIT or MPI_TEST. Also, if an error occurs subsequently during the communication, an error code cannot be returned to the user --- such an error must be treated as fatal. Questions arise as to how one knows when the operations have completed when using MPI_REQUEST_FREE. Depending on the program logic, there may be other ways in which the program knows that certain operations have completed and this makes usage of MPI_REQUEST_FREE practical. For example, an active send request could be freed when the logic of the program is such that the receiver sends a reply to the message sent --- the arrival of the reply informs the sender that the send has completed and the send buffer can be reused. An active receive request should never be freed as the receiver will have no way to verify that the receive has completed and the receive buffer can be reused. ( End of advice to users.)

  
 
An example using  MPI_REQUEST_FREE. 
 
 
CALL MPI_COMM_RANK(MPI_COMM_WORLD, rank) 
IF(rank.EQ.0) THEN 
    DO i=1, n 
      CALL MPI_ISEND(outval, 1, MPI_REAL, 1, 0, req, ierr) 
      CALL MPI_REQUEST_FREE(req, ierr) 
      CALL MPI_IRECV(inval, 1, MPI_REAL, 1, 0, req, ierr) 
      CALL MPI_WAIT(req, status, ierr) 
    END DO 
ELSE    ! rank.EQ.1 
    CALL MPI_IRECV(inval, 1, MPI_REAL, 0, 0, req, ierr) 
    CALL MPI_WAIT(req, status) 
    DO I=1, n-1 
       CALL MPI_ISEND(outval, 1, MPI_REAL, 0, 0, req, ierr) 
       CALL MPI_REQUEST_FREE(req, ierr) 
       CALL MPI_IRECV(inval, 1, MPI_REAL, 0, 0, req, ierr) 
       CALL MPI_WAIT(req, status, ierr) 
    END DO 
    CALL MPI_ISEND(outval, 1, MPI_REAL, 0, 0, req, ierr) 
    CALL MPI_WAIT(req, status) 
END IF