Subject: timeouts - a proposal

From: [email-masked] (Danny Nessett)
Date: Tue, 6 Apr 93 08:04:05 PDT

The Problem
-----------

There are a number of situations in which it is desirable/necessary to timeout
communications supported by MPI. For example, if a process wishes to test for
non-MPI I/O completion, such as non-blocking file I/O, keyboard/mouse
interrupts or other asynchronous events, it cannot afford to block inside an
MPI routine forever.  Another example is when a long job (say an atmospheric
code that runs for weeks), is submitted via batch and encounters either a
hardware or software error. There will be no one who will be monitoring the
progress of this job to terminate it. If there is no programming facility
within MPI to detect such errors, it is conceivable that such a job could
continue "running" consuming memory and perhaps CPU resources for many hours
before corrective action is taken.

In regards to the last problem, several people I have talked to have commented
that it is not the job of MPI to detect and recover from either hardware
errors or erroneous programs. I believe this is acceptable if we are designing
a prototype or an academic toy, but in real production environments, such an
approach will lead to much less acceptance of MPI as a standard programming
interface.

The Issues As I See Them
------------------------

There seems to be several issues in regards to timeout that are important. If
I have missed some, I encourage others to point them out. The issues I see are:

  o  People are reluctant to add a timeout parameter to every MPI call that
     could possibly block. Not only does it complicate the interface, it also
     requires those programmers who are not interested in timeout to think
     about it.

  o  People want MPI to conform to standard practice. Current practice in
     the homogeneous MPP (NX, CMMD, etc) and workstation cluster world (PVM,
     P4, etc) is not to provide a timeout or to time out communications in the
     supporting library and crash. Current practice in the networking world
     (say sockets) is to provide for a timeout in the programming library.
     Consequently, there is a collision of approaches in current practice.

  o  Some MPP manufacturers may find it difficult to provide 
     timeouts. Either there is no hardware support in the machine's message
     passing mechanism or it is not easily used to provide the necessary
     high-performance message passing semantics.

  o  Timeouts require the programmer to catch timeout errors in if statements
     or other conditional statements and provide the appropriate exception
     handling. Scientific programmers have a habit of not coding for errors.

  o  Timeouts move the responsibility for failure atomicity to the application
     programmer. The underlying MPI library implementation cannot simply crash
     all of the parallel executables when a timeout occurs (at least in all
     cases).

A Proposal
----------

Given the above issues, consider the following proposed change to the MPI
interface to support timeouts. To each of the functions MPI_WAIT, MPI_WAITANY,
and MPI_WAITALL add a timeout argument. This argument should be a double
value representing the number of seconds to wait before timing out the WAIT
call. If the conditions for the WAIT to return are not satisfied before the
timeout occurs, the function returns a return value indicating timeout.

In detail the following changes are proposed :

MPI_WAIT( handle, return_status_handle, timeout)

IN handle communication handle

OUT return_handle handle that is associated with return status object.

IN timeout timeout value (double)

MPI_WAITANY( list_of_handles, index, return_handle, timeout)

IN list_of_handles list of handles to communication objects

OUT index index of handle for for operation that completed (integer)

OUT return_handle handle that is associated with return status object. Set
    to return status of operation that completed.

IN timeout timeout value (double)

MPI_WAITALL( list_of_handles, list_of_return_handles, timeout)

IN list_of_handles list of handles to communication objects

OUT list_of_return_handles Must have the same length as the first list. Each
    return status object is set to the return status of the corresponding
    operation in the first list.

IN timeout timeout value (double)

MPI_WAIT, MPI_WAITANY and MPI_WAITALL return a status value as their 
return value. Thus, in order to use them properly, the programmer must
check their return value in a conditional statement. A suitable value should
represent a TIMEOUT and another value SUCCESS. Since supplying a timeout
value of 0 transforms these wait functions into the equivalent of the 
MPI functions MPI_STATUS, MPI_STATUS_ANY and MPI_STATUS_ALL (NB: I didn't
find the last function in the current draft, but I think it was added at
the last meeting), these function calls can be eliminated from the MPI
standard. Specifying a timeout value of -1 indicates a "wait forever" request
(i.e., do not timeout the calls). 

Analysis of the proposal
------------------------

An examination of the proposal according to the issues described above follows:

  o  Only three MPI calls take a timeout parameter. Consequently, the MPI
     interface may be used without knowledge of or concern with user
     supplied timeouts.

  o  Use of timeouts in the WAIT family of function calls follows the
     specification of a timeout value in the UNIX I/O "select" call. Current
     practice for networked message passing is followed in this family,
     while current practice for MPP and cluster library message passing
     is followed in the other MPI calls.

  o  Provision of a timeout in the WAIT family of calls for MPPs may or
     may not be problematic. Specific vendor advice is required to clarify
     this issue.

  o  Scientific programmers not using the WAIT family of MPI calls (probably
     most) need not check return values. Use of the WAIT family is considered
     to be "advanced MPI programming" and so should only be used by "experts."

  o  Failure atomicity can still be provided by the underlying MPI
     implementation as long as the programmer only uses blocking MPI calls.
     Use of non-blocking I/O is advanced use of MPI and so requires the
     programmer to exercise more skill in developing programs.

---

From: L J Clarke <[email-masked]>
Date: Thu, 8 Apr 93 13:56:24 BST

Regarding time-outs in the non-blocking wait calls, proposed by Danny
Nesset. 

Some of the programmers either within, or assocaited with, EPCC have
specifically asked for this capability.

Exactly this facility is provided in the Meiko system CSN.

I support this proposal, and agree that we need to seek vendor advice on
whether they can sensible implement this, subject to a single amendment. 
The amendment is that the value to pass for an "infinite timeout" should
be a named constant as opposed to the literal constant -1 (although I
realise this is really a langauge binding issues). 

Vendors - please advise us.

I should point out that I strongly disagree with the authors statement
that nonblocking communication is advanced MPI usage that scientific
programmers do not have to think about.  In order for the programmer to
write a portable (i.e.  safe) program using the point-to-point
communications the programmer will have to write programs using a
mixture of blocking and nonblocking sends and receives.  Of course, the
programmer is free to write an unsafe program (which is potentailly
simpler) using only blocking sends and receives.  Let us hope that such
programmers realise that such programs are not really portable. 

Best Wishes
Lyndon

         /--------------------------------------------------------\
    e||) | Lyndon J Clarke    Edinburgh Parallel Computing Centre | e||) 
    c||c | Tel: 031 650 5021  Email: [email-masked]  | c||c 
         \--------------------------------------------------------/

---

From: [email-masked] (Dan Nessett)
Date: Fri, 9 Apr 1993 08:36:04 -0800

I agree with Lyndon's suggestion that MPI should use a named constant for
representing an "infinite timeout". Whether or not non-blocking
communication is advanced MPI usage rests on a subjective opinion of what
is elementary and what is advanced.

Dan

---

From: L J Clarke <[email-masked]>
Date: Fri, 9 Apr 93 16:28:49 BST

> is elementary and what is advanced.

I concur!

Best Wishes
Lyndon