Subject: cross-language support - a proposal (long)

From: [email-masked] (Danny Nessett)
Date: Fri, 9 Apr 93 10:00:54 PDT

I am cross posting this message to the point-to-point list, since it contains
a proposed modification to the point-to-point proposal. Those interested in
point-to-point, but not language binding issues should skip to section 3,
and evaluate whether the changes are acceptable.

1. The Problem
--------------

At the recent MPI meeting in Dallas (31 March - 2 April), the language binding
subcommittee proposed that the MPI standard make no provision for
interoperability between MPI-based programs written in different programming
languages. I objected to that suggestion and volunteered to develop a proposal
that would allow such language interoperability. My objections were and are
based on the following considerations :

  o  MPI is a mulit-use standard, being developed for homogeneous multi-
     processors, homogeneous computer clusters and heterogeneous distributed
     systems. The objective is to provide a message passing interface that
     can be used in all of these environments. In addition, there is an
     objective to promote portability of MPI-based codes from one environment
     to another.

     Correct operation of MPI-based codes in a heterogeneous environment
     requires the MPI implementation to accommodate different formats
     for the communicated data. As a side effect, programs written in
     different programming languages will interoperate as long as the data
     types are conceptually the same. For example, REALs in Fortran and
     floats in C are conceptually the same. Unless there is a requirement
     that MPI support cross-language interoperability, some programs will
     work in heterogeneous environments, but not work in homogeneous
     environments. This will give the appearance that MPI is poorly designed.

  o  MPI can be used in at least two different ways. It can be used for
     intercommunication between application executables that are aware
     (from the programmer's point of view) of each other's internal data
     structures and algorithms. It can also be used to provide services,
     much like a library provides services, in which the internal data
     structures and algorithms are not visible to the service client.
     In this second class of use, MPI acts as a generic service interface,
     supporting a wide variety of applications. Commonly, this generic service
     interface is known as client/server.

     Traditionally, libraries define their interface data structures and
     procedure prototypes in a specific programming language. Some libraries
     provide multiple interface definitions, one for each particular
     programming language they support. However, when using a generic
     service interface to access what is the equivalent of library services,
     this approach becomes problematic. The MPI interface is itself specified
     in terms of one or more programming language bindings. There is no
     useful way to specify another language binding for the client/server
     interface.

     Consequently, another approach is used for the client/server interface,
     which is programming language independent. Generally, server writers
     specify an interface in terms of a message parameter that represents
     the function or procedure to execute and in conjunction with a particular
     value of that parameter, specify a set of parameters that are the
     function's or procedure's arguments. Note that with a message passing
     based generic service interface, such as MPI (as opposed to a remote
     procedure call generic service interface), the service call is not
     constrained to obey function or procedure semantics, since the client is
     not obliged to block while the server provides the service. However,
     that is an aside.

     Since the server's interface is specified in terms of MPI data types,
     rather than in terms of the data types of the programming language used
     to write the server, there is a complete decoupling of the programming
     language used to write a server from the programming language used to
     write the client (and vice versa). The server can use one MPI language
     binding to offer service and the client can use another MPI language
     binding to access those services.

     An example may clarify this somewhat. Suppose I wish to write a server
     that provides a matrix manipulation service. I define my interface as
     follows :

       first parameter : operation to be performed (integer) -

			1 = matrix sum

			2 = matrix multiply

			3 = matrix inversion

			4 = matrix transposition

			5 = return result

       second parameter: number of rows (integer) in matrix
                         (or matrices)

       third parameter : number of columns (integer) in matrix
                         (or matrices)

       fourth parameter: first matrix for all operations (single precision
			  real array, row major order)

       fifth parameter : second matrix for operations 1 and 2 (single
			  precision real array, row major order)

     For this interface, the operation and parameters will be specified as
     MPI data types. For a C or C++ language binding of MPI, these data types
     will be "int" and "float". For a Fortran language binding of MPI, these
     data types will be "INTEGER" and "REAL". This means there really is no
     way to prevent cross language interactions, other than to provide
     a "caveat emptor" warning to the programmer. Some programmers will
     not read the manual that closely; will implement a cross language
     service interface; will use it on various machines with no problems;
     and finally will curse the MPI implementors when it doesn't work on
     other machines. This will reduce confidence in MPI as a standard
     message passing interface.

     Note the property described in the previous bullet. In a heterogeneous
     environment, even if both client and server are written in the same
     language, the MPI implementation must accommodate conversion between
     different ranges of values for a particular type.  For example, if
     one machine represents integers as 32 bit quantities, while another
     represents them as 64 bit quantities, sending an integer parameter
     from the second to the first requires a check to ensure the 64 bit
     value can be represented as a 32 bit value (e.g., that 100,000
     represented as a 64 bit number fits into a 32 bit representation).
     This check also allows a client written in one language (say, C) to call
     a server written in another language (say, Fortran).

2. My view of the issues
------------------------

Below I summarize my view of the issues in regards to cross-language MPI
service. I am sure that there are other issues that I have not thought of
and welcome others to contribute them to the discussion.

  o  As I have stated above, MPI is a multi-use standard being targeted for
     homogeneous multi-processors, homogeneous computer clusters and
     heterogeneous distributed systems. Vendor and user communities
     representing each of these environments are participating in
     the MPI standardization process. This invariably leads to a conflict
     of objectives. For example, the community most interested in homogeneous
     multi-processors does not want to sacrifice the performance of
     parallel programs running on these machines in order to accommodate
     heterogenous distributed processing or cross-language MPI support.
     Alternatively, those interested in heterogenous distributed systems
     don't want to constrain the MPI standard in such a way that severly
     limits its applicability in those environments. Those interested in
     homogeneous computer clusters are probably somewhere in between
     with their objectives.

     Consequently, if the MPI standard is to provide cross-language support,
     it should do so in a way that doesn't penalize the performance of
     programs running on homogeneous multi-processors, while at the same
     time minimizing the implementation effort for cross-language and
     heterogenous distributed system support. Also, programmers who intend
     to use only one language should not have to think about cross-language
     issues.

  o  Cross-language support raises the question of translation between data
     values that are of a fundamentally different type (as opposed to being of
     a different "kind" in the Fortran 90 sense). For example, should the MPI
     standard provide support that would allow a Fortran program to send a
     "complex" value to a program written in C? (By this I mean support
     the transmission of the Fortran "complex" type to a C program, not
     the transmission of a "complex" value represented in some user
     specified way, like two real values). A cross-language facility may
     or may not allow this, depending on the degree of automation the
     standard is designed to provide. It is conceivable for a cross-language
     standard to support the communication of only those data types that
     are common to all of the target programming languages.

     There is also the issue of type coercion of transmitted values within
     a single language. For example C allows you to coerce a real value
     into an integer value when you assign a real value to an integer
     variable. Taking this approach would imply a sender could specify a
     real value in an MPI_ADD_BLOCK call, while the receiver specified
     an integer variable and expect MPI to perform the conversion.
     Allowing such coercion in the MPI interface would significantly increase
     its complexity, since the programmer would now have to specify not
     only the type being sent or received, but also the type being coerced.
     For this reason, I believe we have decided not to support such
     type coercion in MPI.

  o  Translation between values of the same type (but perhaps different
     "kind") across languages requires knowledge of the mapping between
     a particular language type and its machine representation. For example,
     in Fortran 77 a REAL might map to an IEEE 32 and a DOUBLE PRECISION to an
     IEEE 64. In C a float might map to an IEEE 32, a double to an IEEE 64
     and a long double to an IEEE 64. In Fortran 90 a REAL with a given
     kind parameter might map to either an IEEE 32 or an IEEE 64.  Similar
     mapping of int, long, INTEGER (with different kind parameters in
     Fortran 90), char, CHARACTER, LOGICAL and COMPLEX to underlying
     representations is required to properly support cross-language
     interoperability. Some language types that have been suggested as
     MPI types would require definition in certain languages. For example,
     LOGICAL in Fortran has no direct type analog in C. Similarly, there is
     no COMPLEX data type in C. It would be possible to simply declare
     use of these MPI types in a program written in an incompatible
     programming language as erroneous or the standard could specify
     a mapping between these type and non-native types in the appropriate
     languages (e.g., LOGICAL could map to unsigned char and COMPLEX to
     a structure containing two reals).

     In a heterogeneous environment the mapping information can be more
     complex. On a SUN Workstation REALs, DOUBLE PRECISIONS, floats, doubles
     and long doubles may map as specified above. On a CRAY, however, a
     REAL may map to a CRAY 64, a DOUBLE PRECISION to a CRAY 64, a float
     to a CRAY 64, a double to a CRAY 64 and a long double to a CRAY 128.

  o  Cross-language interoperability may require an MPI implementation
     that supports a particular language binding to be aware in some
     way of the existence of other language bindings. This may cause
     a problem when new MPI language bindings are developed. The features
     of MPI that support cross-language interoperability should allow
     the graceful integration of new language bindings into the standard.

3. A Proposal
-------------

In order to support cross-language use of MPI, I propose the following
modification to the point-to-point draft. The proposal is in two completely
independent parts. One can be accepted without accepting the other.

3.1 First Part

Modify the datatype parameter values of the MPI_ADD_... functions so that they
come from the following set :

   MPI_REAL
   MPI_DOUBLEPRECISION
   MPI_COMPLEX
   MPI_INTEGER
   MPI_LOGICAL
   MPI_CHARACTER

   MPI_FLOAT
   MPI_DOUBLE
   MPI_LONGDOUBLE
   MPI_SHORT
   MPI_INT
   MPI_LONG
   MPI_CHAR    (a character array)
   MPI_UCHAR

   MPI_BYTE

These values for the datatype parameter are allowed in any MPI implementation
irrespective of its language binding. They specify the type of the data that
is being sent/received. Notice that the first group (i.e., MPI_REAL,...,
MPI_CHARACTER) are Fortran types; the second group (MPI_FLOAT,...,MPI_UCHAR)
are C (C++) types and MPI_BYTE is a language independent type. The use of a
type from the first group in a Fortran program indicates that the data being
sent/received is in Fortran format and consequently does not require
translation. The use of a type from the first group in a C (C++) program
indicates that the data being sent/received is in Fortran format and so may
require translaton. Similarly, the use of a type from the second group in a
Fortran program indicates the data being sent/received is in C (C++) format
and so may require translation. The use of a type from the second group in a C
program indicates the data being sent/received is in C (C++) format and so
need not be translated.

A client/server interface would specify its interface in terms of these MPI
types. When a MPI_ADD_... function is called, the specified type would be
used as the datatype parameter irregardless of the MPI language binding.
Thus, if a parameter is specified as MPI_REAL, that datatype would be
specified both in Fortran and C programs using the client/server interface.
To properly pass messages, both the client and server must use the same
datatype for the parameter.

In order for the MPI implementation to take advantage of the provided type
information in a cross-language communication, it must know which Fortran
types map into which C types. Therefore, the standard should specify
this information. Let me make the following strawman proposal : MPI_REAL
maps to MPI_FLOAT; MPI_DOUBLEPRECISION maps to MPI_DOUBLE; MPI_COMPLEX
maps either to a C struct or has no mapping, which would cause an error
in a cross-language communication; MPI_INTEGER maps to MPI_INT; MPI_LOGICAL
maps to MPI_UCHAR or has no mapping, which would cause an error; and
MPI_CHARACTER maps to MPI_CHAR.

There are three C types that have no natural analogs in Fortran : MPI_SHORT,
MPI_LONG, and MPI_LONGDOUBLE. MPI_SHORT and MPI_LONG probably should map
to MPI_INTEGER, since it is the only integer type available in Fortran 77.
MPI_LONGDOUBLE probably should map to MPI_DOUBLEPRECISION. However, that means
that MPI_DOUBLE and MPI_LONGDOUBLE map to the same Fortran data type. An
alternate mapping would map MPI_DOUBLE to MPI_REAL and MPI_LONGDOUBLE to 
MPI_DOUBLEPRECISION. This requires discussion.

In addition to a standard mapping between programming language types, a
particular MPI implementation must know how the supported language types
map into underlying machine representations. For example, it must know
that REAL maps into IEEE 32, int maps into a 32 bit 2's complement value, etc.

An MPI implementation would operate as follows. I categorize the implementations
by environment in order to demonstrate properties of the previously discussed
issues.

   3.1.1 Homogeneous Mulit-processors and Computer Clusters
   --------------------------------------------------------

I describe the behavior of an MPI implementation with a Fortran language
binding. The corresponding actions of an MPI implemenation with a C language
binding should be obvious.

   3.1.1.1 Send

If the datatype parameter is in the Fortran set, block copy the data using the
underlying system network. If the datatype parameter is in the C set, use the
type mapping to decide what is the corresponding Fortran type and use the
per implementation machine representation mapping to decide the underlying
representation for both the Fortran and C type. If they are the same, block
copy the data using the underlying system network. If not, convert the data
to the appropriate underlying representation for the C type and send the
converted data.

   3.1.1.2 Receive

If the datatype parameter is in the Fortran set, the buffer in which the data
arrived is in the proper format. Make it available to the MPI caller. If
the datatype parameter is in the C set, use the type mapping to decide what
is the corresponding Fortran type and use the per implementation machine
representation maping to decide the underlying representation for both
the Fortran and C type. If they are the same, the buffer in which the
data arrived is in the proper format. Make it available to the MPI caller.
Otherwise, convert the buffer to the appropriate underlying representation
and make it available to the MPI caller.

A comment on implementation strategy. It is possible for both a send
and receive operation to know before hand whether it needs to translate
the buffer data or not (i.e., the above algorithms can be run before the
actual machine transmission is sent or received). Consequently, in those
situations in which translation is necessary, the implementation can supply
an intermediate buffer in which to translate or from which to translate the
data. By knowing both the Fortran and C types as well as their underlying
representations, the implementation can precalculate the size of the
translation buffer. 

   3.1.2 Heterogeneous Distributed Systems
   ---------------------------------------

Supporting MPI communications in a heterogeneous distributed system is more
complicated than in a homogeneous environment. Not only must differences in
programming language data types be accommodated, differences in underlying
machine represenations are also a concern. The exact algorithms to use when
sending and receiving depend on the particular presentation-level protocol
employed. Protocols like XDR and ASN.1 use a intermediate representation
of data for transmission purposes. A protocol like NDR (used in OSF DCE)
transmits the data in the format of the sender, placing the burden on the
receiver to translate it. In the following discussion I finesse the protocol
issue by using the generic description "call the off-machine protocol
translation module" to mean execute the appropriate presentation protocol
algorithm. Some implementations will combine the protocol translation activity
with sending or receiving the data.

   3.1.2.1 Send

If the datatype parameter is in the Fortran set, determine the underlying
machine represenation and call the off-machine protocol translation module
to put it into the proper format. Send the result to the receiver. If the
datatype parameter is in the C set, use the type mapping to decide what is
the corresponding Fortran type. Determine the Fortran type's underlying
representation and call the off-machine protocol translation module to put
it into the proper format. Send the result to the receiver.

   3.1.2.2 Receive

If the datatype parameter is in the Fortran set, determine the underlying
machine representation and call the off-machine protocol translation module
to put it into the proper format. Return this result to the MPI caller. If the
datatype parameter is in the C set, use the type mapping to decide what is
the corresponding Fortran type. Determine the Fortran type's underlying
representation and call the off-machine protocol translation module to
put it into the proper format. Return this result to the MPI caller.

3.2 Second Part

Cross-language interoperability raises the issue of handling data types in
one programming language that have no exact analog in another programming
language. With the current suggested language bindings for MPI (i.e., Fortran
77, Fortran 90, C and C++), I think the following types fall into this
category, one way or another :

   Fortran 77

   LOGICAL
   COMPLEX

   Fortran 90

   REAL(SELECTED_REAL_KIND(--,--))
   INTEGER(SELECTED_INT_KIND(--))
   LOGICAL
   COMPLEX
   COMPLEX(SELECTED_REAL_KIND(--,--))

C and C++ types included in the MPI standard have natural analogs in Fortran 77
and Fortran 90.

There is at least two ways to handle these type mismatches. The simplest
strategy is to generate an error when these types are referenced in an
inappropriate language. This approach has the advantage of being simple to
implement. It has the disadvantage that MPI-based programs written in one
language without thought of using it from a program written in another language
will likely use inappropriate types.

The second strategy is to define analogs in each language for the non-common
types. For example, LOGICAL in both Fortran 77 and Fortran 90 could map to
unsigned char in C. COMPLEX in Fortran 77 and Fortran 90 could map to a
struct with two float members in C. However, REAL, INT and COMPLEX with
KIND parameter information cannot be handled in this way, since the
"length" of the type is a machine dependent quantity. For example, on
some machines INTEGER(SELECTED_INT_KIND(4)) might be equivalent to an int,
while on other machines it is equivalent to a long (and not an int). Similar
"length" problems exist with the REAL(SELECTED_REAL_KIND(--,--)) type.

With these points in mind I propose the following. Map LOGICAL into unsigned
char, COMPLEX into a C struct with two floats and disallow specification
of the Fortran 90 types that specify a KIND parameter. Since Fortran 90
supports a KIND function call that the programmer can use to determine when
INTEGER and INTERGER(SELECTED_INT_KIND(--)), REAL and
REAL(SELECTED_REAL_KIND(--,--)), and COMPLEX and
COMPLEX(SELECTED_REAL_KIND(--,--)) are equivalent, there is a work around
for most cases.

4. Analysis of the Proposal
---------------------------

Following is an analysis of the proposal according to the issues specified
in section 2.

  o  As described in section 3.1.1 the proposal can be implemented in such
     a way that it does not adversely impact the performance of either
     homogeneous multi-processors or homogeneous computer clusters. The 
     programmer who writes programs in one language need never consider cross-
     language issues. Furthermore, the MPI types he/she would use would be
     natural for the programming language in which he/she is working.

  o  The issue of mapping data types not found in one programming language
     into data types found in another is addressed by the second part, which
     covers mapping of data types by defining analagous types in the
     programming language from which a type is missing. Type coercion is
     not supported between types, since it is simple to first coerce the
     type (e.g., float to int) before sending it. It is not simple to support
     this kind of type coercion from within the MPI implementation.

  o  The proposal handles translation of values of the same type
     by requiring the programmer to specify the type of data in the
     datatype parameter of the MPI_ADD_... functions. There is a limited
     amount of translation between types of different kinds due to the
     mapping required to associate a type in one language with a type
     in another language. This has the advantage of automating much
     of the work required to communicate values in a cross-language
     situation. However, there is also a possible disadvantage. In a
     heterogeneous distributed system the mapping could lead to the loss
     of information or to an error. For example, the suggested mapping
     associates a float with a REAL. On a CRAY a REAL is a 64-bit floating
     point number, while on a SUN a float is a 32-bit floating point.
     Transfering a REAL on a CRAY to a float on a SUN causes loss of
     precision and potentially could result in a translation error
     because the value on the CRAY cannot be represented by a 32-bit
     floating point. This problem also occurs on a single machine if
     a REAL maps to, say, an IEEE 64 and a float to an IEEE 32 (which
     doesn't seem likely).

  o  Adding a new language binding to the MPI standard requires the definition
     of new MPI_<type> values and the mapping between these new values and
     the existing MPI_<type> values. Introduction of MPI implementations with
     support for the new language bindings can be accomplished gradually, since
     they will support the old MPI_<types>, eventhough the old implementations
     do not support the new MPI_<types>. As vendors upgrade their MPI
     implementations to conform to the new language binding standards, the
     new MPI_<type>s can be used more and more until all useful implementations
     support them. Thus, the proposal supports the gradual introduction of
     new language bindings without requiring all implementations to immediately
     support them.