Subject: counting the number of elements received

From: [email-masked] (Dan Nessett)
Date: Mon, 28 Jun 1993 07:55:41 -0800

During the pt2pt subcommittee meeting on Thursday evening, we discussed what
the appropriate units were for the counts returned as the result of receive
operations. There were two proposals remaining at the end of the session :
1) return the number of elements received according to the "flattened"
type (i.e., the number of "basic type" elements received), and 2) return
the number of elements received at the next level below the "top" level
specified by a buffer descriptor or type. The latter proposal treats arrays
differently than struts, since a buffer descriptor or type consisting of a
list of basic types, structs and arrays used in a receive returns the
number of basic types and structs received plus the number of elements received
in the arrays. Furthermore, if a receive terminates without receiving enough
data to complete the type or buffer descriptor, it is an error if the struct
is only partially filled in, while it is not an error if the incomplete data
received ends within one of the arrays specified in the type or buffer
descriptor.

After the meeting on Friday, some of us were having lunch and discussing
this issue. Since my position is that we should use proposal #1, I was
defending it. In the process I tried to come up with an example for which
proposal #2 is not the suitable semantics. I couldn't do so on the spur of
the moment and asked if I might retire from the discussion until I could give
the matter some thought. The other participants graceously consented.

On the flight home from the meeting, I worked out the details of such an
example. While it is fictional, I believe it is plausable. I also
believe given enough time and motivation I could come up with other examples
that illustrate the problems with proposal #2. However, allow me to stick
to this example for the moment.

Consider someone designing a deep space vehicle that will be put in orbit
around Jupiter and send probes into its atmosphere. The vehicle will collect
data about the planet's atmosphere and send it back to earth. The designer
wishes to use MPI to send the data messages.

The characteristics of this engineering problem are as follows. The power
budget for a probe is such that it is not possible for it to send back
messages containing telemetric data periodically. Instead, it monitors
both the temperature and pressure of its environment and when either has
changed sufficiently, takes a sample of, say, the atmosphere's chemistry,
returning a message with both the chemistry/press/temp data and its position.
To make this concrete, suppose the chemistry data is represented by an
array of 10 reals, press/temp by a double each and position by a struct.
Since the chemistry sampling takes a finite time to carry out, the pressure
and temperature will most likely change and so the probe sends back the
pressure and temperature at the beginning of the sample, the temperature
and pressure at the end of the sample along with the position and chemistry
data.  In order to reduce the amount of data the probe sends, the temperature,
pressure and position data in most messages are not their absolute values.
Rather, they are the difference between the current values and those represented
in the previous message. These deltas are represented as reals rather than as
doubles. However, every 100th message contains the more precise values for
pressure, temperature and position. Consequently, the messages sent back have
the following form (as represented in C) :

Delta messages
--------------

struct delta_data
  {
  struct position
    {
    real range;
    real phi;
    real theta;
    } position;
  real start_temp;
  real start_press;
  real end_temp;
  real end_press;
  real chemistry_data[10];
  } delta_data;

Full messages
-------------

struct full_data
  {
  struct position
    {
    double range;
    double phi;
    double theta;
    } position;
  double start_temp;
  double start_press;
  double end_temp;
  double end_press;
  real chemistry_data[10];
  } full_data;

The space vehicle receives this data from the probe, archives it and at the
end of the probe's descent, relays the archived message stream to earth.
The probe descent takes approximately 4 hours, during which time much more
data is generated than can fit into the space vehicle's RAM. Consequently, the
space vehicle's computer system stores the data on an onboard tape system
for later replay. The space vehicle is also running several other experiments
and needs to archive this data as well. Therefore, the vehicle's platform
computer periodically dumps its complete RAM to tape. This dump occurs after a
suitable interval of time has elapsed and when the last of the delta messages
in a 100 message sequence has been received. The amount of memory allocated
to the probe message data is approximately 80% of the RAM available. The
tape subsystem is architected with a cache memory equal in size to the RAM
memory. This is also the size of a tape block. A tape copy begins by
a fast RAM to cache transfer, followed by a much slower cache to tape copy.

Since the probe's descent into the Jovian atmosphere will eventually destroy
it, its data stream can end unpredictably. Furthermore, the "tail" data may
be the most interesting and therefore cannot be simply dropped. When the
probe data ends, the space vehicle rewinds its tape and reads it a block
at a time (i.e., a RAM dump at a time), sending the message stream to earth.
The MPI send/receive buffer descriptor corresponds to the following C data
structure (assume the probe data is at the end of the RAM memory):

struct memory_dump
  {
  struct experiment1_data
     {
     .
     .
     .
     } experiment1_data;
  struct experiment2_data
     {
     .
     .
     .
     } experiment2_data;
  struct experiment3_data
     {
     .
     .
     .
     } experiment3_data;
  struct experiment4_data
     {
     .
     .
     .
     } experiment4_data;
  struct experiment5_data
     {
     .
     .
     .
     } experiment5_data;
  int number_of_chunks;
  struct probe_message_chunk
    {
    int number_of_intermediates;
    struct full_data calibration;
    struct delta_data intermediates[99];
    } probe_message_chunk[1000];
  } memory_dump;

Here I am assuming that 1000 "minor cycles" of probe_message_chunk is sufficient
to hold all the probe data before a RAM write to tape occurs.

If MPI receives behave according to proposal #2, the last RAM dump on the tape
would cause an MPI receive error, since the last component in memory_dump (i.e.,
probe_message_chunk) will in general not be received completely.

Let me anticipate some questions about this example and attempt to respond to
them:

QUESTION: Why not stride out the calibration data through an append_vec
call and similarly stride out the intermediates data?

ANSWER: Of course, that is possible. However, we were trying to come up with
a way to send structures without using the "stride trick."

QUESTION: Why not promote probe_message_chunk to the "top level in memory_dump?

ANSWER: This would require 1000 copies of probe_message_chunk (called, I
suppose, something like probe_message_chunk1, probe_message_chunk2, etc.)
As a programmer, I would strongly object to being forced to do something like
this.

I realize that this is a somewhat artificial example.  Given more time
I think I could come up with one that is focused on scientific computing.
However, I believe the principal is clear. At the very least, the programmer
should have a way of turning off (e.g., through an environmental variable)
the property that counts are based on a "level 2" view of the data and that
receives will fail when only a partial struct is received.

Regards,

Dan

P.S. I am about to go incommunicado for about a month, so I'll have to pick
up the discussion of this issue when I get back. I wanted to get this out
before I left on business travel and some vacation.