Subject: "Message Capsule" Proposal (long)

From: [email-masked] (Tom Henderson)
Date: Fri, 15 Jan 93 10:23:53 MST

Hi all, 

Here is a proposal that we think may help reduce the number of separate 
send and receive routines to something much less than "512" without 
eliminating any of the nice features that we haven't been able to say "no" 
to.  The proposal also allows for "simple" versions of send and receive 
routines for novice users.  The proposal borrows ideas from Jim Cownie, Paul 
Pierce, and others (without their consent-- don't blame anyone but us!).  
We hope that this proposal will at least start a few good arguments.  :)

Leslie Hart
[email-masked]

Tom Henderson
[email-masked]

Bernardo Rodriguez
[email-masked]

NOAA Forecast Systems Lab

-----------------------------------------------------------------------------

                     A PROPOSAL FOR "MESSAGE CAPSULES"

1  Introduction to Message Capsules

To send a message, a user must first create and initialize a data structure 
called a "message capsule".  (In Fortran77 a message capsule might be an 
integer array, possibly containing pointers used by underlying C routines.  
Another way might be to use the POSIX approach mentioned by Bob Knighten.)  
The message capsule is an opaque object:  a user should only access the 
contents of a message capsule using a special set of routines.  A message 
capsule contains a complete description of the user data in the message 
including user data location, data type(s), and optional stride information.  
The capsule will also contain information describing the data in an incoming 
message.  (In the current proposal this information is transmitted with each 
message.  This is only necessary if we allow reception of a message by a 
process that does not know its complete data description.)  A message capsule 
also contains message tag, context, control flags, and "matching" 
information.  "Matching" information is used by a receiving process to select 
a message for receipt.  "Matching" criteria has not been defined for MPI.  

A receiving process must also create and initialize a message capsule.  When a 
message is received, the receive routine uses the information in the message 
capsule to move data from system buffers (or communication hardware) to 
location(s) in user space described in the capsule.  The information in the 
message capsule is used by the receive routine to guide any data translation 
that may be necessary in a heterogeneous system.  Note that the use of message 
capsules does not imply extra data movement.  The message capsule just tells 
the system "Here's where to put the data when it arrives...".  

In the current proposal, message capsules are dynamic data objects.  A message 
capsule can be reused once the data described by it is no longer required for 
message passing communication.  Routines are provided to create and destroy 
message capsules, attach descriptions of user data, inquire about the contents 
of a message capsule, and modify message context.  Some proposed routines are 
described below.  

The send and receive routines have "message capsule" as an additional 
parameter.  Context is not required as an explicit input parameter to these 
routines.  Proposed (incomplete) syntax for these routines is described below 
along with syntax for check, wait, probe, and release routines.  Proposed 
syntax for "simplified" versions of send and receive routines are also 
described below.  

2  Program Flow

  General steps for typical send -> receive communications using 
  message capsules:  

  Case 1.  Receiving process knows the description of data in the message.  

1)   SENDING PROCESS:  
     Sending process creates and initializes a message capsule by calling 
     capsule creation and data description routines.  Message context 
     specification routine may also be called to select context if "default" 
     context is not desired.  A flag is set in the message capsule to indicate 
     that it has been initialized by the user.  If multiple messages with 
     identical data descriptions are to be sent sequentially, this step only 
     needs to be done for the first message.  

2)   RECEIVING PROCESS:  
     Receiving process creates and initializes a message capsule by calling 
     capsule creation and data description routines.  Message context 
     specification routine may also be called to select context if "default" 
     context is not desired.  A flag is set in the message capsule to indicate 
     that it has been initialized by the user.  If multiple messages with 
     identical data descriptions are to be sent sequentially, this step only 
     needs to be done for the first message.  

3)   SENDING PROCESS:  
     "Send" message.  During a send operation, the send (or wait for a 
     non-blocking send) routine uses information in the message capsule to 
     collect user data, construct a "message" (using system buffer(s) if 
     necessary), and send the message to a specified destination.  Depending 
     on the implementation, the send routine may also perform data translation 
     in a heterogeneous system.  While a send operation is in progress, a flag 
     is set in the capsule to indicate that the capsule should not be used for 
     anything else.  After the operation completes, this flag is cleared.  A 
     send routine will return an error if the message capsule has not been 
     properly initialized.  

4)   RECEIVING PROCESS:  
     "Receive" message.  During a receive operation, the receive (or possibly 
     wait for a non-blocking receive) routine distributes received data to 
     user location(s) described in the message capsule.  Depending on the 
     implementation, the receive routine may also perform data translation in 
     a heterogeneous system.  While a receive operation is in progress, a flag 
     is set in the capsule to indicate that the capsule should not be used for 
     anything else.  After the operation completes, this flag is cleared.  The 
     receive (or wait) routine will return an error if the data in the 
     received message does not match the description in the capsule (ie. 
     mismatch in data length(s) or type(s)).  An error will also be returned 
     if the message capsule has not been properly initialized.  

5)   RECEIVING PROCESS:  
     (OPTIONAL)  "Information" about message described by a capsule.  
     Information about a message may be obtained by calling various "info" 
     routines with the corresponding message capsule specified as an 
     argument.  Information returned by these routines includes data types, 
     data sizes, data strides, sending process, message tag, and message 
     context.  "Info" routines will return an error if a message has not been 
     successfully "probed" or received.  

  Case 2.  Receiving process does not know the description of data in the 
           message.  

1)   SENDING PROCESS:  
     Sending process creates and initializes a message capsule by calling 
     capsule creation and data description routines (same as Case 1).  

2)   RECEIVING PROCESS:  
     Receiving process creates a message capsule by calling the capsule 
     creation routine.  The message capsule may optionally be initialized in 
     this step by calling the data description routines (though it may need to 
     be re-initialized after the message arrives).  Message context 
     specification routine may also be called to select context if "default" 
     context is not desired.  

3)   SENDING PROCESS:  
     "Send" message.  Same as Case 1.  

4)   RECEIVING PROCESS:  
     (NOT OPTIONAL)  "Probe" message (inquire if a message with "matching" 
     characteristics has arrived-- "matching" criteria have not yet been 
     defined for MPI).  If a message with "matching" characteristics has 
     arrived, data description information is loaded into a message capsule 
     specified as an argument to the probe routine.  The message is 
     removed from the system message queue so no other thread may probe or 
     receive the message unless it is "released" by the thread (see "release" 
     in step 5 below).  A flag is set in the message capsule to indicate that 
     the data description of the arrived message is valid.  This flag is used 
     by the "info" routines.  Note that the message capsule must be able to 
     separately store the user's data description and the data description of 
     the "arrived" message.  "Info" routines will only return information 
     about "arrived" messages.  

5)   RECEIVING PROCESS:  
     (NOT OPTIONAL)  "Info" routines MUST be called before a receive is posted 
     when data description is not known by the receiving process.  In a C 
     program, information returned by these routines may be used to guide 
     dynamic allocation of user data structures to store message data.  In a 
     Fortran77 program, this information can be used to check if available 
     static arrays are large enough to hold message data before a receive is 
     posted.  

6)   RECEIVING PROCESS:  
     (OPTIONAL)  "Release" message.  If a thread does not wish to receive a 
     message after a successful probe, it must release the message so 
     other threads will have an opportunity to probe or receive the 
     message.  The release routine clears the flag mentioned in step 4 and 
     puts the message back into the system message queue.  Return to step 4.  

7)   RECEIVING PROCESS:  
     Receiving process initializes the message capsule by calling data 
     description routine(s) using information returned by the "info" 
     routines.  A flag is set in the message capsule to indicate that it has 
     been initialized by the user.  If the message capsule was initialized in 
     step 2 and it does not have the same description as the arrived message, 
     then the data description must be "reset" using a special routine, and 
     then initialized.  

8)   RECEIVING PROCESS:  
     "Receive" message.  Same as Case 1.  

3  Proposed Message Capsule Initialization and Inquiry Routines

Basic functionality of proposed routines are described below.  A sample syntax 
is shown just to make explanation easier.  (For example, the issue of whether 
these routines are functions or subroutines is not important at this point, 
parameter ordering has not been given any thought, etc.)  Also, routine names 
have been chosen to be descriptive an do not conform to any string length 
limitations.  

3.1  Message Capsule Creation and Destruction Routines

MPI_CREATE_CAPSULE (CAPSULE)

  Create message capsule CAPSULE.  

MPI_DESTROY_CAPSULE (CAPSULE)

  Destroy message capsule CAPSULE.  

3.2  Data Description Routines

Each message capsule contains one or more "data items" to describe the data in 
the corresponding message.  Each data item contains information about data 
type, data length, and storage location(s).  Each data item is created by a 
single call to a MPI_ATTACH_xxx() routine.  

Two kinds of data items are supported:  "vector" and "strided".  A vector data 
item is used for user data stored in contiguous memory.  A strided data item 
is used for non-contiguous user data that can be described as a sequence of 
identically sized data blocks separated by a fixed stride.  More general user 
data arrangements may be described by multiple vector and strided data items.  
(Note that the vector data item is a subset of the strided data item.  Vector 
data items could be removed to get rid of almost half of the MPI_ATTACH_xxx() 
routines.  We included the vector data item because it simplifies the most 
commonly used data description.)  

MPI_ATTACH_INT_VECTOR (CAPSULE, LENGTH, ARRAY)

  The next data item in the message described by message capsule CAPSULE 
  is LENGTH integers.  Storage location is user array ARRAY.  

MPI_ATTACH_REAL_VECTOR (CAPSULE, LENGTH, ARRAY)

  The next data item in the message described by message capsule CAPSULE 
  is LENGTH real numbers (floats) stored in user array ARRAY.  

(etc. for other data types)...  

MPI_ATTACH_INT_STRIDE (CAPSULE, NUM_BLOCKS, BLOCK_LENGTH, STRIDE, ARRAY)

  The next data item in the message described by message capsule CAPSULE 
  is NUM_BLOCKS blocks of integers.  Each block has BLOCK_LENGTH integers.  
  Starting elements in each block are separated by STRIDE integers.  Storage 
  location of first block is user array ARRAY.  

MPI_ATTACH_REAL_STRIDE (CAPSULE, NUM_BLOCKS, BLOCK_LENGTH, STRIDE, ARRAY)

  The next data item in the message described by message capsule CAPSULE 
  is NUM_BLOCKS blocks of real numbers (floats).  Each block has BLOCK_LENGTH 
  real numbers.  Starting elements in each block are separated by STRIDE 
  real numbers.  Storage location of first block is user array ARRAY.  

(etc. for other data types)...  

MPI_ATTACH_RESET (CAPSULE)

  All data items are removed from message capsule CAPSULE.  The flag that 
  indicates that CAPSULE has been initialized by the user is cleared.  This 
  routine can be used in step 7 of Case 2 in Section 2 above to remove any 
  existing data item(s) before re-initializing the message capsule.  It must 
  be used in this situation if the message capsule contains a user data 
  description that does not match the data description of the arrived 
  message.  This routine is also used to discard data descriptions that are 
  no longer needed allowing message capsules to be re-used.  

3.3  Context Modification Routine

MPI_SET_CONTEXT (CAPSULE, CONTEXT)

  The context for the message described by message capsule CAPSULE is set to 
  context CONTEXT.  

3.4  "Information" Routines

These routines must be used when a receiving process does not know the 
data description of a message it wants to receive.  Use of these routines in 
other cases is optional.  In all cases, "length" refers to number of elements 
(integers, floats, ...) not number of bytes.  All of these routines return an 
error if the specified message has not been successfully probed or received.  
Only information about messages that have actually arrived is returned.  Note 
that no "info" routine is required for context since this must always be 
known by the receiving routine.  

MPI_INFO_NUM_DATA_ITEMS (CAPSULE)

  Returns the number of data items in the message.  

MPI_INFO_DATA_ITEM_DESCRIPTION (CAPSULE, ITEM_NUMBER)

  Returns a description of data item number ITEM_NUMBER in the message 
  described by message capsule CAPSULE.  Valid return values include 
  INT_VECTOR_ITEM, REAL_VECTOR_ITEM, INT_STRIDE_ITEM, REAL_STRIDE_ITEM, etc.  

MPI_INFO_VECTOR_ITEM (CAPSULE, ITEM_NUMBER, LENGTH)

  LENGTH is set to the length of the vector data described by data item 
  number ITEM_NUMBER in message capsule CAPSULE.  An error is returned if 
  the data item is not a vector data item.  

MPI_INFO_STRIDE_ITEM (CAPSULE, ITEM_NUMBER, NUM_BLOCKS, BLOCK_LENGTH)

  NUM_BLOCKS is set to the number of blocks in the strided data described 
  by data item number ITEM_NUMBER in message capsule CAPSULE.  BLOCK_LENGTH is 
  set to the length of each block.  An error is returned if the data item is 
  not a strided data item.  Note that stride is not returned since there is no 
  requirement that strides used by the sending and receiving processes match.  

MPI_INFO_TAG (CAPSULE)

  Returns the message tag for the message described by message capsule 
  CAPSULE.  

MPI_INFO_SENDER (CAPSULE)

  Returns the sending process for the message described by message capsule 
  CAPSULE.  

4  Proposed (Partial) Syntax for Send, Receive, Check, Wait, Probe, and 
   Release Routines

Syntax examples showing how message capsules affect send, receive, check, 
wait, probe, and release routines are shown below.  Routine naming is for 
clarity only and does not conform to any of the MPI naming styles.  

4.1  Send

Syntaxes for the three flavors of send discussed at the Dallas meeting 
("synchronous", "blocking", and "non-blocking") are shown below.  (These three 
routines could be reduced to one if the "communication mode" parameter were 
used.  It is also possible to embed even more information in the message 
capsule such as "communication mode", message tag, and "matching" criteria.  
We show examples for this case in Section 8.)  

MPI_SEND_SYNCH (CAPSULE, DESTINATION, TAG)

  Send the message described by message capsule CAPSULE to process DESTINATION 
  using message tag TAG.  This is a "synchronous" send.  

MPI_SEND_BLOCKING (CAPSULE, DESTINATION, TAG)

  Send the message described by message capsule CAPSULE to process DESTINATION 
  using message tag TAG.  This is a "blocking" send.  

MPI_SEND_NON_BLOCKING (CAPSULE, DESTINATION, TAG)

  Send the message described by message capsule CAPSULE to process DESTINATION 
  using message tag TAG.  This is a "non-blocking" send.  

4.2  Receive

Syntaxes for the two flavors of receive discussed at the Dallas meeting 
("blocking" and "non-blocking") are shown below.  (These two routines could 
be reduced to one if the "communication mode" parameter were used.)  
"Matching" criteria have been left undefined.  

MPI_RECV_BLOCKING (CAPSULE, "MATCHING CRITERIA")

  Receive the message with characteristics satisfying "MATCHING CRITERIA" and 
  store message data in location(s) described in message capsule CAPSULE.  
  This is a "blocking" receive.  Error conditions may be returned if the 
  user-provided data description in the message capsule is not consistent with 
  the data description of the received message (message too long, wrong data 
  types, etc.).  

MPI_RECV_NON_BLOCKING (CAPSULE, "MATCHING CRITERIA")

  Receive the message with characteristics satisfying "MATCHING CRITERIA" and 
  store message data in location(s) described in message capsule CAPSULE.  
  This is a "non-blocking" receive.  Error conditions may be returned if the 
  user-provided data description in the message capsule is not consistent with 
  the data description of the received message (message too long, wrong data 
  types, etc.).  (This will only happen if this routine is called when a 
  "matching" message has already arrived.)  

4.3  Wait and Check

The wait and check routines use the message capsule in a way that is 
very similar to Jim Cownie's proposal (option 5 in "Removing internal 
state").  These routines are used with the non-blocking send and receive 
routines.  

MPI_CHECK (CAPSULE)

  CALLED BY A SENDING PROCESS:  
  Return value indicates whether a send operation on the message described by 
  message capsule CAPSULE has completed or not.  An error will be returned if 
  a non-blocking send has not been previously posted for CAPSULE.  If the send 
  operation has completed, clear the flag in CAPSULE that indicates that 
  communication is in progress.  

  CALLED BY A RECEIVING PROCESS:  
  Return value indicates whether a receive operation on the message described 
  by message capsule CAPSULE has completed or not.  An error will be returned 
  if a non-blocking receive has not been previously posted for CAPSULE.  If 
  the receive operation has completed, error conditions may be returned if the 
  user-provided data description in the message capsule is not consistent with 
  the data description of the received message (message too long, wrong data 
  types, etc.).  If the receive operation has completed, clear the flag in 
  CAPSULE that indicates that communication is in progress.  (Depending on the 
  implementation, user data locations described in CAPSULE may be loaded by 
  this routine, or they may already be loaded by the time this routine is 
  called.)  

MPI_WAIT (CAPSULE)

  Same as MPI_CHECK() except that the routine will not return until a 
  "matching" message arrives.  

4.4  Probe and Release

The probe routines also uses the message capsule in a way that is 
very similar to Jim Cownie's proposal (option 5 in "Removing internal 
state").  Both wait and check styles of probe are proposed.  

MPI_PROBE_CHECK (CAPSULE, "MATCHING CRITERIA")

  If a message with characteristics satisfying "MATCHING CRITERIA" has 
  arrived, copy its description into message capsule CAPSULE.  The message is 
  removed from the system "message queue" and may not be probed or received by 
  another thread unless explicitly released using the release routine.  
  "Information" routines can be called using CAPSULE to get information about 
  the message.  A receive can also be posted for this message by the thread 
  that owns CAPSULE (after modifying CAPSULE using information from the "info" 
  routines).  If a "matching" message has not arrived, return immediately 
  without modifying CAPSULE.  Return value indicates whether a message has 
  arrived or not.  

MPI_PROBE_WAIT (CAPSULE, "MATCHING CRITERIA")

  Same as MPI_PROBE_CHECK() except that the routine will not return until a 
  "matching" message arrives.  

MPI_RELEASE (CAPSULE)

  The previously probed message described by message capsule CAPSULE is put 
  back in the system "message queue".  It may now be probed or received by 
  any thread.  An error will be returned if the message has not been probed.  
  Data description of arrived message in CAPSULE is invalidated.  

4.5  Simplified Versions of Send and Receive Routines

Good arguments have been made (we think) for "simplified" versions of send 
and receive routines that are less flexible and "easier to use".  These 
routines could be built on top of the routines proposed above or implemented 
directly.  The simplified routines proposed below cannot be used with contexts 
or probe or "info" routines.  Only contiguous user data is supported.  
Receiving process must know message data description.  Actually, it is not 
clear to us that these routines are really easier to use.  They do conform 
more closely to "common practice" though.  Simplified versions of the check 
and wait routines can also be made (we've left them out-- this thing is long 
enough already!).  

MPI_SEND_SYNCH_SIMPLE (DESTINATION, TAG, ARRAY, LENGTH, DATA_TYPE)

  Send  message consisting of LENGTH elements from location ARRAY to process 
  DESTINATION using message tag TAG.  DATA_TYPE is the data type for ARRAY.  
  (Pre-defined values for DATA_TYPE are MPI_INT, MPI_REAL, etc.  DATA_TYPE is 
  required for heterogeneous communication.)  This is a "synchronous" send.  

MPI_SEND_BLOCKING_SIMPLE (DESTINATION, TAG, ARRAY, LENGTH, DATA_TYPE)

  Same as MPI_SEND_SYNCH_SIMPLE() except that this is a "blocking" send.  

MPI_SEND_NON_BLOCKING_SIMPLE (DESTINATION, TAG, ARRAY, LENGTH, DATA_TYPE)

  Same as MPI_SEND_SYNCH_SIMPLE() except that this is a "non-blocking" send.  

MPI_RECV_BLOCKING_SIMPLE ("MATCHING CRITERIA", ARRAY, LENGTH, DATA_TYPE)

  Receive the message with characteristics satisfying "MATCHING CRITERIA" and 
  store message data in array ARRAY.  Received data type must be DATA_TYPE.  
  Data length must be LENGTH.  This is a "blocking" receive.  Error 
  conditions will be returned if LENGTH or DATA_TYPE are not consistent with 
  the content of the received message.  

MPI_RECV_NON_BLOCKING_SIMPLE ("MATCHING CRITERIA", ARRAY, LENGTH, DATA_TYPE)

  Same as MPI_RECV_BLOCKING_SIMPLE() except that this is a "non-blocking" 
  receive.  

5  Code Fragment Examples in "C"  

Here are some code fragment examples of how programs using message capsules 
might look.  Lots of stuff has been left out (some variable declarations and 
initializations, error control using return values, etc.) for brevity.  Code 
segments for sending and receiving processes are shown separately for 
clarity.  "Matching" criteria for receives is left undefined.  Process ID is 
assumed to be an integer.  

5.1  Blocking Send -> Receive Vector Data Item:  Receiver Knows Data 
     Description

  DECLARATIONS FOR SENDING AND RECEIVING PROCESSES:  

Capsule *capsule;                             /* pointer to message capsule */
int i, tag, receiver_pid;
float my_vector[1000];                                         /* user data */

  SENDING PROCESS CODE:  

...  
MPI_CREATE_CAPSULE(capsule);                      /* create message capsule */
...  
MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector);     /* initialize capsule */
...  
for (i=0; i<10; i++)                                       /* loop 10 times */
    {
    ...  
    compute(my_vector, ...);                 /* user does something to data */
    MPI_SEND_BLOCKING(capsule, receiver_pid, tag);  /* send msg to receiver */
    ...  
    }  /* end of i for loop */
...  
MPI_DESTROY_CAPSULE(capsule);      /* destroy capsule before end of program */
...  

  RECEIVING PROCESS CODE:  

...  
MPI_CREATE_CAPSULE(capsule);                      /* create message capsule */
...  
MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector);     /* initialize capsule */
...  
for (i=0; i<10; i++)                                       /* loop 10 times */
    {
    ...  
    MPI_RECV_BLOCKING(capsule, "MATCHING CRITERIA");     /* receive message */
    mess_with(my_vector, ...);             /* user does something with data */
    ...  
    }  /* end of i for loop */
...  
MPI_DESTROY_CAPSULE(capsule);      /* destroy capsule before end of program */
...  

5.2  Blocking Send -> Receive Vector Data Item:  Receiver Does Not Know Data 
     Description

  DECLARATIONS FOR SENDING AND RECEIVING PROCESSES:  

Capsule *capsule;                             /* pointer to message capsule */
int tag, receiver_pid, vec_length, num_items, item, data_item;
float my_vector[1000];                                         /* user data */

  SENDING PROCESS CODE:  

...  
MPI_CREATE_CAPSULE(capsule);                      /* create message capsule */
...  
MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector);     /* initialize capsule */
...  
MPI_SEND_BLOCKING(capsule, receiver_pid, tag);  /* send msg to receiver */
...  
MPI_DESTROY_CAPSULE(capsule);      /* destroy capsule before end of program */
...  

  RECEIVING PROCESS CODE:  

...  
MPI_CREATE_CAPSULE(capsule);                      /* create message capsule */
...  
MPI_PROBE_WAIT(capsule, "MATCHING CRITERIA"); /* probe until message arrives*/
/* Get data description of arrived message using "info" routines and */
/* initialize message capsule. */
num_items = MPI_INFO_NUM_DATA_ITEMS(capsule);   /* get number of data items */
/* (Resetting capsule is optional if this is the only place this occurs in */
/* the code.  Resetting is necessary if this code fragment (excluding */
/* capsule creation and destruction) is in a loop.  Resetting does not */
/* affect information contained in the capsule about the arrived message.) */
MPI_ATTACH_RESET(capsule);
if (num_items == 1)                  /* Deal with single data item message. */
    {
    /* What kind of data item is in the message? */
    data_item = MPI_INFO_DATA_ITEM_DESCRIPTION(capsule, 0);
    if (data_item == REAL_VECTOR_ITEM)  /* Deal with real vector data item. */
        {
        /* Get length of vector item and store in "vec_length". */
        MPI_INFO_VECTOR_ITEM (capsule, 0, &vec_length);
        if (vec_length <= 1000)        /* Data fits in existing user array. */
            {
            /* Initialize first data item to be "vec_length" floats stored */
            /* at location "my_vector". */
            MPI_ATTACH_REAL_VECTOR(capsule, vec_length, my_vector);
            }  /* end of vec_length if */
        else
            {
            ...            /* allocate a larger buffer to hold message data */
            }  /* end of vec_length else */
        }  /* end of data_item if */
    else
        {
        ...            /* deal with other possibilities for first data item */
        }  /* end of data_item else */
    }  /* end of num_items if */
else
    {
    ...                                    /* deal with multiple data items */
    }  /* end of num_items if */
MPI_RECV_BLOCKING(capsule, "MATCHING CRITERIA");         /* receive message */
...  
MPI_DESTROY_CAPSULE(capsule);      /* destroy capsule before end of program */
...  

5.3  Blocking Send -> Non-Blocking Receive Complicated Message With Context:  
     Receiver Knows Data Description

  DECLARATIONS FOR SENDING AND RECEIVING PROCESSES:  

Capsule *capsule;                             /* pointer to message capsule */
int i, tag, receiver_pid, my_context;
float my_vector[1000], my_array[2000];                         /* user data */
int my_indices[200];                                          /* user data */

  SENDING PROCESS CODE:  

...  
MPI_CREATE_CAPSULE(capsule);                      /* create message capsule */
...  
MPI_SET_CONTEXT (capsule, my_context); /* my_context must be set up earlier */
MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector);     /* initialize capsule */
MPI_ATTACH_REAL_STRIDE (capsule, 10, 20, 200, my_array);
MPI_ATTACH_INT_VECTOR(capsule, 200, my_indices);
...  
MPI_SEND_BLOCKING(capsule, receiver_pid, tag);      /* send msg to receiver */
...  
MPI_DESTROY_CAPSULE(capsule);      /* destroy capsule before end of program */
...  

  RECEIVING PROCESS CODE:  

...  
MPI_CREATE_CAPSULE(capsule);                      /* create message capsule */
...  
MPI_SET_CONTEXT (capsule, my_context);  /* must be same as sender's context */
MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector);     /* initialize capsule */
MPI_ATTACH_REAL_STRIDE (capsule, 10, 20, 200, my_array);
MPI_ATTACH_INT_VECTOR(capsule, 200, my_indices);
...  
MPI_RECV_NON_BLOCKING(capsule, "MATCHING CRITERIA");        /* post receive */
...                                                       /* "do something" */
MPI_WAIT (capsule);                           /* wait for message when done */
...  
MPI_DESTROY_CAPSULE(capsule);      /* destroy capsule before end of program */
...  

5.4  Blocking Send -> Receive With "Simplified" Routines

  DECLARATIONS FOR SENDING AND RECEIVING PROCESSES:  

int i, tag, receiver_pid;
float my_vector[1000];                                         /* user data */

  SENDING PROCESS CODE:  

...  
for (i=0; i<10; i++)                                       /* loop 10 times */
    {
    ...  
    compute(my_vector, ...);                 /* user does something to data */
    /* send msg to receiver */
    MPI_SEND_BLOCKING_SIMPLE (receiver_pid, tag, my_vector, 1000, MPI_REAL);
    ...  
    }  /* end of i for loop */
...  

  RECEIVING PROCESS CODE:  

...  
for (i=0; i<10; i++)                                       /* loop 10 times */
    {
    ...  
    /* receive message */
    MPI_RECV_BLOCKING_SIMPLE ("MATCHING CRITERIA", my_vector, 1000, MPI_REAL);
    mess_with(my_vector, ...);             /* user does something with data */
    ...  
    }  /* end of i for loop */
...  

6  Advantages and Disadvantages of "Message Capsule" Approach  

6.1  Advantages

  - The number of separate send and receive routines is greatly reduced 
    without sacrificing functionality.  
  - A user who is used to "common practice" can use the simplified 
    versions of the routines.  
  - A user who wants more flexibility only needs to learn about the features 
    required for his/her specific application.  (For example, if I only need 
    contiguous messages, then I don't need to know anything about strided data 
    items.  If the receiving process always knows the data description of 
    received messages, then I don't need to know about the probe and "info" 
    routines.)  
  - "Hidden states" are removed so multi-threaded applications won't get 
    confused.  
  - Encapsulation of features in message capsules allows new features to be 
    added later without modifying syntax of existing routines.  A new feature 
    would require addition of one or more new routines to modify and examine 
    message capsules.  

6.2  Disadvantages

  - The use of message capsules does not conform exactly to "common 
    practice".  It's fairly close to PVM though.  
  - There is no way to prevent a user from messing with a message capsule 
    directly.  This is the same problem encountered in C with (FILE *) 
    structures.  The problem can be partially alleviated by including "magic 
    numbers" in message capsules.  
  - Possible slight performance degradation due to "casing" in receive 
    routines.  
  - Data description must be sent along with each message.  This is only a 
    significant problem if lots of short messages are sent by a program 
    running on a machine with low communication latency.  Sending data 
    description information could be avoided if we do not allow a receiving 
    process to be ignorant of the data description of incoming messages.  

7  Unresolved Issues  

  - Can the message capsule concept work well with collective communication 
    routines?  
  - Is it really necessary to support multiple data items in a message 
    capsule?  It is if we want to allow mixed-type messages.  

8  Code Fragment Example With Even More Information Hidden In the Message 
   Capsule

The code fragment presented in Section 5.1 has been modified slightly to 
indicate how the capsule could possibly contain additional information to 
further reduce the number of send/receive calls and the number of 
parameters.  Message tag and "matching" criteria for receive operations are 
now "hidden" in the message capsule.  Communication "mode" (synchronous, 
blocking, or non-blocking) is used to reduce the number of send and receive 
routines.  Communication "mode" is also "hidden" in the message capsule.  

Four new routines have been added.  Routine MPI_ASSIGN_TAG() sets message 
tag.  Routines MPI_SET_SEND_MODE() and MPI_SET_RECV_MODE() set communication 
"mode" (synchronous, blocking, or non-blocking for send, blocking or 
non-blocking for receive).  Routine MPI_SET_MATCH() sets matching 
characteristics for receive operations.  The send routines (MPI_SEND_SYNCH(), 
MPI_SEND_BLOCKING(), and MPI_SEND_BLOCKING()) have been combined into one 
routine (MPI_SEND()).  The receive routines (MPI_RECV_BLOCKING() and 
MPI_RECV_NON_BLOCKING()) have been combined into one routine (MPI_RECV()).  
Both MPI_SEND() and MPI_RECV() have a reduced number of parameters.  

In the example below, lines containing "###" indicate changes from the example 
in Section 5.1.  

(Example from 5.1) Blocking Send -> Receive Vector Data Item:  Receiver Knows 
                   Data Description

  DECLARATIONS FOR SENDING AND RECEIVING PROCESSES:  

Capsule *capsule;                             /* pointer to message capsule */
int i, tag, receiver_pid;
float my_vector[1000];                                         /* user data */

  SENDING PROCESS CODE:  

...  
MPI_CREATE_CAPSULE(capsule);                      /* create message capsule */
...  
/* ### Following 4 lines added ### */
MPI_ASSIGN_TAG (capsule, tag);             /* Assign the tag to the capsule */
/* Set send mode to "blocking".  Valid values for send mode are */
/* MPI_MODE_SEND_NON_BLOCK, MPI_MODE_SEND_BLOCK, and MPI_MODE_SEND_SYNC. */
MPI_SET_SEND_MODE (capsule, MPI_MODE_SEND_BLOCK);
MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector);     /* initialize capsule */
...  
for (i=0; i<10; i++)                                       /* loop 10 times */
    {
    ...  
    compute(my_vector, ...);                 /* user does something to data */
    /* ### tag parameter disappears from parameter list ### */
    MPI_SEND(capsule, receiver_pid);                /* send msg to receiver */
    ...  
    }  /* end of i for loop */
...  
MPI_DESTROY_CAPSULE(capsule);      /* destroy capsule before end of program */
...  

  RECEIVING PROCESS CODE:  

...  
MPI_CREATE_CAPSULE(capsule);                      /* create message capsule */
...  
/* ### Following 5 lines added ### */
/* Set matching criteria in capsule */
MPI_ASSIGN_MATCH(capsule, "MATCHING CRITERIA");
/* Set the receive mode to "blocking".  Valid values for receive mode are */
/* MPI_MODE_RECV_NONBLOCK and MPI_MODE_RECV_BLOCK. */
MPI_SET_RECV_MODE (capsule, MPI_MODE_RECV_BLOCK);
MPI_ATTACH_REAL_VECTOR(capsule, 1000, my_vector);     /* initialize capsule */
...  
for (i=0; i<10; i++)                                       /* loop 10 times */
    {
    ...  
    /* ### tag parameter disappears from parameter list ### */
    MPI_RECV(capsule);                                   /* receive message */
    mess_with(my_vector, ...);             /* user does something with data */
    ...  
    }  /* end of i for loop */
...  
MPI_DESTROY_CAPSULE(capsule);      /* destroy capsule before end of program */
...