Subject: Re: Precise meaning of MPI_COMMIT

From: [email-masked]
Date: Wed, 21 Apr 93 15:25:14 PDT

David Walker writes:

> So my suggestion is that maybe we should stipulate that no data gets moved
> around until the call to mpi_commit_buffer.

My understanding is that no data gets moved until the send call is
made.  Otherwise I see no use for persistent buffer descriptors.
The document should state this explicitly, however.

This solves David's problem on the send side.

However, it does NOT solve the corresponding problem on the receive
side of David's application.  

The receive buffer descriptor must be generated before the receive
call, thus before any of the data is transferred to user variables,
thus the descriptor cannot depend on contents of the message.

(Aside.. This problem could be addressed by allowing descriptors to
contain pointers to values (as opposed to containing the values
themselves), or by allowing messages to be received in pieces.  The
latter is a more general approach, e.g. it allows a later portion
of a message to be received using an expression derived from
an earlier part.  However, all of this is MUCH different from
commonly accepted practice in message-passing systems.  I for one
would vote early and often against such extensions.)

I'm with Moose about being puzzled by the usage of David's example.

I'd guess that the code David shows is designed to be efficient by
avoiding an extra copy to rearrange the data after receiving it.

But that's only worthwhile if one assumes that MPI can rearrange the
data faster by using the buffer descriptors (one per particle!) than
his application could by using application-specific data.

Maybe I'm jaded, but this seems highly optimistic to me.  
Certainly it would be an extraordinary Fortran runtime system 
that would handle

  READ (IUNIT) (J(I),I=1,N),(A(J(I)),I=1,N)

anywhere near as fast as

  INTEGER J(N)
  REAL DATA(N), A(*)
  READ (IUNIT) J, DATA
  DO I = 1,N
    A(J(I)) = DATA(I)
  ENDDO

My guess is that the application would run faster, be easier to
understand, and use less total memory if it didn't ask MPI to do
so much.

--Rik
----------------------------------------------------------------------
[email-masked] (alias 'd39135')   Rik Littlefield
Tel: 509-375-3927                         Pacific Northwest Lab, MS K1-87
Fax: 509-375-6631                         P.O.Box 999, Richland, WA  99352

---

From: [email-masked] (Tom Henderson)
Date: Wed, 21 Apr 93 17:16:08 MDT

Rik writes,

> thus the descriptor cannot depend on contents of the message.

I seem to remember that we actually voted against providing support in MPI for 
reception of a message when the receiver is not aware of message contents 
(data types, numbers of items, etc.).  I think this was the meeting before 
last?  

Tom Henderson

---

From: [email-masked] (Anne C. Elster)
Date: Thu, 22 Apr 93 03:47:10 -0400

>Suppose I have an array of particles:
>
>struct { int type;
>         double position[3];
>         double velocity[3];
>} particle[1000];
>
>and I want to take the positions of all the particles of type=1 and type=2 and
>send them somewhere else. In addition I want to also send the number of
>particles of each type that I'm sending. And I only want to send one message.
>Can MPI do this?
>
>Here's one approach that I think doesn't work according to the current
>version of MPI.
>
>      int count1=0;
>      int count2=0;
>      mpi_create_buffer (handle, MPI_PERSISTENT);
>      mpi_add_contiguous(handle, count1, 1, MPI_INT);
>      mpi_add_contiguous(handle, count2, 1, MPI_INT);
>      for (i=0;i<1000;i++) {
>         if ( particle[i].itype == 1 ) {
>            count1++;
>            mpi_add_contiguous (handle, particle[i].position, 3, MPI_DOUBLE);
>         } else if ( particle[i].itype == 2 ) {
>            count2++;
>            mpi_add_contiguous (handle, particle[i].position, 3, MPI_DOUBLE);
>         }
>      }
>      mpi_commit_buffer (handle);

----

Am I interpreting this code right in assuming the syntax would
read as follows (assuming the March 25 draft) or have I been "sleeping"
and missed something major?
 
      int count1=0;
      int count2=0;
      mpi_create (handle, MPI_BUFFER, MPI_PERSISTENT);
      mpi_add_block(handle, count1, 1, MPI_INT);
      mpi_add_block(handle, count2, 1, MPI_INT);
      for (i=0;i<1000;i++) {
         if ( particle[i].itype == 1 ) {
            count1++;
            mpi_add_block(handle, particle[i].position, 3, MPI_DOUBLE);
         } else if ( particle[i].itype == 2 ) {
            count2++;
            mpi_add_block(handle, particle[i].position, 3, MPI_DOUBLE);
         }
      }
      start(handle);
----

>This doesn't work because MPI only guarantees that the message associated
>with the handle is no longer volatile once  mpi_commit_buffer returns. So
>the above example will fail if an implementation actually copies count1 and
>count2 to the message buffer when mpi_add_contiguous is called. The example
>will work in an implementation that copies the data into the message buffer
>only when mpi_commit_buffer is called.
>

Why are you calculating "count1" and "count2" within the mpi_add_xxx
loop? It seems, to me , that it is, in general, a bad idea to be
changing data as the buffers are being created. It seems to me that
your program could have figured out these values before the loop.

>around until the call to mpi_commit_buffer.

Sounds good to me, although I agree with "moose" and still worry about
the practice.

>
>BTW this is not a contrived example. You need to do something similar in
>molecular dynamics simulations in which you cells containing differing
>numbers of particles between processors.

Why do you NEED this? I am "deadly" curious since I also do particle
codes ...  Seems to me that the situation you are describing above
is sending some particles (say ions=type1, electrons = type2) that have
wandered out of their local cell(s) in a PIC simulation. Surely their
numbers can be kept track of earlier? 

BTW, the above code segment seems to loose the individual particle's
type info. unless you are assuming (reasonably, I may add) that all electrons
are numbered before ions, or vice versa.

Pardon me. if I am missing something -- it is 3:45am in the morning...

						Anne C. Elster
[email-masked]

>Regards,
>David

---

From: [email-masked] (David Walker)
Date: Thu, 22 Apr 1993 08:32:10 -0400

>The document should state this explicitly, however.

Yes the reuse of persistent buffer descriptors is a more fundamental reason why
data should not be moved before the commit. Once the description of the
message (using MPI_ADD_XXXX) and the composition of the message (using
MPI_COMMIT_BUFFER, or better MPI_COMPOSE) are clearly separated then it
is no longer clear why you need a commit/compose routine, since the
assembly/disassembly of the message would be done by the send/receive routines.

So I propose that the draft makes clear the difference between describing
a message and composing it, and that MPI_COMMIT_BUFFER be deleted.

>thus the descriptor cannot depend on contents of the message.

Isn't this just the usual problem that you have when you don't know
how many things you're going to receive. You just have to provide
a buffer that's big enough to handle the largest number of things that
you expect to receive.

>would vote early and often against such extensions.)

Me too.

>avoiding an extra copy to rearrange the data after receiving it.

My example was trying to use buffer descriptors to send a message
containing mixed data types. I want to send everything in one message
in the (possibly mistaken) belief that it will be faster to send everything in
one message rather than sending one message for the integer counts and
another one for the particle positions. If the time to send a message is
dominated by the time to handle the buffer descriptors then nobody will
use them. Do you think my code would run faster if I cast the counts
as doubles, pack them and the particle positions into a contiguous buffer
myself and then communciate them using MPI_SENDC/MPI_RECVC (the simple
version for contiguous buffer)?

>understand, and use less total memory if it didn't ask MPI to do

Well can we have some examples from real applications in which the
generality and flexibility offered by MPI buffer descriptors can be
used, and still have the code run reasonably efficiently.

David

--------------------------------------------------------------------------
| David W. Walker                 |   Office   : (615) 574-7401          |
| Oak Ridge National Laboratory   |   Fax      : (615) 574-0680          |
| Building 6012/MS-6367           |   Messages : (615) 574-1936          |
| P. O. Box 2008                  |   Email    : [email-masked] |
| Oak Ridge, TN 37831-6367        |                                      |
--------------------------------------------------------------------------

---

From: [email-masked] (Dan Nessett)
Date: Thu, 22 Apr 1993 16:34:23 -0800

Here is another question about MPI_COMMIT. Suppose I want to build a buffer
and reuse it several times, changing the data in it between uses. For
example, I have a data structure similar to the one David Walker suggests :

struct { int type;
         double position[3];
         double velocity[3];
} particle;

struct particle particles[1000];

Instead of sending the data based on the value of type, I simply want to
send the whole array for each iteration of some algorithm. I build the
buffer descriptor as follows :

      mpi_create_buffer (handle, MPI_PERSISTENT);
      mpi_append_hvec(handle, &(particles[0].type), 1000, sizeof(particles),
                      sizeof(particle), 1, MPI_INT);
      mpi_append_hvec(handle, &(particles[0].position), 1000, sizeof(particles),
                      sizeof(particle), 1, MPI_DOUBLE);
      mpi_append_hvec(handle, &(particles[0].velocity), 1000, sizeof(particles),
                      sizeof(particle), 1, MPI_DOUBLE);
      mpi_commit_buffer (handle);

After the MPI_COMMIT, I am not allowed to change the data refered to by the
buffer object. Is this true forever, implying I must execute the
buffer_creation code each time I want to send the buffer? Or can I change
the information in the particles array after it has been sent and provide
the same buffer handle in a new send operation? Similarly, after receiving
information into the particles array and processing it, am I allowed to
resubmit the same buffer handle to a new receive?

Dan