Subject: MPI task identifiers

From: [email-masked] (Peter Rigsbee)
Date: Tue, 26 Jan 93 17:02:44 CST

At the meeting in Dallas, I asked a question about how tasks were 
identified, referring (for example) to the argument that one would pass
to one of the variants of send.  I was surprised to hear that the October 
draft specified this, and that it was defined as sequential integers 
starting at 0. 

I've scanned through the document since then, and didn't notice where this 
was stated.  (The only indication I saw to such identifiers was an example 
included as "Table 1" on page 9, which listed "PID"s of 0-7.)  Each place 
I'd expect to see a definition, says only "the ID number of the process..." 
and leaves it as that.  But I'm willing to accept that it is there and that 
I just missed it (or the authors forgot to print it ;-).

In any case, I'd like to propose that the final spec should *not* define the
internals of task identifiers.  Please consider this as such a proposal.

I'd appreciate hearing any comments.  Assuming the October spec does
describe what a regular task identifier is, I'd like someone to point out
where and to hear reasons why it is important to dictate a 0..n ranges for
task identifiers.

Thanks for your time.

	- Peter Rigsbee
	  [email-masked]

			PROPOSAL -- OPAQUE TASK IDENTIFIERS

1. Proposal

	TASK IDENTIFIER

	All tasks in MPI are identified by unique integer identifiers that
	are assigned by the system upon which MPI is implemented.  The 
	MPI spec does not define the contents of these identifiers and 
	users writing portable applications should not make any decisions
	about nor perform any operations based on their contents.

2. Discussion

a. The major benefit of a 0..n range of identifiers is that users on many
systems (most notably those supporting one process per CPU) can write
efficient applications without having to deal with groups or virtual 
topologies.  If the relationship between process identifier and process
location is well-defined for a particular system, codes can operate
directly on identifiers (send to "myid()+1", for example, and have the
data sent to a nearest neighbor CPU).

b. The drawback is that the above advantage leads programmers away from
writing efficient, portable code.  A code written using the convenient 
method above may work great on a few systems, acceptably on another set
of systems, and horribly on a third set.  It should get the correct 
answers in each case, but, for most people writing MPI codes, horrible 
performance would be considered a bug.  (This of course assumes that the 
MPI specs for groups and virtual topologies will allow for efficient,
portable applications.  If not, we need to fix those specs, and not
simply introduce a faster alternative.)

c. For an MPP system with one process per CPU, a 0..n assignment is easy
and natural.  For almost any other system, it will be more complex.  For
a network implementation, it may require the use of a "process-identifier
server" or some sort of globally shared "next identifier" variable.
These can be very costly to implement.  It also raises ugly questions about
where and how to map these identifiers to the information actually locating
the task.

d. In contrast, the proposal allows an implementation to use the most
appropriate and efficient method.  It may simply be a Unix PID.  It could
be a range of 0..n integers.  Or it could be a complex encoding of 
host/PID information that would allow a network implementation to get
better performance.  It's up to the implementation.

e. The MPI spec does not address the fact that some implementations may be
provided on top of systems allowing dynamic creation and termination of
tasks.  A 0..n requirement assumes a static system.  What happens when
task goes away?  There is no longer a sequential list from 0..n, so does
this implemenation still adhere to the MPI spec?  (If so, one could argue 
that the 0..n spec is then just a red herring, since one could assign any 
numbers at all, and claim simply that all missing numbers represented 
tasks that never started.)  With the proposal, the answer is simple --
that identifier no longer represents a task, and the implementation still
adheres to the spec.

f. The above proposal does not prohibit implementations from assigning
0..n identifiers, or for users of such implementations relying on this
assignment.  But it makes it clear that such reliance is non-standard and
non-portable.

3. Summary

The above proposal has a number of benefits, particularly when compared
to the 0..n alternative:
	- allows implementations to use the most appropriate and/or 
	  efficient method for assigning task identifiers (including
	  0..n)
	- encourages portable programming that does not rely on operations
	  on task identifiers
	- no confusing semantics for MPI implementations built on top of
	  dynamic systems

---

From: Tony Skjellum <[email-masked]>
Date: Tue, 26 Jan 93 18:24:11 CST

Concerning Peter's proposal...

	It may have been a mistake to simplify the naming of processes
to a single integer when MPI was defined.  Commonly, systems like 
RK, and so on had used pairs of integers on multicomputers and networks
before that; the two numbers did have semantic content: the first number,
a node number, connoted the location of the process.  The second, a PID,
could provide an implementation-dependent way to name a process on a 
processor.  Hence, at the low level, specification of WHERE processes
ran was permitted, but it was not necessary to restrict how they were
named (beyond it being an integer quantity) on a processor.  At a low
level, it is convenient to be able to control process placement.

When processes are created, the user could specify the node and PID
on some systems, with either or both picked by the system on some 
implementations.  Such composite information could then be used as part of
message passing.   nCUBE, for instance, specifies the PID's but
does not restrict the locations (nodes are selected by user).  
The Mosaic system will generally
place processes at will, and a message-passing interface for it will
have neither option.  For scalable spawning of large numbers of identical
processes, some mechanism of this form is needed (user specifies collection
of process locations, gets back identifier describing these as a group;
eg, a common PID).

In early systems, and arguably still, it is important to provide low-level
control over process placement.  Arguably, this need not be done in the
message passing interface, but it must be done portably somewhere in
the system.  Otherwise, it will be difficult to provide guarantees of
performance behavior (ie, force one process per processor, if desired;
clump processes together to get overlapping capability when desired, etc).

Our Zipcode system uses {node,pid} in addressee lists, but these are not
used by the programmer directly, in most message classes (eg, 3D logical
process grids) but are hidden with further mappings to application-relevant
names for processes.  In fact, we would like to generalize these addressee
lists to include other information besides this naming, but that remains
a future-release feature.

It does not matter so much to us what convention names processes, but it
is interesting to have access to semantic information about processes
named by the system (eg, accessors to location, etc).   Such information
would be used when providing services, like Zipcode, that are higher level.
The user might not want them at all for direct use.  
Forbidding a user code or application layer like Zipcode to take advantage
of architecture (albeit transparently to the user code) is a disadvantage.
Typically, process placement remains an aspect of architecture we need
to control.

Just some thoughts...

- Tony Skjellum