Subject: All about threads

From: Tony Skjellum <[email-masked]>
Date: Sun, 18 Jul 93 10:21:07 CDT

Topic: Tacit Thread Safety Requirement of MPI1, Context Chapter, etc.

Dear colleagues:

    In reviewing comments about the latest context draft, it has
been repeatedly told me that we are at a crucial stage in MPI, 
because we have to agree on the context model, etc, as soon as
possible.  I concur with that assessment.  In trying to find
a consistent way to acquire safe communication space for groups,
the issue of thread safety arises, because overlapping concurrent
threads would have to work correctly.  I am currently confident about the
single-threaded case, and I am NOT CONFIDENT about the multi-threaded case.
Does anyone have real experience with multi-threaded message passing
(has it been done in an application setting, like we assume for MPI)?

    I need immediate guidance (specific guidance) about what multi-threaded
programming MEANS in MPI, if this was in fact a reasonable requirement
for MPI in the first place, and how multi-threading impacts point-to-point
and collective communication (that is, real programs with examples).
For instance, do we assume named threads or unnamed threads (and would
this help).  Is there an examplar threads package ???

    Here is one problem in a nutshell.  We discussed "statically 
initialized libraries" from time to time.  Well, if there are multiple
overlapping threads, then one would need to have separate contexts
statically initialized for each concurrent thread.  Such threads have
group scope.  Hence, groups would have to cache contexts for each
concurrent thread (notice: groups cacheing contexts). 

    I propose that we have a serious discussion on what thread safety
really means for MPI1.  I need for there to be well-formulated guidelines and
in-depth debate immediately, so that the context committee can work
effectively within these requirements, or give feedback as to why they
are unreasonable.  Otherwise, I/we can't really make the context chapter
bullet-proof in time for the next meeting (except for the single-thread
case).

    We have discussed how contexts provide group safety, but not
temporal safety from multiple invocations of operations on a context
(for which a programming paradihm must be described; e.g., synchronizing
or implicitly synchronizing ... also could be called quiescent-at-exit).
Now we need to have a notion of how to provide safety with multiple
threads, or how to program the multi-threaded environment consistently,
with interspersesd MPI calls.  

				-	-	-

    To summarize, I seriously propose that in absence of an in-depth
debate and specification of what thread safety means in MPI1, that we
abandon this requirement altogether (analogous to the abandonment of
non-blocking collective operations).  If thread safety were to remain
a de jure requirement of MPI1, then I ask that there be examples
(analogous to or supersets of our contexts examples, pt2pt examples,
and collective examples) illustrating same.  If this is to be an added
task of my subcommittee [which makes reasonablee sense to me] then I
am eager for assistance nonetheless.  I would want to see what people think
existing thread practice is, what the design choices are, and which we
choose to support, as well.  It is not obvious to me that we really
know what we mean (formally, practically) by "thread safety" for
SPMD/MPMD message passing applications.  Recall that there are at
least three kinds of threads: O/S threads, compiler threads, user threads
(we seeem to really mean the latter in our discussions).

    Thanks + please advise soonest.

				Tony Skjellum

PS References to accessible texts or papers or software (eg, portable
thread packages) are acceptable forms of advice.
    
PPS I would like to have a new draft of the context chapter out by
August 1 (with possible revisions by August 5).  I am getting one
extremely negative set of feedback from a single vendor
representative, and one more balanced feedback (ie, only two people
are communicating with me on the context chapter).  I am not seeing
widespread debate over the context chapter.  This MUST happen now,
between the meetings, since we have our best current draft available.
We will not be successful if we are debating it all again at the next
meeting without careful thought now (eg, on the threads issue).

---

From: Ian Glendinning <[email-masked]>
Date: Sun, 18 Jul 93 19:38:15 BST

> From: Tony Skjellum <[email-masked]>

> (has it been done in an application setting, like we assume for MPI)?

Yes.  Most code written for transputer-based machines is multithreaded.
Until quite recently (a couple of years ago) almost all the machines we had
at Southampton were transputer based, so we developed quite a number of
multi-threaded applications.  We are not unique in this respect.  Transputers
have been very popular in Europe, and there are a large number of programmers
on this side of the atlantic who are used to programming parallel machines in
a multi-threaded style, using synchronous unbuffered message-passing
primitives.

> programming MEANS in MPI, if this was in fact a reasonable requirement

Hmmm - I fear I'm sailing into dangerous waters here, but here goes...
My understanding of what I thought was a consensus on threads is that in a
multi-threaded environment, each thread can use the usual MPI primitives in
the normal way, to communicate with other tasks, oblivious (almost) to any
other threads within its own process.  I say `almost' obvlivious, because
obviously if two threads each try to receive a message with the same source,
tag and context, then only one of them should succeed if a matching message
arrives. I would suggest that MPI stipulate that it be unspecified which
thread would get the message in such a situation.  As far as I can see,
this is the only place where extra semantics creep in.  In particular, I
don't see why contexts should introduce any further complication.  A message
with a particular context value should match a receive specifying the same
context value (and source, tag), regardless of which thread posted it (the
receive).  Different contexts are distinct regardless of any considerations
to do with threads.

A good use for contexts in a multi-threaded environment would be to enable
messages to be directed at particular threads within a process.  Of course,
in such a scheme the threads need somehow to find out which contexts they are
supposed to be using, since contexts are allocated on a per-process (rather
than per-thread) basis.  As far as I'm concerned, this is outside the scope
of MPI as currently defined, and would depend on the details of the
particular threads implementation.  However, an example of how things might
proceed would be if there was initially one thread which allocated a number
of contexts, then created some threads, and passed them each some contexts to
use, in a manner specific to the thread creation scheme.

> and collective communication (that is, real programs with examples).

I don't think we need examples - just some words along the above lines.

> this help).  Is there an examplar threads package ???

I don't think this is an issue, since MPI will not contain any operations
that explicitly affect threads.

> concurrent thread (notice: groups cacheing contexts). 

Not a problem if contexts are allocated on a per-process basis, as I presumed
they would be.  What do other people think about this?  By the way though
Tony, I never did really understand what you meant by "statically initialized
libraries".  I hate to show my ignorance, but this is not a term I have
encountered before - would you care to explain please?

> really means for MPI1.  I need for there to be well-formulated guidelines and

Well, I just had my $0.02 worth...

> abandon this requirement altogether (analogous to the abandonment of

Well let's have some debate then.  I think it would be disastrous not to
address the issue of thread safety.

> am eager for assistance nonetheless.  I would want to see what people think

If my world view is seen as acceptable, then there is no implied extra work
for your subcommittee.
   Ian
--
[email-masked]        Ian Glendinning
Tel: +44 703 593368                  Dept of Electronics and Computer Science
Fax: +44 703 593045                  University of Southampton SO9 5NH England