The ScaLAPACK routines have been specifically designed to allow for an even distribution of the computational load and thus to achieve the highest possible performance. Therefore, the overall execution time is strongly related to the rate of floating-point operations per second (flop/s) that the slowest processor in the machine configuration can achieve.
This behavior
can easily be
observed if
some factor
slows a
particular
processor of
the system.
Consider, for
instance, a ten-processor machine
configuration.
Suppose
that
nine of the processors
can
deliver a peak
performance
of 100 megaflop/s
(Mflop/s) but that
the tenth processor
can achieve
only 20 Mflop/s.
(On a homogeneous
system, different
versions of the
operating system
and/or memory
capacities, I/O
traffic, or
simply another
user's program
can easily cause
such a performance
degradation.)
On such a ten-processor machine,
the overall ScaLAPACK
peak performance is
thus limited to 200
Mflop/s, whereas the
performance of the
machine with nine 100-megaflop/s processors
is 900 Mflop/s.
Specifically, the most
heavily loaded
processor
controls execution time.
The implications are clear.
If a user's code is
running on nine unloaded
processors and one
processor with a
load factor of 5,
one can observe no
more than a factor
of 
speedup.
Similarly,
it is possible on
some systems to
spawn multiple
processes on a
single processor.
In such a case,
performance is
limited by the
slowest processor,
presumably the one
with the most processes.
For example, if 10
processes are spawned
on 9 identical
processors, the
speedup is limited
to 
.
The load of the machine, in addition to the direct effect of offering a program only a portion of the total cycles, can have several indirect effects. If each processor is individually scheduled, performance can be arbitrarily poor because significant progress is possible only when all processes are concurrently scheduled. A loaded machine may also cause one's data to be swapped out to disk, which can greatly reduce peak performance.