Subject: a note on metrics

From: Vladimir Getov <[email-masked]>
Date: Mon, 24 May 93 08:26:59 BST

Here follows my rough note on metrics, which you may like to consider
in today's meeting. I wish you fruitful and successful discussions.

Best regards,

Vladimir Getov

		    A Few Notes on Metrics
		    ----------------------

1. The elapsed time is the only prime metric.

1.1. It must be the wall-clock time, otherwise there will be some hidden
delays and the interpretation of results may not be correct.

1.2. Do we need to know the components of the wall-clock time? The
computer user will probably answer: - `No, I am only interested in the
total elapsed time to complete my task.' However, if one would like to
improve the application performance as much as possible, the knowledge
of all components of the wall-clock time is necessary. This information
is essential for computer architects, but is also very useful for
computer users. As usual, these components include the calculation time,
the communication time, the memory-access time, the I/O time, the system
time, the wait time, etc. 

2. In order to define the secondary (derivative) metrics I would suggest
answering the question: -`What is our goal? Why we need benchmark
measurements?'  There is a few answers to this question and I believe 
every answer defines its appropriate derivative (secondary) performance 
metrics. My current `goals' list includes:

     - Comparative benchmarking analysis of different parallel machines;

     - Analysis and comparison of different algorithms, methods and
     program implementations for a given application;

     - Benchmarking analysis and identification of bottlenecks for
     particular parallel architecture (shell we include the compiler
     benchmarks, etc. here?).

In my opinion the most useful derivative benchmarking metrics are the 
benchmark performance, the speed-up and the efficiency. Concerning
the speed-up and the efficiency I would think about different
definitions. For example, it seems very useful to calculate the speed-up
as the ratio of the elapsed time of two different algorithms on the same
computer. This defines straight-away which one is faster for given
problem size and number of processors. The efficiency could be defined
as the ratio of the benchmark performance to the peak performance. This
gives a clear picture of how far the measurement results are from the
theoretical peak. In fact the list of derivative metrics may be quite
long and it would be difficult to decide which are the important and
useful metrics to be included in the PBWG documents.

[email-masked]                  Dr Vladimir Getov 
Tel: +44 703 593368                  Dept of Electronics and Computer Science
Fax: +44 703 593045                  University of Southampton 
                                     Southampton SO9 5NH, U.K.