Subject: Meeting Minneapolis 18th November, 1992

From: Christine Collier <[email-masked]>
Date: Mon, 11 Jan 1993 17:25:46 +0000 (GMT)

Parallel Benchmark Working Group

                                              Summary of 1st Meeting held in Minneapolis
Convention Centre
                      Wednesday November 18th, 1992

1.   Introduction and Welcome

     The meeting was opened by Professor Roger Hockney who welcomed all the
     participants and asked Tony Hey to say a few words about the background to the
     meeting.
     Tony Hey outlined the history of attempts in Europe to establish credible and
     useful benchmarks for the evaluation of Distributed Memory MIMD systems.  At
     the time the European Genesis work began in 1988 there were no suitable DM
     message-passing benchmarks - although the Caltech group also undertook a study
     at about the same time.  Four years later the scene has moved on, and besides the
     Genesis benchmarks for message-passing Fortran programs, HPF/Fortran 90
     benchmarks are now appearing.  The Perfect Club are also now looking at
     benchmarks for DM systems and the NASA Ames 'Pencil and Paper' benchmarks
     are being taken seriously by the  vendors.  From a situation where there were very
     few suitable benchmark codes we are now approaching a situation where there is
     an over-abundance of benchmarks and inevitable duplication of effort.  This is
     undesirable both for the vendors and for the end users of systems. With the
     adoption of HPF and the current proposal for a standard message-passing interface
     (MPI), discussed yesterday at the message-passing standards workshop, there
     seems to be a real window of opportunity to gather together US and European
     benchmarkers to agree on a useful subset.

     Roger Hockney then asked those present to introduce themselves and make some
     remark about their interest in this activity.  A list of attendees is attached to
this
     summary.  There was widespread support for such an activity and a summary of
     contributions as follows:
     
     Horst Simon of NASA Ames suggested that we follow the HPF model and form
     working groups in identified areas.  He also voiced reservations about any attempt
     to create a fully comprehensive benchmark suite. 

     Denis Parkinson of TMC supported the activity but pointed out that it was
     important that this did not become "yet another" set of benchmarks for vendors to
     implement.  He also stressed the need for HPF versions as well as message-passing
     versions, and raised the question of scalability of benchmarks.

     Aard van der Steen raised the question of Japanese participation in such an
     activity.  

     Siamak Hassanzadeh of Fujitsu supported the proposal and suggested the
     inclusion of seismic benchmarking codes.

     Rolf Hempel of the GMD in Sankt Augustin stressed his organization's support
     for the activity and briefly described the RAPS benchmarking initiative  in Europe. 
     This is a consortium of users and software houses, led by ECMWF and including
     ESI and AVL, supported by Convex, Cray, Fujitsu, IBM, Intel and Meiko RAPS is
     an acronym for Real Applications on Parallel Systems.  The Genesis benchmark
     codes are included as a public domain subset of RAPS but the major codes would
     not be publicly available.

     Bill Minto of Cray UK stressed that real applications were important and that it
     was also necessary to address as wide a spread of applications as possible.

     Geoffrey Fox of Syracuse suggested that it was only realistic at this stage at
     attempt to co-ordinate benchmarking activities.  His activities at Syracuse had
     
     lately been concerned with constructing benchmarks for validating HPF and
     Fortran 90 compilers.

     Trevor Chambers of NPL, UK gave his support to the activity and announced that
     a major new European benchmarking project PEPS with NPL were involvement
     has just started.

     Gordon Harp of DRA, Malvern UK said that the DRA (Defence Research Agency)
     were involved in a collaboration with agencies in the US, Canada and Australia
     looking at benchmarks for defence applications.  He was concerned that there was
     a need for real applications and issues such as scalability and power consumption
     were important from the DRA perspective.

     Francis Wray of Parsytec and Paul Garrett of Meiko both voiced concern that this
     activity should not generate more rather than less work in procurement activities.

     David Schneider from CRSD Illinois, representing the Perfect Club, welcomed
     an attempt to eliminate redundant effort.  He stressed that benchmarks had
     multiple uses - such as education and compiler evaluation - as well as specific
     application knowledge.  He thought that there may be budget problems if a very
     organised activity was envisaged.  Benchmarks spanning a range of architectures
     and the need for public domain codes were also stressed.

     Joanne Martin of the HPSSL IBM Kingston welcomed the initiative and stressed
     how important it was  for vendors not to be confronted with many sets of
     "standard" benchmarks.

2.   Objectives

     There were no objections to the draft objectives for the group.  These objectives
     are:

       1.   To establish a comprehensive set of parallel benchmarks that is generally
            accepted by both users and vendors of parallel system.

       2.   To provide a focus for parallel benchmark activities and avoid
            unnecessary duplication of effort and proliferation of benchmarks.

       3.   To set standards for benchmarking methodology and result-reporting
            together with a control database/repository for both the benchmarks and
            the results.

3.   Mode of Working

3.1  It was agreed that on HPF-like forum style of working should be adopted with a
     view to convergence to an agreed set of benchmarks and procedures within 12
     months.

3.2  There was not seen to be a need for meetings every six  weeks but in order to
     generate momentum for the project it was thought that two meetings a year were
     too few.

3.3  Jack Dongarra volunteered to set up a database for benchmarks and results at
     ORNL.  NPL were willing to maintain a European copy of this database.

3.4  It was agreed that all present should send existing benchmarks to Jack Dongarra
     at ORNL.

3.5  Jack Dongarra and Aard van der Steen agreed to examine the available
     benchmarks submitted and attempt to classify them appropriately.

3.6  Three other working groups were identified with named individuals taking
     responsibility for each group.  These were as follows:

       Methodology:             Bailey, Hockney and Schneider
       Kernel Benchmarks:       Dongarra, Hockney, van der Steen, Wray
       Compiler Benchmarks:          Fox, Grassl

3.7  A number of application areas were discussed as possible working groups e.g.
     CFD, Seismic, QCD etc., but it was thought premature to activate such groups at
     this time.

4.   Future Activity

4.1  Jack Dongarra agreed to set up a mail refector at ORNL for the Parallel
     Benchmark Working Group (PBWG) and to organize the relevant subgroups along
     the lines of the HPF forum.

4.2  Jack Dongarra also agreed to host the next meeting of the PBWG.  Subsequent
     discussions after the formal close of the meeting led to the dates of March 1st/2nd
     being selected.

4.3  At the March meeting, each subgroup will produce a discussion document and a
     benchmark database classification will be proposed.  Further discussion on
     procedures for the PBWG was also deferred until then.