Subject: parkbench meeting

From: Jack Dongarra <[email-masked]>
Date: Wed, 22 Mar 1995 13:52:26 -0500

Here is a draft agenda for the ParkBench meeting on Monday.

	Agenda for PARKBENCH Meeting on 27 March '95
	---------------------------------------------

1) Minutes of Last Meeting

2) Status of Parkbench - see below for more detail

	Reports and discussion from subgroups
	Analysis subgroup 
	Status of Codes/repository on WWW
	Communication within ParkBench

3) Funding of ParkBench Activities

4) ParkBench Reports/Publications

5) Date of next meeting

-------------------------------------------------------------------------

The ParkBench (Parallel Benchmark Working Group)
will meet in Knoxville, Tennessee on
March 27th, 1994.

The SPEC-HPSC people will be having their meeting at the Hilton
after the Parkbench meeting and you are welcome to attend that
meeting as well. The dates for the SPEC-HPSC meeting are
March 28-29, 1995.

The meeting site will be the Knoxville Downtown Hilton Hotel.
We have made arrangements with the Hilton Hotel in Knoxville.

  Hilton Hotel
  501 W. Church Street
  Knoxville, TN
  Phone:  615-523-2300

When making arrangements tell the hotel you are associated with
the Parallel Benchmarking or Spec Meeting. The rate about $68.00/night.
You can download a postscript map of the area by anonymous ftp'ing
to netlib2.cs.utk.edu, cd shpcc94, get knx-downtown.ps.

You can rent a car or get a cab from the airport to the hotel.

We should plan to start at 9:00 am March 27th and finish about 5:00 pm.
If you will be attending the meeting please send me email so we can better
arrange for the meeting.

-------------------------------------------------------------------------

2) Status of Parkbench - In more detail:-

Status of Codes/repository on WWW

	- See "Other comments on status of codes" below

	- WWW repository needs much work
		Restructure pages, page hierarchy could be for e.g.

			Rules for running Benchmarks

			Source Codes
				low-level
					...
				kernels
					Matrix,
					...
				CA's
				HPF

			Method for reporting bugs

			Method for reporting results

	- Page containing source codes on WWW, very confusing at present.
		- Codes present which do not belong to Parkbench
		- Difficult to work out where low-level, kernels etc are.
	 	- Old versions e.g. low-level should reflect GENESIS 3.0
		- Update all to latest versions
		- Overall release no. for Parkbench codes?
		- Check if documentation included

Analysis subgroup
	Alistair Dunlop has some results from analysis of a kernel

Communication within ParkBench
	Duplicate work e.g. on low-level benchmarks
		perhaps people should email committee with details of intended 
		work at outset, so they can be warned if duplicating work?
	Strategy for answering questions mailed to whole committee

Benchmark Results

	Few results received
	GBIS - Tennessee mirror - script sent but I have little documentation 
	       to send as yet.

	Regular reports? see item 4) on agenda

________________________________________________________________________
Other comments on status of codes:-

from Mark Baker.

1) Timer - common timer in all codes use something akin to GENESIS 
   host/node library which encapsulates and wraps system dependent 
   routines for the timer and date stamp.

2) Header - common header file - like GENESIS

3) Makefile - common makefile GENERIC makefiles that work even with the 
   least commonly used version of make.

4) I/O - common methodology of input data: command-line/file/#include

5) Documentation - more, with better description of how input
   parameters affect memory usage etc - perhaps recommendations
   of PVM environmental setting should be made.

6) Standard problem sizes (small, medium and large).

7) PD PVM vs Vendor specific versions of PVM  - The latter often 
   lacks the functionality of the former - (pvm_spawn, pvm_joingroup) 

8) Adding RESULTS to database important.

9) NAS paper and pencil versus PARKBENCH versions of the codes !!!

10) Rules/guidelines for running codes (baseline and optimised).

11) All codes should have test input and output to certify results.

12) Obvious bug/problem reporting mechanism

13) Someone responsible for the repository - update and stabilise
    existing codes. 

14) Recommended subset of codes to run... (to many at moment !!)

----------------------------------------------------------------------
   The single most important thing that is not explicitly mentioned, is that
the codes should use a common timer!  In fact, they should also use something
akin to the GENESIS node library, which encapsulates the system dependent
routines needed for the timer, and time of day.  The codes should also all
print out a standard header in the results file, with information like the
name of the benchmark, the date and time of the run, the system it was run on,
etc., as is done by the SETDTL() routine in the GENESIS library.

   Other things that it would be nice to see would be consistently named
executables and result files, and a policy on makefiles.  Some people advocate
single source benchmarks with no makefile, but I personally think that that
is unrealisitic, and it would be better if each benchmark were supplied with
a makefile based on a "standard" makefile template, such as the one that comes
with PVM.  The PVM versions of the GENESIS codes use such a template.  Even if
you're running on a system that doesn't have "make", however unlikely that may
be, having a makefile at least documents how to build the executable, and if
it's based on a template, then peculiarities of individual benchmarks are easy
to spot.

   Depending how much work people are prepared to put into it, there are other
things that could be made more uniform, such as the means of defining input
parameters - interactively from standard input, from a steering file, or
compiled into the program being three options.  Not all the benchmarks are
adequately documented, and not all come with a well defined set of "standard"
problem sizes - the Matrix kernels being a particular case in point.