Subject: PAR93 announcement

From: [email-masked] (Bodo Parady - SMCC Systems Performance)
Date: Wed, 3 Nov 1993 10:21:05 +0800

All of the members of the PARKBENCH group are welcome to participate
in discussion of the PAR93 suite at SC93.  The following is an announcement 
of the suite.  Many thanks to Joanne Martin for scheduling a room that
would not conflict with PARKBENCH.  I hope to see you all in Portland.

						Bodo

PAR93
-----

Charter:

 	PAR93 -- is a benchmark suite designed to measure cache based RISC
		SMP system performance using well behaved codes 
		that parallelize automatically.  While no one benchmark can 
		fully characterize performance,  it is hoped the results of a 
		variety of realistic benchmarks can give valuable insight 
		into expected real performance.

PAR93 and its use in the performance analysis of SMPs will be discussed at:

		Supercomputing 93
		Room A-107, Portland Convention Center
		4:30-6:30  
		Tuesday, November 16, 1993
		1.800.go2.sc93 (1.800.462.7293)

Papers will be delivered by Mike Humphrey of SGI, Vinod Grover of Sun, Bodo
Parady of Sun, and Bruce Leasure of Kuck and Associates.  This will be
followed by a panel discussion which, in addition to the presenters, 
includes Forrest Baskett of SGI, and John Hennessey of Stanford.

	Focus --
 		This suite is for users of small to medium 
 		(<64 cpus) symmetric multiprocessing systems who wish
		to compare the performance of various systems, but who
 		do not wish to change source code to attain parallel
 		speedups.  As such, the benchmarks allow for
 		no tuning of the code beyond the base, and no addition 
 		of compiler directives.
 
 		It is assumed that code tuning through source code
 		changes and the addition of compiler directives can 
 		attain greater speedups, but this is not the scope of 
 		this benchmark.  Neither are distributed memory, vector 
 		architectures, nor massively parallel systems within the 
 		goals of this benchmark, although nothing in the suite
 		prevents any system architecture from being used;
 		other architectures are encouraged to run this benchmark.
		Vector and MP vector architectures may find that this suite
		offers an excellent method to evaluate the performance of
		their systems and are heartily encouraged to run this
		suite.

 		This suite is intended to characterize the subset 
 		of applications for which compilers can parallelize 
 		automatically (compilation without the use of directives). 
		As such this suite offers examples of 
 		parallelizable  code and an indication  of the 
 		speedups attainable from compilers on some applications.
 		This benchmark offers users, system designers and 
 		compiler writers a method to compare their products
 		on a completely level playing field, with no parallel
 		programming paradigm favored.

		PAR93 is based on codes from SPEC CFP92, Perfect Club,
		NASA Ames, ARCO,  We gratefully acknowledge the pioneering
		work by David Bailey, George Cybenko, Dave Schneider, and
		Wolgang Gentzsch, among others, and solicit their guidance 
		in continued evolution of the suite.

Q:	What equipment do I need to run PAR93?

A:	Here is a minimum configuration:

		320 MB Main Memory
		500 MB Swap disk
		1 processor
		An OS
		A Fortran compiler

	Here is a suggested configuration:

		500 MB Main Memory
		1 GB swap disk
		2 processors
		An OS capable of serving multiple OS's (eg. Unix)
		An MP Fortran compiler

Q:	Why is it named PAR93 and not SPECpar93?

A:	It has not been approved by SPEC.   PAR93 has been submitted to
	SPEC for further development.  Before release from SPEC it will
	undergo a rigorous approval process to ensure it is truly portable,
	and provides a level playing field for comparison of certain types
	of systems.  It could become SPECpar94.

Q:	Will PAR93 indicate the performance of SPECpar94?

A:	Not necessarily.  During the SPEC development process, benchmarks
	may be added, dropped, or modified.

Q:	I have strong opinions on what benchmarks ought to be added
	(dropped, or modified).  How can I influence the process?

A:	Get involved.  Bring your ideas, your data and analysis, your
	time and energy to SPEC.  Consider joining SPEC.  Write to the
	following address for more information:

	    SPEC [Systems Performance Evaluation Corporation]
	    c/o NCGA [National Computer Graphics Association]
	    2722 Merrilee Drive
	    Suite 200
	    Fairfax, VA 22031
	    USA

	    Phone:  +1-703-698-9600 Ext. 318
	    FAX:    +1-703-560-2752
	    E-Mail: [email-masked]

	For technical questions regarding the SPEC benchmarks (e.g., problems
	with execution of the benchmarks), Dianne Dean (she is the person
	normally handling SPEC matters at NCGA) refers the caller to an
	expert at a SPEC member company.

Q:	What are the run rules:

A:	The run rules are identical to the SPEC run rules, that is to
	say:

		.	No code changes, except for portability

		.	No compiler directives

		.	No benchmark specific optimizations

	All but the last are easy to enforce and understand.

Q:	Why are compiler directives not allowed, since with my parallel
	system, the addition of one directive would improve 
	performance of typical user codes by a great deal, and it
	only takes a couple of minutes to put in the directives, and
	everyone does it.

A:	All of this may be true, but PAR93 has been constructed with
	with the goal of measuring parallel system performance,
	where a system includes the cpu, cache, memory system, OS, 
	and compiler.  The goal is measurement of system performance
	using codes that parallelize automatically.

	Many codes that parallelize well have been excluded for PAR93
	because in order to parallelize well, they require the
	intervention of the programmer.  Due to the availability of
	many codes that present day compilers can parallelize, the
	run rules were restricted to those that compilers can deal with.
	This eliminates human factors in the comparison of various
	systems.  Most users will find that the codes provided 
	demonstrate excellent speedups on their SMP system.  From
	this he might be able to infer a little about what he
	can do by tuning.

	Barring compiler directives levels the playing field, since
	no directive format is favored and no types of directives
	are favored.  It also spares performance engineers the
	the career agony of constantly tuning benchmarks.  There
	have been other benchmarks that try to deal with user tuning
	like Perfect, and others that try to measure unrestricted CPU
	performance like the TPP linpack, but  PAR93 is
	a benchmark that measuress the compiler, and OS in addition
	to the effects of the hardware.

Q:	Is automatic parallelization mature enough to be the basis
	of a benchmark suite?

A:	Automatic parallelization has been commercially available since 
	1986 on Sequent, and possibly earlier on Alliant machines 
	(their first machine shipped with automatic parallization), and
	on multi-processor Cray machines (in the late 1980's).  Now, 
	there are lots of commercial automatic parallelizers with
	David Kuck and his group at UICU being on of the leading
	academic organizations researching automatic parallelization.

Q:      What application areas are addressed in PAR93?

A:      The following areas are addressed:

                101.tomcatv     2d mesh generation.  This program
                                is the author's modernized version(1991)
                                compared to the SPECfp92 version (1987) of
                                the original tomcatv.  It is also
                                scaled up in size.

                102.swim        The original SPEC program to compare
                                performance of shallow water equations
                                by Swartztrauber and Sato ahs been scaled
                                up to arrays of 513x513 from 256x256.

                103.su2cor      The same quantum physics program
                                using Monte-Carlo methods from Prof. Bunk
                                that is found in SPECfp92.

                104.hydro2d     The same SPECfp92 program that solves
                                the astrophysical Navier-Stokes
                                equations.  The problem is scaled up from
                                102x102 to 127x127.

                105.tfft        David Bailey's NASA FFT set.  The
                                FFT length is 4M elements.

                106.fdmod       Siamak Hassanzadeh's 2d finite difference
                                scheme for 3d seismic wave propagation.
                                This is part of the original ARCO
                                Benchmark Suite.  It was selected
                                because this portion of seismic data
                                processing consume over 80% of the
                                computer cycles.

                107.mgrid       Eric Barszcz and Paul Fredrickson's
                                multigrid solver from NASA Ames.

                109.appbt       Sisira Weeratunga's block ADI solver
                                for nonlinear PDE's from NASA Ames, using
                                block tridiagonal solvers.

                111.appsp       Sisira Weeratunga's block ADI solver
                                for nonlinear PDE's from NASA Ames
                                based on the solution of penta-
                                diagonal systems of equations.

                113.ora         An update of the original SPEC 048.ora
                                to prevent constant propagation by
                                reading input.  The problem size
                                has been updated.  ora is a ray
                                tracing program.

                117.nump        Portions of Peter Montgomery's
                                number theory program that
                                parallelize.

Q:	I want to use profiling tools and beautiful GUI's to 
	assist in parallization.  Why does this suite not
	encourage their use and allow me to demonstrate them?

A:	On the contrary, this suite will get the attention of 
	parallel computer vendors, and every GUI and tool will be
	applied to get better compilers out.  Some users like GUIs
	and tools, but a greater majority prefer compilers.

	Furthermore, there could be future evolutions of PAR93
	that address the more advanced issues of parallelization.

Q:	What is the reporting methodology?

A:	The principal is to report both raw performance and scaling
	of MP systems.  To the first level, one configuration
	must be reported, for example the run times for 8 cpus
	applied to the PAR93 suite could be reported.  To a second
	level, all of the performance for 1,2,4, and 8 ( or any sequence
	of cpus, with up to 5 configurations in a single report) can be 
	reported.  Scaling with the number of processors is displayed, but 
	as with the present SPEC benchmark, the single figure of merit
	is the ratio to a reference system, in this case a Sun 
	SPARCstation 10 Model 41 with 40 MHz SuperSPARC and 1MB external
	cache.

	The baseline uniprocessor values for the suite are:

		Benchmark		Seconds

		101.tomcatv		1171.5
		102.swim		1220.6
		103.su2cor		 161.5
		104.hydro2d		 990.5
		105.tfft		2628.4
		106.fdmod		 929.2
		107.mgrid		 630.6
		109.appbt		2424.4
		111.appsp		2476.2
		113.ora			 832.0
		117.nump		1005.1

	This takes less than 4 hours to run on a single cpu 40.33MHz
	SS10/41.
		
Q:	Some of the benchmarks are not full applications.  Why not
	substitute full applications for these abstractions?

A:	Some, like SWIM, are abstractions from full programs, but
	issues of availability, code size, and test cases
	have prevented SPEC from using full applications in
	all cases.  These same limitations apply to the PAR93 suite.
	As this suite matures, and SPEC members have their input
	and are able to contribute more to this suite, more
	full applications are expected.

Q:	Some codes like tomcatv, represent 2D problems when in
	reality, 3D mesh generation and 3d Fluid problems are
	more realistic.  Why not use more modern programs.

A:	There is an effort to port a 3d mesh generator, but this
	will take time given its huge size.  When complete, this could add to
	the quality of the suite.  Consumers of PAR93 realize
	that this is the first generation, and that future 
	generations aim to  attain much higher levels of quality.

Q:	What about the duplication of similar numerical techniques
	such at swim and fdmod using explicit finite differences?

A:	Even though swim and fdmod have similar methods, they
	represent different application areas, and the methods
	and program setup are quite different.

	Compare this for example to SPECfp92 where mdljsp2 and
	mdljdp2 are simply the same program, but one is 
	single precision and the other is double precision.

	The comparison of results from the two similar programs
	can provide insights not otherwise available.

Q:	Why not report two numbers for this suite?  One obtained
	by autoparallelization and the other obtained by hand
	parallelization or the addition of directives?

A:	This is similar to the reporting rules of PERFECT and
	some of the PARKBENCH suite.  The purpose here has
	been stated above.  Reporting results in this manner
	could be a possible rule  for a future suite.

Q:	What about adding codes that have subroutine and loop
	level parallelism that encourage the use of directives
	and having various granularities?

A:	This is a good suggestion for a future suite.

Please direct responses to:

	Bodo Parady		[email-masked]
	Michael Humphrey	[email-masked]

Bodo Parady                   | (415) 336-0388
SMCC, Sun Microsystems        | [email-masked]
Mail Stop MTV15-404           | Domain:  [email-masked]
2550 Garcia Ave.              | Alt:     [email-masked]
Mountain View, CA  94043-1100 | FAX:     (415) 336-4636