Subject: Minutes from 11-3-95

From: "Michael W. Berry" <[email-masked]>
Date: Sat, 11 Nov 1995 23:09:33 -0500

-----------------------------------------------------------------
Minutes of Parkbench Meeting - Knoxville Hilton, November 3, 1995
-----------------------------------------------------------------
Attendee List:

           Mark Baker       Syracuse University   [email-masked]
           Michael Berry    Univ. of Tennessee    [email-masked]
           Jack Dongarra    Univ. of Tenn./ORNL   [email-masked]
           Tony Hey         Univ. of Southampton  [email-masked]
           Edgar Kalns      IBM                   [email-masked]
           Kenneth Koch     Los Alamos Nat'l Lab  [email-masked]
           Mike Kolatis     Univ. of Tennessee    [email-masked]
           David MacKay     Intel SSD             [email-masked]
           Ron Sercely      Convex                [email-masked]
           Fiona Sim        IBM                   [email-masked]
           Erich Strohmaier Univ. of Tennessee    [email-masked]
           Chegu Vinod      Convex                [email-masked]
           David Walker     Oak Ride Nat'l Lab    [email-masked]
           Pat Worley       Oak Ride Nat'l Lab    [email-masked]

At 9:07am EST, Tony H. opened the meeting and itemized the agenda items for the
day:
1. Minutes
2. Results
3. Electronic Benchmarking Journal
4. SC'95
5. Run Rules
6. Policy
7. AOB (Any Other Business)

Minutes from March 1995 meeting accepted with a minor correction to
indicate that QCD is in fact not in the first release
of PARKBENCH.  The 'Action items' from the May video meeting between UTK and
Southampton were then reviewed.  Jack D. suggested that most efficient version
of PVM be used on SP2.

At 9:25am, Tony H. reviewed the Results & Codes.  We should have a release 
number on the codes.  Mark B. was asked to comment on low-level codes.  They 
are available in PVM version (latest version).  Results are available on most
systems (SPMD).  Codes are debugged as of a month.   Fiona S. was
concerned that results may not be "acceptable" - run rules are not clear.  Jack
D.  suggested that a file be provided for each code to specify input 
parameters/files and that an output file could be sent to PARKBENCH for
validation.  Should IBM be constrained to publish PVM-E results or
can MPL results be used?  Tony H. asked that Mark B. work on making sure
low-level run rules available.  Jack suggested that a subcommittee be formed
to create a formal set of run rules for PARKBENCH codes.  Outcome could be 
reported electronically to resolve them and announce them at SC'95.  A review
of the results needs to be specified.
Ken K. asked who will be assigned to run the codes?  Tony H. suggested
(like Genesis) that vendors and academics can both run codes and produce
results.  There was a general discussion on how to report results but this was
curtailed for later discussion during the day. Jack D. suggested that users
can use vendor-supplied PVM not generic PVM (T3D has several versions of PVM).
Tony H. concurred with this philosophy.  The issue is really how the codes are
invoked not really the performance of the measured tasks.
the measured tasks.  Ron S. reiterated that vendors would be allowed to
replace/modify part of code related to task creation/management.  Tony H.
suggested that a subcommittee should be created to resolve such issues.

Tony H. then asked Mike K. to summarize the status of the kernels:

Lowlevel:
   Single processor
      TICK1, TICK2 (interactive), RINF1, POLY1, POLY2
      (non-PVM, f77)
      gettimeofday - used for timing (wall-clock time is the requirement);
      timer calls are isolated for replacement by other timers. 
      Tony H. indicated that users should be allowed to replace timers and
      this can be noted in run rules.
   Multiple processor
      COMMS1 
      COMMS2 
      COMMS3
      POLY3
      SYNCH1
      Machines run on: CM5(UTK,LANL), Convex, SP2 (Ames), CS2, T3D,
      Paragon(ORNL), Sun4, DEC Alpha, IBM RS/6000, HP, SGI.
      ** No results on KSR machines *
  Linear Algebra Kernels:
      Versions w/ PVM-BLACS, MPI-BLACS.
      Some PVM versions spawn tasks which cause problems on machines like
      CM5, T3D.   T3D has problems w/ BLACS (SCALAPACK group looking into
      these problems).   PVM 3.3.7,3.3.8 are available.  Version 3.3.10 was
      just released.  Need to specify a minimal PVM version (3.3.6?). 
      Tony H. requested that recommendation be made or users.  Mike K.
      suggested version 3.3.8 should be the base version.
   NAS (based on PVM)
      C codes (ala V. Sunderum at Emory Univ. and his colleagues)
      EP, IS, MG (recursive memory calls)
      Ames will provide a MPI-f77 version to PARKBENCH soon - so group must
      decide which version(s) to support in the suite.  These will be NAS
      Parallell Benchmarks version 1.5 (in MPI).

      f77 codes: CG (memory concerns), FT

    Compact Applications:  (4 CA's with PVM and MPI)
    installed ->   NPB (MPI,PVM) - 3 codes (LU,SP,BT)
                   ARCO (PVM)  -- "future directory"
                   POLMP(f77,PVM,HPF,PARMACS) -- "future directory"
    installed ->   SHALLOW (f77,PVM,MPI)
                   HPF (in a separate directory)
                   *** dropped SOLVER, 1-DFFT, GAMESS ***

All levels of benchmarks will be available in PVM and MPI: 2 sources for each 
code.  Jack D. suggested that we should have 2 files - 1 per version.  Pat W.
mentioned that "wrappers" could be used as an alternative.  Ken K. pointed
out that real application writers use wrappers.  No final decision on wrappers
was made at this time.

---------------------------------------
At 11:15am, group took a 5-minute break.
---------------------------------------

Tony H. summarized status afterwards at 11:30am:

o low-level codes (PVM,MPI-soon)
    Results on SP,Cray,Convex,Paragon,SGI

o Kernels (MPI,PVM soon)
    LinAlg
    NAS

o Compact Applications (MPI,PVM soon)
    LU,SP,BT :=NAS
    SHALLOW    

*** MPI versions are the primary focus with optional PVM versions available ***

Ken K. was then asked to present his results on the T3D at LANL.
He pointed out the differences with MPP PVM and NOW PVM.  For example,
no spawns allowed, pvm_parent90 is useless to determine the master.
Could not get LinAlg to work on T3D - complicated call tree for benchmarks.
Fortran-to-C char strings on CRI machines must use special FCD type.  BLACS
are different on CRI machines.  For NAS/EP, the LANL results with Problems
A and B were consistently about 17 times slower than the NAS reported results.
For NAS/MG the results do not scale well.  For the small number of processors,
the results were more comparable with NAS report.  NAS CG could not be run due
to memory  requirements - appeared to designed for NOW's and not MPP's with 
respect to memory.  For NAS/FT, the LANL results were about 5 times slower.
For NAS/IS, when replaced gettimeofday with "rtclock()" the results were more
comparable (remove system calls for timer).   Ken K. then summarized issues
for refining the PARKBENCH's version of NAS for running on T3D.  Results on
T3D were obtained by Sowmini Varadhan (PhD student from UT interning at
LANL this past summer).

At 11:50am, Tony H. then asked for a general discussion on results & codes from
attendees.  Ron S. shared similar concerns about NOW-based design of codes.
Mapping of memory per processor configuration is needed (Ken K. concurred with
Convex on this).   Fiona S. was very concerned about the differences between
Ken's results on the NAS results with those reported in the NAS report.  
Fiona S. indicated that it is difficult to access what is a "real"
indication of performance and that if the vanilla starting point
was not a reasonable one, not only would it discredit the
vendor's results, but also the NAS PB themselves.  Fiona S. also pointed
out the comparison was not made with the codes which are going into the 
Parkbench suite.  There is a new set of MPI based codes from NASA, and no
data has been collected from those codes.  Tony H. stressed that the database
be available at SC'95.

-----------------------------------------------------------
At 12:07pm EST, Tony H. suggested the group break for lunch.
-----------------------------------------------------------

At 1:25pm EST, the group reassembled and Tony stressed the current
                ***  PARKBENCH ACTION ITEMS ***

o Release 1.0 of PARKBENCH codes and results by December 3rd.

1. PVM spawning/MPI run problems (Jack D + David W./Eric S. have agreed to
    address this problem - Jac D. takes lead)
2. Low-level codes: have PVM and will convert to MPI soon (Eric S + Mike K./
   Mark B.) will handle this conversion and isolate timers in codes.

3. Kernels: BLACS problems related to T3D and will get NAS NPB 1.5 MPI version
   and address size problems (e.g., NAS CG) and remove NOW-based bias; a README
   file to specify memory requirements per processor configuration (Jack D. +
   Eric S./David B. will address the size problems and README files; don't worry
   about the Class C problems for now)

4. Compact Applications; NAS 3 applications + SHALLOW with MPI and PVM
    optional versions soon  (David W. + Pat W. / David B + Eric S. will
    address these codes)

5. Must decide the "Run Rules" for the codes (Mike B. + Jack D. will take SPEC 
   document, modify it and circulate with vendor input) - generally specify 
   sizes and that specific details for the parameters are in the README files
   per codes.  Fiona S. suggested that the problem sizes scale.   Reference to
   the individual codes.  

   Will put NAS (MPI) results from David Bailey; this release will have a
   mixture of PVM and MPI-based codes.  Specifically , here's the first release:

   ------------------------------------
   |  PARKBENCH RELEASE 0.9  (Nov. 10)|
   ------------------------------------
   | Low-level - PVM versions only    |
   | Kernel       - LinAlg (MPI,PVM)  |
   |                NAS    (MPI)      |
   | Compact                          |
   | Applics      - NAS   (MPI)       |
   |                SHALLOW (MPI,PVM) |
   ------------------------------------
   Note: RELEASE 1.0 at SC'95 will fill 'holes'  + spawning rewrites

6. Results:

           Release 0.9 (codes)        ... as of Nov. 10
           Run Rules                         

    Current
    Databases:      MPI            PVM             Vendor-Opt
                    ---            ----             ----------
                                   NAS
                                   SHALLOW

---------------------------------------------------------------------------
Machine assignments:

SP2       :   NASA runs NPB's, Southampton runs low-level, Daresbury?
Convex    :   ?Kentucky, ?Convex, ?NCSA, Ken K.  
SGI       :   Jack D. will ask Mark Brown at SGI, Tony H. will see about 
              Power Challenge results (moderate number of processors),    
              Ken K. will contact Paul Woodward at Univ. of Minnesota.
Intel     :   ORNL's Paragon (Pat W./David W.)
Cray      :   Edinburgh's 256-processor system - Tony H. to get MPI results
---------------------------------------------------------------------------

Source of current and future results:

         SP     Convex     SGI      Intel      Cray       IBM
         --     ------     ---      -----      ----       ----
Low-Lvl  x        x                    x       Tony H.

LinAlg   IBM? or NAS                                      Mike  K.
         Daresbury

NAS Ker. NAS/MPI                     NAS/MPI    Tony H/MPI

NAS App. NAS                         NAS        Tony H/MPI

SHALLOW  Pat W.                      Pat W.      Pat W.

Second tier machines: Meiko CS2, Parsytec, CM-5, KSR, DEC Turbolaser (8600
                      server)

At 2:30pm EST, Jack D. announced that there will be a joint SC'95 BOF session on
Tuesday  (12/05) at 5:15 pm in Room 8 with SPEC-HPSC.   Jack then passed around
David Snelling's draft of a joint code policy with SPEC-HPSC.  Fiona S.
suggested that there should be another PARKBENCH meeting at SC'95 to discuss 
results and sensitive information.  Myron Ginsberg's BOF session is at 6:30pm.
Jack D. and Tony H. felt the second Parkbench meeting should be closed-door
and can be at 6pm on Wednesday, December 6.
Tony H. then mentioned the existence of an Electronic Benchmarking
Journal in Europe and US with Jack D. and Tony H. as co-editors.  Associate 
editors are being solicited and journal would be scheduled to being in 
January '96.  Suggestions for a title are welcome.

Tony H. will try to modify the current Joint Code Policy and a motion to
update the Parkbench subgroups was made.  Jack D. and Tony H. agreed that
PARKBENCH database can be mirrored in the UK (perhaps at Southampton).

-------------------------------------------
Tony H. adjourned the meeting at 2:55pm EST.
-------------------------------------------