CS 594-31883 Understanding Parallel Architectures:
From Theory To Practice Spring 2000 - 3 credits - CS Library, Room 105, Ayres Hall

Jack Dongarra, Professor; with help from David Walker, Graham Fagg, Shirley Browne, and Erich Strohmaier.

Email: dongarra@cs.utk.edu
Phone: 865-974-8295
Fax: 865-974-8296
Office hours: Wednesday 11:00 - 1:00, or by appointment

TA: Ken Roche ,

320 Science and Engineering Bldg.
Phone: 865-974-9989

To find out more about this Course click here.
Lecture Notes: (Tentative outline of the class)
  • Jan 12 (NOTE: This class will meet on Friday, January 15th in room 217 Ayres Hall at 1:30.)
    Slides: Course Introduction
    Slides: Overview of High Performance Computing
    Reading: Overview of Scientific Computing
  • Jan 19 David Walker lecture on MPI.
    Slides: Message Passing and MPI
    Slides: Additional material on MPI
    Example: MPI program1
    Example: MPI program2
    Example: MPI program3
  • Jan 26
    Slides: Introduction to Parallel Architectures and Programming Models
    Slides: Floating Point Arithmetic
    Reading: Draft notes on Parallel Programming Considerations
  • Feb 2
    Slides(postscript version): Memory Hierarchy and Cache
    Slides(pdf version): Memory Hierarchy and Cache
  • Feb 9 Erich Strohmaier - Performance Eval and OpenMP
    Slides(postscript version): Performance Eval and OpenMP
    Slides(pdf version): Performance Eval and OpenMP
  • Feb 16 Memory Hierarchy and Cache (Continued)
    Slides(postscript version): Memory Hierarchy and Cache (Continued)
    Slides(pdf version): Memory Hierarchy and Cache (Continued)
    Not yet ready.
  • Feb 23 Linear Algebra
    Slides(pdf version): Linear Algebra
  • Mar 1 Linear Algebra (Continued)
    Slides(pdf version): Linear Algebra (Continued)
  • Mar 8 David Walker - Applications
    Slides(ps version): Applications
    Slides(pdf version): Applications
  • Mar 15 David Walker - Applications
    See slides from March 8th.
  • Mar 22
    Spring Break
  • Mar 29 Graham Fagg -
    Metacomputing parts ( 1 , 2 , 3 , 4 )
  • Apr 5 Graham Fagg -
    Metacomputing 2 parts ( 1 , 2 , 3 , 4 )
  • Apr 12 - More on Grid Computing
    Slides (pdf version) Grid Computing
  • Apr 19
    Grids 2
  • Apr 26 Shirley Browne - Parallel Debugging
    ( cs594-debug.html ) ( totalview-arl.ppt )
  • May 5
    Class Projects Reports
  • May ? Final exam
    • Assignments
  • Homework 1: Due January 25, 2000
    Home Work problems to be solved.
  • Homework 2: Due February 16, 2000
    Home Work problems to be solved.
  • Homework 3: Due February 23, 2000
    Home Work problems to be solved.
    Data for homework.
  • Homework 4: Due March 1, 2000
    Home Work problems to be solved.
  • Homework 5: Due March 22, 2000
    Home Work problems to be solved.

  • Projects (due May 5, 2000)
    Preliminary list of projects click here.
    • Additional Reading Materials

    Message Passing Systems.

    The PVM home page.


    The MPI home page.
    This is the best place to get info on MPI from, including implementations and the MPI forum itself.

    The implementation that you should use, that is installed on the TORC (Tennessee Oak Ridge Cluster) cluster is MPICH .
    A duplex postcript version of the MPI 1.1 API (with thanks to the LAM team) is available.

    Other useful reference material


  • A good introduction to message passing systems.


    J.J. Dongarra, G.E. Fagg, R. Hempl and D. Walker, Chapter in Wiley Encyclopedia of Electrical and Electronics Enginerring,
    to appear. ( postscript version )

    ``Message Passing Interfaces'', Special issue of Parallel Computing , vol 20(4), April 1994.

  • A paper by members of the PVM team on the differences between PVM and MPI.

    Geist, G.A, J.A. Kohl, P.M. Papadopoulos, `` PVM and MPI: A Comparison of Features '', Calculateurs Paralleles , 8(2), pp. 137--150, June, 1996.

  • Papers by members of the MPI team on the differences between PVM and MPI.

    ``Why are PVM and MPI So Different'', William Gropp and Ewing Lusk (submitted to The Fourth European PVM - MPI Users' Group Meeting)

    and

    ``PVM and MPI are completely different'', William Gropp and Ewing Lusk, to appear in the journal Future Generation Computer Systems, 1998.


    Standard Books on Message Passing

    ``MPI - The Complete Reference, Volume 1, The MPI-1 Core, Second Edition'',
    by Marc Snir, Steve Otto, Steven Huss-Lederman, David Walker, Jack Dongarra, MIT Press, September 1998, ISDN 0-262-69215-5.

    ``Using MPI,''
    by William Gropp, Ewing Lusk, and Anthony Skjellum, published by MIT Press, October 1994; ISBN 0-262-57104-8.

    ``MPI: The Complete Reference - 2nd Edition: Volume 2 - The MPI-2 Extensions'',
    by William Gropp, Steven Huss-Lederman, Andrew Lumsdaine, Ewing Lusk, Bill Nitzberg, William Saphir, and Marc Snir, published by The MIT Press, September, 1998; ISBN 0-262-57123-4.

    Class Roster If your name is not on the list or some information is incorrect, please send mail to TA: .
    Student Names and Their Email addresses





    On-line Documentation and Information about Machines

  • Cray
  • IBM RS6000
  • Intel
  • Intel ASCI Red Paragon
  • SGI Power Challenge
  • Solaris Threads page
  • Catalog of Commercial Hardware and Software Vendors
  • Convex
    • Exemplar
    • Michielse, P. Programming the Convex Exemplar Series SPP system. Parallel Scientific Computing. First International Workshop, PARA '94. Proceedings. Lyngby, Denmark, 20-23 June 1994). Edited by: Dongarra, J.; Wasniewski, J. Berlin, Germany: Springer-Verlag, 1994. p. 374-82.
  • Cray Research
  • Digitial Equipment Corporation ( System Info
  • Hewlett-Packard
  • IBM
  • Sequent
    • Symmetry 5000
    • NUMA-Q
    • Raetz, G.M., Sequentz, G.M., Sequent general purpose parallel processing system, Northcon/87. Conference Record. Portland, OR, USA, 22-24 Sept. 1987)
  • Silicon Graphics
    • Power Challenge
    • Power Challenge (Techical Report)
    • Challenge XL
    • Galles, M.; Williams, E. Performance optimizations, implementation, and verification of the SGI Challenge multiprocessor. Proceedings of the Twenty-Seventh Hawaii Internation Conference on System Sciences Vol. I: Architecture, Wailea, HI, USA, 4-7 Jan. 1994, Edited by: Mudge, T.N.; Shriver, B.D. Los Alamitos, CA, USA: IEEE Comput. Soc. Press, 1994. p. 134-43.
    • Power Series
  • Sun Microsystems
    • Other Parallel Information Sites
  • NHSE - National HPCC Software Exchange
  • Netlib Repository at UTK/ORNL
  • BLAS Quick Reference Card
  • LAPACK
  • ScaLAPACK
  • GAMS - Guide to Available Math Software
  • Center for Research on Parallel Computation (CRPC)
  • Supercomputing & Parallel Computing: Conferences
  • Supercomputing & Parallel Computing: Journals
  • High Performance Fortran (HPF) reports
  • High Performance Fortran Resource List
  • Fortran 90 Resource List
  • CMU's list of supercomputing and parallel computing resources
  • Major Science Research Institutions from Caltech
  • Message Passing Interface (MPI) Forum
  • High Performance Fortran Forum
  • OpenMP
  • PVM
  • Parallel Tools Consortium
  • DoD High Performance Computing Modernization Program
  • DoE Accelerated Strategic Computing Initiative (ASCI)
  • National Computational Science Alliance
    • Related On-line Textbooks
  • Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods, SIAM Publication, Philadelphia, 1994.
  • PVM - A Users' Guide and Tutorial for Networked Parallel Computing, MIT Press, Boston, 1994.
  • MPI : A Message-Passing Interface Standard
  • LAPACK Users' Guide (Second Edition), SIAM Publications, Philadelphia, 1995.
  • MPI: The Complete Reference, MIT Press, Boston, 1996.
  • Using MPI: Portable Parallel Programming with the Message-Passing Interface by W. Gropp, E. Lusk, and A. Skjellum
  • Parallel Computing Works, by G. Fox, R. Williams, and P. Messina (Morgan Kaufmann Publishers)
  • Computational Science Education Project TextBook.
  • Designing and Building Parallel Programs. A dead-tree version of this book is available by Addison-Wesley.
  • High Performance Fortran (HPF), a course offered by Manchester and North High Performance Computing Training & Education Centre, United Kingdom
  • Parallel Processing Laboratory, Colorado School of Mines

    • For performance analysis:

  • Raj Jain, The Art of Computer Systems Performance Analysis. John Wiley, 1991.

    • Papers on performance analysis tools:

  • Ruth A. Aydt, "The Pablo Self-Defining Data Format," November 1997, click here.
  • Daniel A. Reed, Ruth A. Aydt, Tara M. Madhyastha, Roger J. Noe, Keith A. Shields, and Bradley W. Schwartz, "Pablo: An Extensible Performance Analysis Environment for Parallel Systems", November 1992, click here.
  • Jeffrey K. Hollingsworth, Barton P. Miller, Marcelo J. R. Gongalves, Oscar Naim, Zhichen Xu and Ling Zheng, "MDL: A Language and Compiler for Dynamic Program Instrumentation", International Conference on Parallel Architectures and Compilation Techniques, San Francisco, CA, November 1997, click here.
  • Barton P. Miller, Mark D. Callaghan, Jonathan M. Cargille, Jeffrey K. Hollingsworth, R. Bruce Irvin, Karen L. Karavanic, Krishna Kunchithapadam and Tia Newhall. "The Paradyn Parallel Performance Measurement Tools", IEEE Computer 28(11), (November 1995). click here.
  • Steven T. Hackstadt and Allen D. Malony, "Distributed Array Query and Visualization for High Performance Fortran, February 1996, click here.
  • Jerry Yan and Sekhar Sarukkai and Pankaj Mehra, "Performance Measurement, Visualization and Modeling of Parallel and Distributed Programs using the AIMS toolkit", Software Practice and Experience 25(4), April 1995, 429--461

    • Jack Dongarra
    1/8/2000