LAPACK Working Note 112:
Practical Experience in the Dangers of Heterogeneous Computing

L. S. Blackford, A. Cleary, J. Demmel, I. Dhillon , J. Dongarra,
S. Hammarling, A. Petitet , H. Ren , K. Stanley , R. C. Whaley

Abstract:

Special challenges exist in writing reliable numerical library software for heterogeneous computing environments. Although a lot of software for distributed memory parallel computers has been written, porting this software to a network of workstations requires careful consideration. The symptoms of heterogeneous computing failures can range from erroneous results without warning to deadlock. Some of the problems are straightforward to solve, but for others the solutions are not so obvious, or incur an unacceptable overhead. Making software robust on heterogeneous systems often requires additional communication.

This paper addresses the issue of writing reliable numerical software for networks of heterogeneous computers. We describe and illustrate the problems encountered during the development of ScaLAPACK and the NAG Numerical PVM Library. Where possible, we suggest solutions to avoid potential pitfalls, or if that is not possible, recommend that the software is not used on heterogeneous networks.

 
• Introduction
• Motivation and Background
• Homogeneous and Heterogeneous Computing
• Machine Parameters
• Global Arguments and Floating Point Values
• Communicating Floating Point Values on IEEE Machines
• Considerations Due to Poor Arithmetic
• Algorithmic Integrity
• Closing Remarks
• Acknowledgments
• References
• About this document ...