Two debuggers are available from IBM: pdbx is a command-line debugger built on dbx and pedb is a parallel debugger with an X interface. Both debuggers have reasonable though not outstanding interfaces. Neither understands message queues, so that finding out why a program is deadlocked can be difficult, for instance. A third debugger, Totalview, is available from Dolphin Interconnect Solutions[8]. This is probably the best MPI debugger available in IBM systems, but currently it understands only MPICH, not IBM MPI.
Message trace collection and visualization is integrated with IBM MPI. With a command line option or by setting an environment variable, MPI programs can automatically collect message trace information that can be displayed with a tool called vt.