PVM creates files in /tmp for logging and locking. These will collide if different machines share the same tmp.
Get a PVM release version 3.2.3 or later. It includes a compile-time option to change the names of the files in /tmp to include the host name, avoiding the conflict.
If you want this option to be enabled for all machine types, edit src/Makefile.aimk and add -DSHAREDTMP to the (uncommented) OPTIONS= line near the top. If you want it set only for certain architectures, edit the config files for those machines (conf/*.def) and add -DSHAREDTMP to each ARCHCFLAGS= line.
You'll need to rebuild PVM and relink any applications that use libpvm.