[WIP] Various updates to CI

[adrienbernede]

• Turn on MPI tests in CI

[adrienbernede]

@daboehme do you know why mpi tests are failing only on tioga (cray-mpich) and lassen (spectrum-mpi) ?

[daboehme]

I haven't dug through the logs yet but I suspect it's because Lassen and Tioga need to run MPI jobs on a compute node via lrun / flux run, which the Caliper CI script doesn't do. It just runs an MPI program as-is since it only needs one rank. For now I'd turn off the MPI tests on Tioga/Lassen (there's a CMake switch to do it, -DRUN_MPI_TESTS=Off) although in the long run it would be good to handle this properly.

[adrienbernede]

@daboehme Thank you for the feedback.
In this PR, I was only trying to turn on MPI test on Tioga and Lassen, since they were turned off only for those two.
As you suspect, there is no trace of using "MPIEXEC_EXECUTABLE" in Caliper build system or test driver, which may explain why MPI tests are not "cross platform".
Looking into your testing framework, I was hoping that using BLT could provide an answer to your need. However, it looks like you have your own test driver, written in python, which appears to prevent us to leverage BLT in a straightforward fashion.
For example, in Umpire, BLT handles the various MPI providers depending on variables.
Back to Caliper, the Spack package already sets MPIEXEC_EXECUTABLE in the CMakeCache file, making the CMake config ready to run MPI tests, provided that the project uses BLT (or BLT-like) MPI support.
Something to think about when trying to support MPI in your tests.
I’ll open an issue related to that, if you don't mind.