-
-
Notifications
You must be signed in to change notification settings - Fork 19
Notes on using common debugging tools
Please remember that ddd/gdb are serial debuggers therefore they can only debug processes running on the same node, not on multiple nodes.
Examples of how to use gdb/ddd to debug mpi applications:
mpiexec -np 4 xterm -e gdb ./parmmg_debug
mpiexec -np 2 ddd ./parmmg_debug
When using DDD, one can visualize the contents of specific arrays like this:
graph display parmesh->listgrp[2]
graph display parmesh->listgrp[0].node2int_node_comm_index2[0]@17
According to the openmpi faq, valgrind can be used to check mpi applications like this:
mpirun -np 2 valgrind --leak-check=full --show-leak-kinds=all --read-inline-info=yes --read-var-info=yes ./parmmg_debug -np 2 m.mesh
mpirun -np 1 valgrind --leak-check=full --suppressions=/usr/share/openmpi/openmpi-valgrind.supp ./parmmg_debug m
A suppressor file might be needed to silence false warnings. These can be reduced by:
- compiling valgrind with --with-mpicc
- making sure that the mpi wrapper library is used by valgrind
For example, on debian the following package provides the wrapper library valgrind-mpi and the above commands to execute valgrind become:
LD_PRELOAD=/usr/lib/valgrind/libmpiwrap-amd64-linux.so mpirun -np 2 valgrind --leak-check=full --show-leak-kinds=all ./parmmg_debug m
LD_PRELOAD=/usr/lib/valgrind/libmpiwrap-amd64-linux.so mpirun -np 1 valgrind --leak-check=full --suppressions=/usr/share/openmpi/openmpi-valgrind.supp ./parmmg_debug m
If the valgrind mpi wrapper library is successfully installed and loaded, messages similar to these:
valgrind MPI wrappers 8293: Active for pid 8293
valgrind MPI wrappers 8293: Try MPIWRAP_DEBUG=help for possible options
appear at the beginning of the valgrind output.
The information in this section as well as more information on how to extend the valgrind mpi wrapper library can be found at the relevant entry of valgrind's manual.
valgrind --db-attach=yes ./parmmg_debug
The output of using the massif profiler can be found in files named massif.out.$PID. More info at the relevant valgrind web page
export VALGRIND_LIB=/usr/lib/valgrind/
valgrind --tool=massif ./parmmg_debug coarse
mpirun -n 2 valgrind --tool=cachegrind ./parmmg_debug
perf stat -B
The MPICH library offers a memory allocation verification tool. This is their wiki entry on how to use it. In short they are controlled by:
- Configure MPICH with --enable-g=mem (you may combine mem with other debugging options, such as --enable-g=dbg,log,mem)
- Use the following environment variables MPICH_TRMEM_VALIDATE or MPICH_TRMEM_INITZERO (set to the value YES or yes) to change the behavior of the memory allocation routines)
CLang offers a static analyzer tool: clang-tidy.
This tool requires a compilation database file of the project to be generated at compile time (a file named by default compile_commands.json ). CMake can generate such a file either if
- 'set(CMAKE_EXPORT_COMPILE_COMMANDS "ON")' exists in the CMakeLists.txt file
or if
- configure ccmake with -DCMAKE_EXPORT_COMPILE_COMMANDS.
Having generated the compilation database file, the tool can be used simply by:
/usr/bin/run-clang-tidy-3.9.py > CLANG-TIDY.warnings.txt
Google's address sanitizer tool can be enabled with certain version of the clang and gcc compilers by adding
-fsanitize=address -O1 -fno-omit-frame-pointer
in the compiler and linker flags. More compile time and run time options can be found on the tool's AddressSanitizerFlags wiki page
To remotely launch a session of Arm DDT:
- Open the Arm Forge remote client on your local machine.
- Launch the application on your remote machine with:
$parent_folder_of_your_arm_installation/arm/forge/bin/ddt --connect mpirun $your_program $your_program_arguments
Always check that you are running the same version of remote client (on the local machine) and remote installation.
- Make a copy of the shell scripts compilervars.sh and amplxe-vars.sh (available under $intel_installation_path/compilers_and_libraries/linux/bin/ and $intel_installation_path/vtune_amplifier/, respectively - Look for your Intel installation with module show).
- Modify compilervars.sh in order to avoid setting Intel MPI environment variables. Look for lines containing "mpivars.sh" and comment them, as follows:
if [ -e "${PROD_DIR}/ippcp/bin/ippcpvars.sh" ]; then
. "${PROD_DIR}/ippcp/bin/ippcpvars.sh" ${INTEL_TARGET_ARCH} ${INTEL_TARGET_PLATFORM}
fi
#if [ -e "${PROD_DIR}/mpi/intel64/bin/mpivars.sh" ]; then
# if [ "${INTEL_TARGET_ARCH}" = "intel64" -o "${COMPILERVARS_ARCHITECTURE}" = "intel64" ]; then
# . "${PROD_DIR}/mpi/intel64/bin/mpivars.sh"
# fi
#fi
if [ -e "${PROD_DIR}/pkg_bin/compilervars_arch.sh" ]; then
. "${PROD_DIR}/pkg_bin/compilervars_arch.sh" ${INTEL_TARGET_ARCH} ${INTEL_TARGET_PLATFORM}
fi
- Load the Intel compiler (there is no need to actually compile ParMmg with it!) and source the two scripts compilervars.sh and amplxe-vars.sh before launching ParMmg with the following command (compiler version, architecture, and paths may change):
module load compiler/gcc/9.1.0
module load compiler/intel/64/2019_update4
source ~/compilervars.sh intel64
source ~/amplxe-vars.sh
mpirun amplxe-cl -collect hpc-performance -trace-mpi -r ./vtune_results $your_parmmg $your_parmmg_arguments
# [Home](Home)