Can not get user-functions

[Suiiiii]

Hi, Extrae developers,

I finallly installed extrae on our cluster systems
however, here comes the problem the user-function can not be checked by paraver as my PC did.

And it is showing this:

WARNING: Negative value for MEMUSAGE_INUSE_EV detected (inuse=6569984+-1179709440-180640=-1173320096). Please submit a bug report.

Is that related to checking the user-functions?
How to fix it?

Welcome to Extrae 4.0.4
Extrae: Parsing the configuration file (extrae_v230705.xml) begins
Extrae: Tracing package is located on /.....
Extrae: Generating intermediate files for Paraver traces.
Extrae: MPI routines will collect HW counters information.
Extrae: All MPI_Comm_* calls will be traced.
Extrae: Warning! <openmp> tag will be ignored. This library does not support OpenMP.
Extrae: Number of user functions traced (XL runtime): 28
Extrae: Number of user functions traced (GCC runtime): 11
Extrae: User Function routines will collect HW counters information.
Extrae: Warning! change-at-time time units not specified. Using seconds
Extrae: PAPI domain set to ALL for HWC set 1
Extrae: HWC set 1 contains following counters < PAPI_TOT_INS (0x80000032) PAPI_TOT_CYC (0x8000003b) PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) > - never changes
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 2, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 3, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 4, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 5, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 1, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 0, thread 0)
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
Extrae: Resource usage is enabled at flush buffer.
Extrae: Memory usage is enabled at flush buffer.
Extrae: Tracing buffer can hold 50000000 events
Extrae: Circular buffer disabled.
Extrae: Dynamic memory instrumentation is disabled.
Extrae: Basic I/O memory instrumentation is disabled.
Extrae: System calls instrumentation is disabled.
Extrae: Parsing the configuration file (extrae_v230705.xml) has ended
Extrae: Intermediate traces will be stored in /.......
Extrae: Tracing mode is set to: Detail.
Extrae: Successfully initiated with 6 tasks and 1 threads

MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
Extrae: Temporal directory (/......) is shared among processes.
Extrae: Final directory (/......) is shared among processes.
Extrae: Successfully initiated with 6 tasks and 1 threads

..
.
.
.
.
.
.
.
.

Simulation is finished
.
.
.
.
.



WARNING: Negative value for MEMUSAGE_INUSE_EV detected (inuse=6569984+-1179709440-180640=-1173320096). Please submit a bug report.
Extrae: Intermediate raw trace file created :......mpit
Extrae: Intermediate raw trace file created : ......mpit
Extrae: Intermediate raw sym file created : ......sym
Extrae: Intermediate raw trace file created : .....mpit
Extrae: Intermediate raw sym file created : ......sym
Extrae: Intermediate raw sym file created : ......sym
Extrae: Deallocating memory.
Extrae: Intermediate raw trace file created :......mpit
Extrae: Application has ended. Tracing has been terminated.
Extrae: Intermediate raw sym file created : .......sym
Extrae: Intermediate raw trace file created : .....mpit
Extrae: Intermediate raw sym file created : .......sym
Extrae: Intermediate raw trace file created :......mpit
Extrae: Intermediate raw sym file created : ......sym

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 0 PID 2422615 RUNNING AT node0533
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 2 PID 2422617 RUNNING AT node0533
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 3 PID 2422618 RUNNING AT node0533
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 4 PID 2422619 RUNNING AT node0533
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 5 PID 2422620 RUNNING AT node0533
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

[gllort]

Hi, there seem to be two issues here, one related to capturing of user functions, and another related to instrumenting dynamic memory calls. To try to reduce the issues, I would suggest temporarily disabling support for memory calls, and focusing on the first problem with user functions. To do so, turn off the following option in the configuration file 'extrae_v230705.xml': . Does this change get rid of the message "WARNING: Negative value for MEMUSAGE_INUSE_EV..."? Are you still getting the "BAD TERMINATION" ones?

Turning into the user functions, a few things to check:

• Which compiler are you using to compile the application?
• In case of GNU / Intel compilers, are you using the flag -finstrument-functions? Are the files containing the functions that you want to instrument compiled with this flag?
• Did you use the flag -rdynamic (or -export-dynamic at linking) to make the functions visible?
• In the file containing the functions that you want to instrument, pointed by the option 'user-functions', attribute 'list' in the xml, did you specify the functions just by their name, or did you also set the address? If you set the address, can you try removing it and only specify the name? Please share this file if you're in doubt about it so we can have a look. Also, refer to this link for details: https://tools.bsc.es/doc/html/extrae/xml.html#xml-section-dynamic-memory
• In the resulting trace, are there no events whatsoever for the user functions, or are there some events, but they are all named something like "Unresolved"?

[Suiiiii]

I re-compiled extrae with latest version.
the user-function is back.
maybe there were some mistake with the environment.

But the warning is still there.
and the program is terminated badly.

Welcome to Extrae 4.0.6
Extrae: Parsing the configuration file (extrae_v230705.xml) begins
Extrae: Tracing package is located on <folder>/extrae_install2
Extrae: Generating intermediate files for Paraver traces.
Extrae: MPI routines will collect HW counters information.
Extrae: All MPI_Comm_* calls will be traced.
Extrae: Number of user functions traced (XL runtime): 28
Extrae: Number of user functions traced (GCC runtime): 14
Extrae: User Function routines will NOT collect HW counters information.
Extrae: Warning! change-at-time time units not specified. Using seconds
Extrae: PAPI domain set to ALL for HWC set 1
Extrae: HWC set 1 contains following counters < PAPI_TOT_INS (0x80000032) PAPI_TOT_CYC (0x8000003b) PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) > - never changes
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 9, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 15, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 2, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 3, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 1, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 14, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 7, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 0, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 4, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 8, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 5, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 10, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 11, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 6, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 13, thread 0)
Extrae: Error! Hardware counter PERF_COUNT_HW_STALLED_CYCLES_BACKEND (0x40000026) cannot be added in set 1 (task 12, thread 0)
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
Extrae: Resource usage is enabled at flush buffer.
Extrae: Memory usage is enabled at flush buffer.
Extrae: Tracing buffer can hold 50000000 events
Extrae: Circular buffer disabled.
Extrae: Dynamic memory instrumentation is disabled.
Extrae: Basic I/O memory instrumentation is disabled.
Extrae: System calls instrumentation is disabled.
Extrae: Parsing the configuration file (extrae_v230705.xml) has ended
Extrae: Intermediate traces will be stored in <folder>
Extrae: Tracing mode is set to: Detail.
Extrae: Successfully initiated with 16 tasks and 1 threads

MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
Extrae: Temporal directory (<folder>) is shared among processes.
Extrae: Final directory (<folder>) is shared among processes.
Extrae: Successfully initiated with 16 tasks and 1 threads


Box length Calculated:       19.8444462487654469
Box length Read from file:       19.8749259635630011
Box length is using the file value      19.8749259635630011      19.8749259635630011
 
This simulation is for the thermodynamic properties of [Ar]
molecule_M              =   0.039948 [kg/mole]
molecule_epsilon        = 143.120000 [K]
molecule_rnorm_an       =   3.357000 [A]
rstepinterpol           =   0.000420 [A]
 
With 2-Body setup : 
 
Number of atoms     =          108
box                 =  19.8749 [A]
rcut                =   9.9355 [A]
rcutmax_start       =  10.1068 [A]
rmin2b              =   1.8000 [A]
ru2b                =   1.6785 [A]
 
3-Body setup : 
 
rmin3b              =   2.2500 [A]
ru3b                =   2.2500 [A]
rstepmax3body       =      602
rstepinterpol3body  =   0.0132 [A]
  Simulation ist beendet
Extrae: Intermediate raw trace file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw sym file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw trace file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw trace file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw trace file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw trace file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw trace file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw trace file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw sym file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw trace file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw trace file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw sym file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw trace file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw sym file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw sym file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw trace file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw sym file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw trace file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw trace file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw trace file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw sym file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw sym file created : <folder>/run/set-0/[email protected]
WARNING: Negative value for MEMUSAGE_INUSE_EV detected (inuse=11567104+-1186443264-192880=-1175069040). Please submit a bug report.
Extrae: Intermediate raw sym file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw sym file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw sym file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw sym file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw trace file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw sym file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw sym file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw trace file created : <folder>/run/set-0/[email protected]
Extrae: Intermediate raw sym file created : <folder>/run/set-0/[email protected]
Extrae: Deallocating memory.
Extrae: Intermediate raw sym file created : <folder>/run/set-0/[email protected]
Extrae: Application has ended. Tracing has been terminated.

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 0 PID 2447366 RUNNING AT node0009
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 1 PID 2447367 RUNNING AT node0009
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 2 PID 2447368 RUNNING AT node0009
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 3 PID 2447369 RUNNING AT node0009
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 4 PID 2447370 RUNNING AT node0009
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 5 PID 2447371 RUNNING AT node0009
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 6 PID 2447372 RUNNING AT node0009
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 7 PID 2447373 RUNNING AT node0009
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 8 PID 2447374 RUNNING AT node0009
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 9 PID 2447375 RUNNING AT node0009
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 10 PID 2447376 RUNNING AT node0009
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 12 PID 2447378 RUNNING AT node0009
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 13 PID 2447379 RUNNING AT node0009
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 14 PID 2447380 RUNNING AT node0009
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 15 PID 2447381 RUNNING AT node0009
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================
(base) [suim@node0009 run]$ <folder>/bin/mpi2prv -f TRACE.mpits -e ./99_MCFortran.out -o "${parent_dir_name}.prv";
mpi2prv: Retrieving hardware counters definitions for ptask 1 from global SYM.
mpi2prv: A total of 3 symbols were imported from TRACE.sym file
mpi2prv: 0 function symbols imported
mpi2prv: 3 HWC counter descriptions imported
merger: Output trace format is: Paraver
merger: Extrae 4.0.6
mpi2prv: Assigned nodes < node0009 >
mpi2prv: Assigned size per processor < 1 Mbytes >
mpi2prv: File <folder>/run/set-0/[email protected] is object 1.1.1 on node node0009 assigned to processor 0
mpi2prv: File <folder>/run/set-0/[email protected] is object 1.2.1 on node node0009 assigned to processor 0
mpi2prv: File <folder>/run/set-0/[email protected] is object 1.3.1 on node node0009 assigned to processor 0
mpi2prv: File <folder>/run/set-0/[email protected] is object 1.4.1 on node node0009 assigned to processor 0
mpi2prv: File <folder>/run/set-0/[email protected] is object 1.5.1 on node node0009 assigned to processor 0
mpi2prv: File <folder>/run/set-0/[email protected] is object 1.6.1 on node node0009 assigned to processor 0
mpi2prv: File <folder>/run/set-0/[email protected] is object 1.7.1 on node node0009 assigned to processor 0
mpi2prv: File <folder>/run/set-0/[email protected] is object 1.8.1 on node node0009 assigned to processor 0
mpi2prv: File <folder>/run/set-0/[email protected] is object 1.9.1 on node node0009 assigned to processor 0
mpi2prv: File <folder>/run/set-0/[email protected] is object 1.10.1 on node node0009 assigned to processor 0
mpi2prv: File <folder>/run/set-0/[email protected] is object 1.11.1 on node node0009 assigned to processor 0
mpi2prv: File <folder>/run/set-0/[email protected] is object 1.12.1 on node node0009 assigned to processor 0
mpi2prv: File <folder>/run/set-0/[email protected] is object 1.13.1 on node node0009 assigned to processor 0
mpi2prv: File <folder>/run/set-0/[email protected] is object 1.14.1 on node node0009 assigned to processor 0
mpi2prv: File <folder>/run/set-0/[email protected] is object 1.15.1 on node node0009 assigned to processor 0
mpi2prv: File <folder>/run/set-0/[email protected] is object 1.16.1 on node node0009 assigned to processor 0
mpi2prv: Time synchronization has been turned off
mpi2prv: Checking for target directory existence... exists, ok!
mpi2prv: Selected output trace format is Paraver
mpi2prv: Stored trace format is Paraver
mpi2prv: Enabling Time Synchronization (Node).
mpi2prv: Circular buffer enabled at tracing time? NO
mpi2prv: Parsing intermediate files
mpi2prv: Progress 1 of 2 ... 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% done
mpi2prv: Processor 0 succeeded to translate its assigned files
mpi2prv: Elapsed time translating files: 0 hours 0 minutes 0 seconds
mpi2prv: Elapsed time sorting addresses: 0 hours 0 minutes 0 seconds
mpi2prv: Generating tracefile (intermediate buffers of 419424 events)
         This process can take a while. Please, be patient.
mpi2prv: Progress 2 of 2 ... 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% done
mpi2prv: Elapsed time merge step: 0 hours 0 minutes 0 seconds
mpi2prv: Resulting tracefile occupies 1035118 bytes
mpi2prv: Removing temporal files... done
mpi2prv: Elapsed time removing temporal files: 0 hours 0 minutes 0 seconds
mpi2prv: Congratulations! xxxx.prv has been generated.

[emercadal]

The warning for a negative value in the MEMUSAGE_INUSE_EV is not related to the user functions, as you may have already guessed. The warning can be removed by setting the `resource-usage` and `memory-usage` options under the "counters" section in the `.xml` file to "no". Regarding the other issue, to understand what is generating the BAD TERMINATION error in your execution we'll need to look at how you are running the application, commands and environment variables you are setting and any helper script you may be using.

[Suiiiii]

Hi,
The warning is canceled by your command.

the BAD TERMINATION is only happened at cluster with Extrae.

compiling flag:
-g -xCORE-AVX512 -qopenmp-simd -qopt-prefetch=2 -qopt-zmm-usage=high -fno-fnalias -finstrument-functions
execution:
mpirun -np 16 ./trace_nother.sh ./99_MCFortran.out;

the evironment variables I may send it by email to you.