pcm-sensor-server latest master does not respond on skylake when using -force-rtm-abort-mode option

[ballegre]

Not certain if I should make this an issue or discussion. In any case, when I use -force-rtm-abort-mode option with pcm-sensor-server on skylake the pcm-sensor-server does not respond to /metric scrapes (or curl get command for that matter). Without the option it does respond but with l2 cache hits zero (as expected). Last message displayed is Trying to use Linux perf events....

[root@n-0074-1-a ~]# ./pcm-sensor-server -force-rtm-abort-mode --reset -d

=====  Processor information  =====
Linux arch_perfmon flag  : yes
Hybrid processor         : no
IBRS and IBPB supported  : yes
STIBP supported          : yes
Spec arch caps supported : yes
Max CPUID level          : 22
CPU model number         : 85
Number of physical cores: 40
Number of logical cores: 80
Number of online logical cores: 80
Threads (logical cores) per physical core: 2
Num sockets: 2
Physical cores per socket: 20
Last level cache slices per socket: 20
Core PMU (perfmon) version: 4
Number of core PMU generic (programmable) counters: 3
Width of generic (programmable) counters: 48 bits
Number of core PMU fixed counters: 3
Width of fixed counters: 48 bits
Nominal core frequency: 2000000000 Hz
IBRS enabled in the kernel   : no
STIBP enabled in the kernel  : no
The processor is not susceptible to Rogue Data Cache Load: no
The processor supports enhanced IBRS                     : no
Package thermal spec power: 125 Watt; Package minimum power: 65 Watt; Package maximum power: 264 Watt;

INFO: Linux perf interface to program uncore PMUs is present
Socket 0: 2 memory controllers detected with total number of 6 channels. 3 UPI ports detected. 2 M2M (mesh to memory) blocks detected. 0 HBM M2M blocks detected. 0 EDC/HBM channels detected. 0 Home Agents detected. 3 M3UPI blocks detected.
Socket 1: 2 memory controllers detected with total number of 6 channels. 3 UPI ports detected. 2 M2M (mesh to memory) blocks detected. 0 HBM M2M blocks detected. 0 EDC/HBM channels detected. 0 Home Agents detected. 3 M3UPI blocks detected.
Initializing RMIDs

The number of custom counters is now 4
 Closed perf event handles
Trying to use Linux perf events...
[root@n-0074-1-a ~]# curl http://localhost:9738/
curl: (7) Failed to connect to localhost port 9738: Connection refused
[root@n-0074-1-a ~]# curl http://localhost:9738/metrics
curl: (7) Failed to connect to localhost port 9738: Connection refused

EDIT: Adding version

[root@n-0074-1-a ~]# ./pcm-sensor-server --version
version: ($Format:%ci ID=%h$)

[ballegre]

The pcm command seems to work.

[root@n-0074-1-a ~]# ./pcm 3 -force-rtm-abort-mode -r 

 Intel(r) Performance Counter Monitor ($Format:%ci ID=%h$)



=====  Processor information  =====
Linux arch_perfmon flag  : yes
Hybrid processor         : no
IBRS and IBPB supported  : yes
STIBP supported          : yes
Spec arch caps supported : yes
Max CPUID level          : 22
CPU model number         : 85
Number of physical cores: 40
Number of logical cores: 80
Number of online logical cores: 80
Threads (logical cores) per physical core: 2
Num sockets: 2
Physical cores per socket: 20
Last level cache slices per socket: 20
Core PMU (perfmon) version: 4
Number of core PMU generic (programmable) counters: 3
Width of generic (programmable) counters: 48 bits
Number of core PMU fixed counters: 3
Width of fixed counters: 48 bits
Nominal core frequency: 2000000000 Hz
IBRS enabled in the kernel   : no
STIBP enabled in the kernel  : no
The processor is not susceptible to Rogue Data Cache Load: no
The processor supports enhanced IBRS                     : no
Package thermal spec power: 125 Watt; Package minimum power: 65 Watt; Package maximum power: 264 Watt;

INFO: Linux perf interface to program uncore PMUs is present
Socket 0: 2 memory controllers detected with total number of 6 channels. 3 UPI ports detected. 2 M2M (mesh to memory) blocks detected. 0 HBM M2M blocks detected. 0 EDC/HBM channels detected. 0 Home Agents detected. 3 M3UPI blocks detected.
Socket 1: 2 memory controllers detected with total number of 6 channels. 3 UPI ports detected. 2 M2M (mesh to memory) blocks detected. 0 HBM M2M blocks detected. 0 EDC/HBM channels detected. 0 Home Agents detected. 3 M3UPI blocks detected.
Initializing RMIDs

The number of custom counters is now 4

 Resetting PMU configuration
 Zeroed PMU registers
 Closed perf event handles
Trying to use Linux perf events...
Successfully programmed on-core PMU using Linux perf
Socket 0
Max UPI link 0 speed: 23.3 GBytes/second (10.4 GT/second)
Max UPI link 1 speed: 23.3 GBytes/second (10.4 GT/second)
Max UPI link 2 speed: 21.5 GBytes/second (9.6 GT/second)
Socket 1
Max UPI link 0 speed: 23.3 GBytes/second (10.4 GT/second)
Max UPI link 1 speed: 23.3 GBytes/second (10.4 GT/second)
Max UPI link 2 speed: 21.5 GBytes/second (9.6 GT/second)

Detected Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz "Intel(r) microarchitecture codename Skylake-SP" stepping 4 microcode level 0x2006c0a

 EXEC  : instructions per nominal CPU cycle
 IPC   : instructions per CPU cycle
 FREQ  : relation to nominal CPU frequency='unhalted clock ticks'/'invariant timer ticks' (includes Intel Turbo Boost)
 AFREQ : relation to nominal CPU frequency while in active state (not in power-saving C state)='unhalted clock ticks'/'invariant timer ticks while in C0-state'  (includes Intel Turbo Boost)
 L3MISS: L3 (read) cache misses 
 L2MISS: L2 (read) cache misses (including other core's L2 cache *hits*) 
 L3HIT : L3 (read) cache hit ratio (0.00-1.00)
 L2HIT : L2 cache hit ratio (0.00-1.00)
 L3MPI : number of L3 (read) cache misses per instruction
 L2MPI : number of L2 (read) cache misses per instruction
 READ  : bytes read from main memory controller (in GBytes)
 WRITE : bytes written to main memory controller (in GBytes)
 LOCAL : ratio of local memory requests to memory controller in %
 L3OCC : L3 occupancy (in KBytes)
 TEMP  : Temperature reading in 1 degree Celsius relative to the TjMax temperature (thermal headroom): 0 corresponds to the max temperature
 energy: Energy in Joules


 Core (SKT) | EXEC | IPC  | FREQ  | AFREQ | L3MISS | L2MISS | L3HIT | L2HIT | L3MPI | L2MPI |   L3OCC | TEMP

   0    0     0.02   0.74   0.03    0.76     152 K   1233 K    0.87    0.47  0.0014  0.0112      480     52
   1    0     0.00   0.47   0.01    0.99      39 K     93 K    0.51    0.71  0.0019  0.0046     1360     52
   2    0     0.00   0.32   0.01    1.07    9598       46 K    0.61    0.62  0.0010  0.0048      160     51
   3    0     0.00   0.46   0.01    1.15    6779       49 K    0.71    0.79  0.0005  0.0033      480     52
   4    0     0.00   0.42   0.01    0.99      11 K     65 K    0.73    0.73  0.0008  0.0046        0     52
   5    0     0.16   2.13   0.08    1.33      22 K     80 K    0.70    0.84  0.0000  0.0001     3280     51
   6    0     0.17   2.08   0.08    1.33      18 K     97 K    0.74    0.83  0.0000  0.0001     2800     51
   7    0     0.00   0.33   0.00    1.10    9927       47 K    0.61    0.58  0.0011  0.0052      320     51
   8    0     0.00   0.65   0.01    1.20    5504       45 K    0.75    0.85  0.0003  0.0022      560     52
   9    0     0.00   0.06   0.00    0.86     772     2475      0.67    0.40  0.0026  0.0084        0     52
  10    0     0.00   0.42   0.01    1.21      10 K     64 K    0.69    0.67  0.0008  0.0046      560     52
  11    0     0.00   0.42   0.01    1.22    6506       58 K    0.76    0.67  0.0005  0.0041      480     52
  12    0     0.01   0.76   0.01    1.24      11 K     60 K    0.70    0.79  0.0004  0.0019      720     52
  13    0     0.05   1.62   0.03    1.32      24 K    290 K    0.91    0.80  0.0001  0.0010      880     52
  14    0     0.06   1.62   0.03    1.32      35 K    137 K    0.67    0.60  0.0001  0.0004      560     51
  15    0     0.00   0.44   0.01    1.24    7569       59 K    0.78    0.64  0.0005  0.0039      560     51
  16    0     0.09   1.90   0.05    1.30      47 K    124 K    0.58    0.83  0.0001  0.0002     1840     51
  17    0     0.07   1.53   0.05    1.32      50 K    233 K    0.71    0.86  0.0001  0.0005     1920     52
  18    0     0.00   0.46   0.00    1.19    7623       58 K    0.80    0.63  0.0006  0.0044      480     51
  19    0     0.00   0.26   0.00    1.15    6862       45 K    0.71    0.54  0.0011  0.0071       80     52
  20    1     0.16   1.58   0.10    1.33     183 K    660 K    0.64    0.81  0.0002  0.0007      240     46
  21    1     0.22   1.87   0.12    1.34     191 K    681 K    0.65    0.83  0.0001  0.0005     3440     45
  22    1     0.00   0.42   0.01    1.28      24 K     75 K    0.47    0.70  0.0015  0.0047        0     44
  23    1     0.18   1.99   0.09    1.34     210 K    570 K    0.57    0.77  0.0002  0.0005       80     44
  24    1     0.08   0.97   0.08    1.32     240 K    858 K    0.67    0.78  0.0005  0.0017      240     45
  25    1     0.20   2.16   0.09    1.35      68 K    352 K    0.78    0.83  0.0001  0.0003      240     43
  26    1     0.21   2.05   0.10    1.35     113 K    460 K    0.72    0.81  0.0001  0.0004      400     43
  27    1     0.32   2.15   0.15    1.34     249 K    680 K    0.60    0.83  0.0001  0.0003        0     44
  28    1     0.18   1.99   0.09    1.35     141 K    445 K    0.63    0.79  0.0001  0.0004       80     44
  29    1     0.35   1.73   0.20    1.34    1354 K   2418 K    0.42    0.75  0.0007  0.0012     3200     45
  30    1     0.18   1.96   0.09    1.35     186 K    576 K    0.64    0.90  0.0002  0.0005        0     47
  31    1     0.20   2.12   0.09    1.34     135 K    439 K    0.64    0.79  0.0001  0.0004      400     42
  32    1     0.14   1.99   0.07    1.34     219 K    467 K    0.48    0.88  0.0003  0.0005      400     46
  33    1     0.00   0.75   0.01    1.29    8179       47 K    0.77    0.77  0.0004  0.0021        0     47
  34    1     0.00   0.22   0.00    1.04    2066     7452      0.69    0.46  0.0021  0.0076        0     44
  35    1     0.08   1.74   0.04    1.34      64 K    249 K    0.68    0.78  0.0001  0.0005      400     44
  36    1     0.24   2.64   0.09    1.34      19 K    141 K    0.81    0.80  0.0000  0.0001      320     43
  37    1     0.14   1.76   0.08    1.34     239 K    668 K    0.58    0.75  0.0003  0.0008     2480     44
  38    1     0.24   2.56   0.10    1.35      21 K    169 K    0.83    0.82  0.0000  0.0001        0     45
  39    1     0.00   0.05   0.00    0.93     478      732      0.32    0.48  0.0044  0.0067        0     46
  40    0     0.00   0.44   0.00    0.88      22 K     68 K    0.64    0.56  0.0026  0.0077      400     52
  41    0     0.00   0.22   0.00    1.17    2005       28 K    0.82    0.57  0.0005  0.0067       80     51
  42    0     0.09   1.85   0.05    1.33      45 K    147 K    0.63    0.77  0.0001  0.0003      880     51
  43    0     0.00   0.24   0.00    1.14    3378       29 K    0.74    0.67  0.0006  0.0054        0     52
  44    0     0.00   0.67   0.01    1.20    9234       60 K    0.73    0.83  0.0004  0.0025      240     52
  45    0     0.09   2.01   0.04    1.33      28 K     71 K    0.53    0.85  0.0001  0.0001     1520     51
  46    0     0.00   0.12   0.00    0.80    2077     4711      0.46    0.48  0.0031  0.0071        0     51
  47    0     0.00   0.61   0.01    1.11      16 K    102 K    0.82    0.68  0.0008  0.0051     1440     50
  48    0     0.01   1.09   0.01    1.19    9762       64 K    0.73    0.89  0.0001  0.0010      480     52
  49    0     0.01   0.66   0.02    1.25      77 K    212 K    0.55    0.79  0.0009  0.0025     1840     52
  50    0     0.00   0.46   0.00    1.16    5481       37 K    0.75    0.74  0.0005  0.0034       80     52
  51    0     0.00   0.53   0.00    1.05      16 K     49 K    0.57    0.63  0.0012  0.0035      400     52
  52    0     0.05   1.66   0.03    1.30      28 K    343 K    0.91    0.79  0.0001  0.0011      800     52
  53    0     0.00   0.64   0.00    1.10    8104       34 K    0.71    0.74  0.0006  0.0024      320     52
  54    0     0.01   1.05   0.01    1.20    7916       76 K    0.89    0.60  0.0002  0.0023      240     51
  55    0     0.00   0.56   0.00    1.11    5448       24 K    0.72    0.74  0.0005  0.0024      240     51
  56    0     0.00   0.27   0.00    0.94    5798       12 K    0.48    0.58  0.0020  0.0043      240     50
  57    0     0.26   3.15   0.08    1.34      26 K     66 K    0.56    0.91  0.0000  0.0000      640     52
  58    0     0.00   0.60   0.00    1.11    3312       18 K    0.76    0.76  0.0004  0.0020      480     51
  59    0     0.00   0.65   0.00    0.94      11 K     46 K    0.66    0.74  0.0007  0.0027       80     52
  60    1     0.00   0.37   0.00    1.26      11 K     54 K    0.67    0.58  0.0014  0.0069       80     46
  61    1     0.00   0.07   0.00    0.93     643     1453      0.49    0.40  0.0044  0.0100        0     45
  62    1     0.07   1.28   0.05    1.33     153 K    511 K    0.61    0.74  0.0004  0.0012      320     44
  63    1     0.08   1.33   0.06    1.34      87 K    511 K    0.81    0.82  0.0002  0.0011      400     44
  64    1     0.01   0.79   0.01    1.33      12 K    124 K    0.86    0.79  0.0002  0.0020      320     45
  65    1     0.02   0.87   0.02    1.34      18 K    168 K    0.86    0.81  0.0002  0.0017      720     43
  66    1     0.00   0.57   0.00    1.17    4492       10 K    0.44    0.67  0.0011  0.0024        0     43
  67    1     0.58   1.93   0.30    1.35     609 K   1660 K    0.60    0.90  0.0002  0.0005     3840     44
  68    1     0.00   0.88   0.00    1.30    8592       36 K    0.72    0.75  0.0004  0.0017        0     44
  69    1     0.00   0.24   0.00    1.16    3699       14 K    0.62    0.52  0.0024  0.0092        0     45
  70    1     0.04   1.53   0.02    1.32      31 K    147 K    0.73    0.76  0.0001  0.0007      320     47
  71    1     0.25   1.77   0.14    1.34     342 K   1034 K    0.63    0.88  0.0002  0.0007     5360     42
  72    1     0.18   1.99   0.09    1.35      54 K    158 K    0.60    0.83  0.0001  0.0001        0     46
  73    1     0.11   1.62   0.07    1.34      49 K    308 K    0.80    0.80  0.0001  0.0005      320     47
  74    1     0.20   2.08   0.10    1.35     176 K    559 K    0.65    0.90  0.0001  0.0005      240     44
  75    1     0.00   0.72   0.00    1.30      12 K     35 K    0.56    0.78  0.0008  0.0022      160     44
  76    1     0.23   2.76   0.09    1.35      15 K     86 K    0.77    0.82  0.0000  0.0001        0     43
  77    1     0.00   0.37   0.00    1.22      15 K     30 K    0.39    0.50  0.0032  0.0062        0     44
  78    1     0.02   0.83   0.02    1.34      27 K    169 K    0.79    0.79  0.0003  0.0019      560     45
  79    1     0.07   1.26   0.05    1.35      62 K    412 K    0.81    0.79  0.0002  0.0010     2160     46
---------------------------------------------------------------------------------------------------------------
 SKT    0     0.03   1.71   0.02    1.25     823 K   4395 K    0.77    0.74  0.0001  0.0006    27920     50
 SKT    1     0.12   1.88   0.07    1.34    5370 K     16 M    0.61    0.83  0.0002  0.0005    26720     42
---------------------------------------------------------------------------------------------------------------
 TOTAL  *     0.08   1.85   0.04    1.32    6194 K     20 M    0.65    0.82  0.0002  0.0005     N/A      N/A

 Instructions retired:   37 G ; Active cycles:   20 G ; Time (TSC): 6003 Mticks ; C0 (active,non-halted) core residency: 3.17 %

 C1 core residency: 96.83 %; C6 core residency: 0.00 %;
 C0 package residency: 100.00 %; C2 package residency: 0.00 %; C6 package residency: 0.00 %;
                             \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
 Core    C-state distribution\u250200011111111111111111111111111111111111111111111111111111111111111111111111111111\u2502
                             \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518
                             \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510
 Package C-state distribution\u250200000000000000000000000000000000000000000000000000000000000000000000000000000000\u2502
                             \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518

 PHYSICAL CORE IPC                 : 3.69 => corresponds to 92.36 % utilization for cores in active state
 Instructions per nominal CPU cycle: 0.15 => corresponds to 3.86 % core utilization over time interval
 SMI count: 0

Intel(r) UPI data traffic estimation in bytes (data traffic coming to CPU/socket through UPI links):

               UPI0     UPI1     UPI2    |  UPI0   UPI1   UPI2  
---------------------------------------------------------------------------------------------------------------
 SKT    0       81 M     86 M      0     |    0%     0%     0%   
 SKT    1       69 M     62 M      0     |    0%     0%     0%   
---------------------------------------------------------------------------------------------------------------
Total UPI incoming data traffic:  300 M     UPI data traffic/Memory controller traffic: 0.10

Intel(r) UPI traffic estimation in bytes (data and non-data traffic outgoing from CPU/socket through UPI links):

               UPI0     UPI1     UPI2    |  UPI0   UPI1   UPI2  
---------------------------------------------------------------------------------------------------------------
 SKT    0      276 M    292 M      0     |    0%     0%     0%   
 SKT    1      307 M    292 M      0     |    0%     0%     0%   
---------------------------------------------------------------------------------------------------------------
Total UPI outgoing data and non-data traffic: 1169 M
MEM (GB)->|  READ |  WRITE | LOCAL | CPU energy | DIMM energy | UncFREQ (Ghz)|
---------------------------------------------------------------------------------------------------------------
 SKT   0     0.20     0.18   60 %     179.56      31.71             2.40
 SKT   1     1.71     0.89   94 %     228.26      26.87             2.40
---------------------------------------------------------------------------------------------------------------
       *     1.90     1.07   91 %     407.82      58.58             2.40
^CDEBUG: caught signal to interrupt (Interrupt).
Cleaning up
 Closed perf event handles
The number of custom counters is now 3
 Zeroed uncore PMU registers
 Freeing up all RMIDs

[rdementi]

that is a bug. The option is not compatible with the -d option. Fixing it now...

[rdementi]

the fix is in the master branch

[rdementi]

this one? 32cee3b
this change has been merged.

[ogbrugge]

Yes that one, but perhaps i am mistaken and the line at the bottom "setsockopt failed while setting timeout value, Socket operation on non-socket" is from curl and not an error message from pcm-sensor-daemon

[rdementi]

we have exactly this string in pcm:

$ git grep 'setsockopt failed while setting timeout value'
src/pcm-sensor-server.cpp:            std::cerr << "setsockopt failed while setting timeout value, " << strerror( errno ) << "\n";
src/pcm-sensor-server.cpp:            std::cerr << "setsockopt failed while setting timeout value, " << strerror( errno ) << "\n";

[ballegre]

Perhaps that error is specific to our platform (Nutanix AHV). I don't see the error message when I run the same curl command remotely (from linux machine). For now, doesn't seem to affect how it works.

[ogbrugge]

No that bug does not affect you. Just wondering what code path is triggering this. I guess I need to double check all socket initialization and closing paths again.