Detect CPU throttling events

[saintaardvark]

This adds two checks that take different approaches to detecting CPU throttling:

• check_throttling_dmesg examines dmesg output for throttling events. The original request at Add detection of over-temperature events. #183 was compared with this page;

• check_throttling_vcgencmd examines the output of the Raspberry Pi utility vcgencmd to determine if throttling is currently happening, or has occurred in the past.

Testing, happy case:

# ./checks.sh | jq .
[snip]
    {
      "name": "check_throttling_dmesg",
      "success": true,
      "status": "No cpu throttling events detected"
    },
    {
      "name": "check_throttling_vcgencmd",
      "success": true,
      "status": "No Raspberry Pi throttling events detected"
    },

Sad case, dmesg:

# echo '<4>Temperature above threshold, cpu clock throttled' > /dev/kmsg 
# echo '<4>Temperature above threshold, cpu clock throttled' > /dev/kmsg 
# ./checks.sh | jq 
[snip]
    {
      "name": "check_throttling_dmesg",
      "success": false,
      "status": "2 cpu throttling events detected, check CPU temperature"
    },

Sad case, throttling:

# yes | /dev/null &
# ./checks.sh | jq 
[snip]
    {
      "name": "check_throttling_vcgencmd",
      "success": false,
      "status": "Raspberry Pi throttling events detected:  ARM freq capping has occurred"
    },

(Fun thread on how to create a CPU spike with shell commands alone.)

Connects-to: #183, #209
Change-type: minor
Signed-off-by: Hugh Brown [email protected]

[saintaardvark]

I'm not certain that the output of the Raspberry Pi throttling check is ideal; it might be better to simplify it into "Throttling is happening now / Throttling has happened in the past". @xginn8 , what do you think?

[xginn8]

A general comment: I've recently moved away from lots of checks doing specific things in favor of these "aggregated checks" like check_networking and check_localdisk that perform many tests and report a "federated" status. I think these make sense as tests integrated into the temperature check that already exists.

[saintaardvark]

Updated overview

This adds updates the temperature check with two new tests that take different approaches to detecting CPU throttling:

• test_throttling_dmesg examines dmesg output for throttling events. The original request at Add detection of over-temperature events. #183 was compared with this page;

• test_throttling_vcgencmd examines the output of the Raspberry Pi utility vcgencmd to determine if throttling is currently happening, or has occurred in the past.

Happy case:

{
  "diagnose_version": "4.17.15",
  "checks": [
    [snip]
    {
      "name": "check_temperature",
      "success": true,
      "status": "No temperature issues detected"
    },

Sad case:

{
  "diagnose_version": "4.17.15",
  "checks": [
    [snip]
     {
      "name": "check_temperature",
      "success": false,
      "status": "Some temperature issues detected: \ntest_temperature_now Temperature above 80C detected (/sys/class/thermal/thermal_zone0)\ntest_throttling_vcgencmd Raspberry Pi throttling events detected:  ARM freq capping has occurred"
    },

[xginn8]

Suggested change

	### check_throttling_vcgencmd
	### test_throttling_vcgencmd

[xginn8]

Suggested change

### check_throttling

this section can be deleted.

[saintaardvark]

Dang, thank you, good catch.