[crmsh-4.6] Dev: pre-migration: implement pre-migration checks for SLES 16 (jsc#PED-11808) #1629

nicholasyang2022 · 2024-12-13T07:35:41Z

This pull request implements pre-migration checks for SLES 16. These checks are expected to run before migrating to SLES 16.

These checks ensure all the cluster nodes runs the latest version of corosync 2.x and pacemaker 2.1.x, and report used resource agents and fence agents that will be removed in SLES 16. It also provides advice for further actions need to take after migration.

Use Cases

`crm cluster health sles16`

suse@ha-1-1:~> sudo crm cluster health sles16
------ ha-1-2 ------
[WARN] Cluster services are not running. Check results may be outdated or inaccurate.
       * corosync
       * pacemaker
[WARN] Corosync transport "udpu" will be deprecated in corosync 3.
       After migrating to SLES 16, run "crm cluster health sles16 --fix" to migrate it to transport "knet".
[PASS] Good to migrate to SLES 16.

------ localhost ------
[INFO] Checking dependency version...
[INFO] Checking service status...
[WARN] Cluster services are not running. Check results may be outdated or inaccurate.
       * corosync
       * pacemaker
[INFO] Checking used corosync features...
[WARN] Corosync transport "udpu" will be deprecated in corosync 3.
       After migrating to SLES 16, run "crm cluster health sles16 --fix" to migrate it to transport "knet".
[INFO] Checking used resource agents...
[FAIL] SAPHanaSR Classic will be removed in SLES 16.
       Before migrating to SLES 16, replace it with SAPHanaSR-angi.
[WARN] stonith:external/sbd will be deprecated.
       * After migrating to SLES 16, please replace it with stonith:fence_sbd.
[FAIL] Please fix all the "FAIL" problems above before migrating to SLES 16.

`crm cluster health sles16 --fix`

> sudo crm cluster health sles16 --fix
ERROR: "--fix" is only available in SLES 16.

When some of the nodes are offline

------ ha-1-2 ------
Cannot create SSH connection to root@ha-1-2: ssh: connect to host ha-1-2 port 22: No route to host

------ localhost ------
[INFO] Checking dependency version...
[INFO] Checking service status...
[WARN] Cluster services are not running. Check results may be outdated or inaccurate.
       * corosync
       * pacemaker
[INFO] Checking used corosync features...
[WARN] Corosync transport "udpu" will be deprecated in corosync 3.
       After migrating to SLES 16, run "crm cluster health sles16 --fix" to migrate it to transport "knet".
[INFO] Checking used resource agents...
[FAIL] Please fix all the "FAIL" problems above before migrating to SLES 16.

codecov · 2024-12-16T06:54:24Z

Codecov Report

Attention: Patch coverage is 82.18182% with 49 lines in your changes missing coverage. Please review.

Project coverage is 67.00%. Comparing base (f5b1328) to head (252befd).

Files with missing lines	Patch %	Lines
crmsh/migration.py	82.37%	40 Missing ⚠️
crmsh/ui_cluster.py	75.67%	9 Missing ⚠️

Additional details and impacted files

Flag	Coverage Δ
integration	`52.97% <82.18%> (+0.31%)`	⬆️
unit	`49.21% <28.72%> (-0.20%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
crmsh/cibquery.py	`100.00% <100.00%> (ø)`
crmsh/sh.py	`93.22% <ø> (ø)`
crmsh/xmlutil.py	`69.23% <100.00%> (+0.03%)`	⬆️
crmsh/ui_cluster.py	`74.21% <75.67%> (-0.65%)`	⬇️
crmsh/migration.py	`82.37% <82.37%> (ø)`

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

liangxin1300 · 2025-01-01T14:01:02Z

crmsh/migration-supported-resource-agents.txt

+stonith:fence_vmware-rest
+stonith:fence_wti
+stonith:fence_xenapi
+stonith:fence_zvm


How has the above data been generated?
Can we automatically do that based on the locally installed resource-agents and fence-agents-* rpm?
Or, can we generate the data at the time crmsh using it (_load_supported_resource_agents)?

How has the above data been generated?

Extracted the file list from resource-agents/fence-agents.

Can we automatically do that based on the locally installed resource-agents and fence-agents-* rpm?

No, because we need to know what is supported by SLES 16 instead of SLES 15 SP6.

liangxin1300 · 2025-01-03T05:41:14Z

Suggestion:

1. Add completer for `crm cluster health`, now `hawk2` can be completed, so do `sles16`.

The same point for #1422

2. Add success info

# crm cluster health sles16
------ localhost ------
[INFO] Checking dependency version...
[INFO] Checking service status...
[INFO] Checking used corosync features...
[WARN] Corosync transport "udpu" will be deprecated in corosync 3.
       After migrating to SLES 16, run "crm health sles16 --fix" to migrate it to transport "knet".
[INFO] Checking used resource agents...
[PASS]

How about change the last line as

[PASS] All checks completed successfully.

Or

[INFO] All checks completed successfully.

I prefer this [INFO] one, since [PASS] is not the log level
The same for #1422

3. Add failure error

# crm cluster health sles16
------ localhost ------
[INFO] Checking dependency version...
[INFO] Checking service status...
[FAIL] Cluster services are not running
       * corosync
       * pacemaker
[INFO] Checking used corosync features...
[WARN] Corosync transport "udpu" will be deprecated in corosync 3.
       After migrating to SLES 16, run "crm health sles16 --fix" to migrate it to transport "knet".
[INFO] Checking used resource agents...
[FAIL]

How about change the last line similar:

[ERROR] Some checks failed, please check

And, change [FAIL] Cluster services are not running
as

[ERROR] Cluster services are not running

The same for #1422

4. Add above output into `/var/log/crmsh/crmsh.log` log file, too

The same for #1422

5. Keep the color of node line consistency

Better keep all of node lines white (no color)?

The same for #1422

liangxin1300 · 2025-01-03T06:27:45Z

Suggestion:

6. How about there are Failed Resource Actions?

For some reason, some RA failed. For now, crm cluster health sles16 seems quiet for such an error.
I think we should hint the user to address that failed RA, then do the migration job
The same for #1422

7. crm report collect pre-migration info

I think we also need to collect the output of crm cluster health sles16 in crm report.
How about implementing a collect_pre_migraion function in report.collect.py, and dump the output into pre-migration.txt?

8. Inaccurate usage?

# crm cluster health sles15
usage: crm [-h] [-f] {hawk2,sles16}
crm: error: argument component: invalid choice: 'sles15' (choose from 'hawk2', 'sles16')

Should be

usage: crm cluster health {hawk2,sles16}

The same for #1422

9. Fence agents checking

[FAIL] The following fence agents will be removed in SLES 16.
       * stonith:external/sbd

Better to add what's the alternative?

[ERROR] The following fence agents will be replaced
      - stonith:external/sbd
      + stonith:fence_sbd

liangxin1300 · 2025-01-03T06:43:35Z

crmsh/migration.py

+    handler.log_info("Checking used corosync features...")
+    transport = 'udpu' if corosync.is_unicast() else 'udp'
+    handler.handle_tip(f'Corosync transport "{transport}" will be deprecated in corosync 3.', [
+        'After migrating to SLES 16, run "crm health sles16 --fix" to migrate it to transport "knet".',


Should be crm cluster health sles16 --fix

liangxin1300 · 2025-01-03T06:43:57Z

crmsh/migration.py

+    ])
+    if corosync.get_value("totem.rrp_mode") in {'active', 'passive'}:
+        handler.handle_tip(f'Corosync RRP will be deprecated in corosync 3.', [
+            'After migrating to SLES 16, run "crm health sles16 --fix" to migrate it to knet multilink.',


Should be crm cluster health sles16 --fix

liangxin1300 · 2025-01-13T03:06:10Z

crmsh/cibquery.py

@@ -0,0 +1,25 @@
+"""utilities for parsing CIB xml"""


Maybe it's better to move these codes into xmlutil.py instead of creating a new py file?

No. xmlutil.py is too long, and its name does not reflect its content. Instead of general xml processing utilities, it is a mixed mess with shell utils, CIB specific routines, and even CLI handlers.

liangxin1300 · 2025-01-13T03:08:14Z

crmsh/cibquery.py

+    )
+
+
+def has_primitive_filesystem_ocfs2(cib: lxml.etree.Element) -> bool:


Considering we might use this to detect gfs2, could you please change 'ocfs2' as a parameter?

liangxin1300 · 2025-01-13T03:11:37Z

crmsh/migration.py

+
+def check_unsupported_resource_agents(handler: CheckResultHandler):
+    handler.log_info("Checking used resource agents...")
+    crm_mon = xmlutil.CrmMonXmlParser()


This is not used

liangxin1300 · 2025-01-13T03:21:58Z

crmsh/migration-supported-resource-agents.txt

How about moving this file under crmsh/crmsh/data, in case there are other txt files in the future?

I prefer not to do that. I think this will need to add a python module crmsh.data, making our build scripts more complex.

liangxin1300 · 2025-01-13T03:35:28Z

crmsh/migration.py

+            sys.stdout.write(' Good to migrate to SLES 16.\n\n')
+
+
+def check(args: typing.Sequence[str]) -> int:


It is better to add a docstring to:

explain what the --json and --local option are used for

introduce this function and return code

This is quite self-explained. Adding docstrings like "migration.check: run migration checks", or "--json output the result in json format" does not make sense. And the return code is also the common main() style return code.

liangxin1300 · 2025-01-13T06:27:39Z

This usage is correct

# crm cluster health xxx
usage: crm cluster health [-h] [-f] {hawk2,sles16}
crm cluster health: error: argument component: invalid choice: 'xxx' (choose from 'hawk2', 'sles16')

But not this

# crm cluster health sles16 fs
usage: crm [-h] [--json [{oneline,pretty}]] [--local]
crm: error: unrecognized arguments: fs

…11808)

…c#PED-11808)

…jsc#PED-11808)

nicholasyang2022 · 2025-01-13T07:06:30Z

Usages changed to

suse@ha-1-1:~> sudo crm cluster health foo
usage: health [-h] [-f] {hawk2,sles16}
health: error: argument component: invalid choice: 'foo' (choose from 'hawk2', 'sles16')

suse@ha-1-1:~> sudo crm cluster health sles16 foo
usage: sles16 [-h] [--json [{oneline,pretty}]] [--local]
sles16: error: unrecognized arguments: foo

liangxin1300 · 2025-01-13T06:45:23Z

crmsh/migration.py

+    handler.handle_tip(f'Corosync transport "{transport}" will be deprecated in corosync 3.', [
+        'After migrating to SLES 16, run "crm cluster health sles16 --fix" to migrate it to transport "knet".',
+    ])
+    if corosync.get_value("totem.rrp_mode") in {'active', 'passive'}:


Although with low possibility, rrp_mode can be set to none for the one-ring
should include none value here

none means SRP, which is supported and does not needs any actions.

liangxin1300 · 2025-01-13T08:02:32Z

doc/crm.8.adoc

+* `sles16`: check whether the cluster is good to migrate to SLES 16.
+
+The optional `--fix` argument attempts to automatically resolve any detected
+issues.


Please add note that for sles16, --fix only available at SLES 16

…PED-11808) should use recursive queries

…1808)

…ilesystem_with_fstype

liangxin1300 · 2025-01-13T12:29:04Z

crmsh/migration.py

+    if remote_ret > ret:
+            ret = remote_ret
+    if not parsed_args.json:
+        print('------ summary ------')


Better change this line as

****** Summary ******

To mark this is not the node name

liangxin1300

Looks good to me, great work!

Thanks!

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch from 70f74a6 to 1a80eee Compare December 16, 2024 06:23

nicholasyang2022 changed the title ~~[crmsh] Dev: pre-migration: implement pre-migration checks for corosync 3 (jsc#PED-8252)~~ [crmsh-4.6] Dev: pre-migration: implement pre-migration checks for corosync 3 (jsc#PED-8252) Dec 16, 2024

zzhou1 changed the title ~~[crmsh-4.6] Dev: pre-migration: implement pre-migration checks for corosync 3 (jsc#PED-8252)~~ [crmsh-4.6] Dev: pre-migration: implement pre-migration checks for corosync 3 (jsc#PED-11808) Dec 17, 2024

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch 2 times, most recently from d9038c6 to a2518dd Compare December 24, 2024 05:34

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch 7 times, most recently from a740b3f to 2bcad35 Compare December 31, 2024 09:28

liangxin1300 self-requested a review January 1, 2025 13:54

liangxin1300 reviewed Jan 1, 2025

View reviewed changes

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch 3 times, most recently from 2932b25 to 8cf8ef3 Compare January 2, 2025 00:15

nicholasyang2022 changed the title ~~[crmsh-4.6] Dev: pre-migration: implement pre-migration checks for corosync 3 (jsc#PED-11808)~~ [crmsh-4.6] Dev: pre-migration: implement pre-migration checks for corosync 3 and pacemaker 3 (jsc#PED-11808) Jan 2, 2025

nicholasyang2022 marked this pull request as ready for review January 2, 2025 00:57

nicholasyang2022 changed the title ~~[crmsh-4.6] Dev: pre-migration: implement pre-migration checks for corosync 3 and pacemaker 3 (jsc#PED-11808)~~ [crmsh-4.6] Dev: pre-migration: implement pre-migration checks for SLES 16 (jsc#PED-11808) Jan 2, 2025

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch 4 times, most recently from 22ea8a0 to a338bc6 Compare January 2, 2025 08:57

liangxin1300 reviewed Jan 3, 2025

View reviewed changes

Dev: ui_cluster: add 'crm cluster health sles [--fix]' (jsc#PED-8252)

4488385

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch from 81e76eb to 2dff378 Compare January 13, 2025 02:34

liangxin1300 reviewed Jan 13, 2025

View reviewed changes

nicholasyang2022 added 8 commits January 13, 2025 14:37

Dev: pre-migration: check SAPHanaSR Classic resource agents (jsc#PED-…

f061b4f

…11808)

Dev: pre-migration: check removed resource agents (jsc#PED-11808)

a38ec9a

Dev: pre-migration: check removed fence agents (jsc#PED-11808)

440ea0e

Dev: pre-migration: check if ocfs2 is used (jsc#PED-11808)

bbfcfa4

Dev: add pre-migration checks for pacemaker version (jsc#PED-11808)

36f7d23

Dev: pre-migration: better PASS/FAIL messages (jsc#PED-11808)

52dac4d

Dev: pre-migration: do not colorize hostname header (jsc#PED-11808)

14f5b2e

Dev: pre-migration: add completer for 'crm cluster health sles16' (js…

4dae2e3

…c#PED-11808)

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch from 2dff378 to d1563aa Compare January 13, 2025 06:41

nicholasyang2022 added 3 commits January 13, 2025 15:05

Fix: ui_cluster: fix do_health usage output

22fd915

Dev: pre-migration: add message about removing stonith:external/sbd (…

abbbc3e

…jsc#PED-11808)

Dev: pre-migration: refine the message for stopped cluster services (…

843d637

…jsc#PED-11808)

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch from d1563aa to d9c0449 Compare January 13, 2025 07:05

liangxin1300 reviewed Jan 13, 2025

View reviewed changes

nicholasyang2022 added 4 commits January 13, 2025 16:54

Doc: ui_cluster: add document for crm cluster health hawk2|sles16

d8b62b1

Fix: cibquery: grouped primitives is missing from query results (jsc#…

2368846

…PED-11808) should use recursive queries

Dev: behave: add functional tests for pre-migration checks (jsc#PED-1…

b6b8987

…1808)

Refactor: cibquery: has_primitive_filesystem_ocfs2 to has_primitive_f…

26ccee4

…ilesystem_with_fstype

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch from d9c0449 to 94926c5 Compare January 13, 2025 08:54

liangxin1300 reviewed Jan 13, 2025

View reviewed changes

liangxin1300 approved these changes Jan 13, 2025

View reviewed changes

liangxin1300 requested a review from zzhou1 January 13, 2025 12:30

liangxin1300 mentioned this pull request Jan 13, 2025

crm report collect pre-migration info #1664

Open

Dev: pre-migration: add summary section to output (jsc#PED-11808)

252befd

nicholasyang2022 force-pushed the ped-8252-20241204-4.6 branch from 94926c5 to 252befd Compare January 14, 2025 03:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[crmsh-4.6] Dev: pre-migration: implement pre-migration checks for SLES 16 (jsc#PED-11808) #1629

[crmsh-4.6] Dev: pre-migration: implement pre-migration checks for SLES 16 (jsc#PED-11808) #1629

nicholasyang2022 commented Dec 13, 2024 •

edited

Loading

codecov bot commented Dec 16, 2024 •

edited

Loading

liangxin1300 Jan 1, 2025 •

edited

Loading

nicholasyang2022 Jan 1, 2025

liangxin1300 commented Jan 3, 2025 •

edited

Loading

liangxin1300 commented Jan 3, 2025 •

edited

Loading

liangxin1300 Jan 3, 2025

liangxin1300 Jan 3, 2025

liangxin1300 Jan 13, 2025

nicholasyang2022 Jan 13, 2025

liangxin1300 Jan 13, 2025

liangxin1300 Jan 13, 2025

liangxin1300 Jan 13, 2025

nicholasyang2022 Jan 13, 2025

liangxin1300 Jan 13, 2025

nicholasyang2022 Jan 13, 2025

liangxin1300 commented Jan 13, 2025

nicholasyang2022 commented Jan 13, 2025

liangxin1300 Jan 13, 2025 •

edited

Loading

nicholasyang2022 Jan 13, 2025

liangxin1300 Jan 13, 2025 •

edited

Loading

liangxin1300 Jan 13, 2025

liangxin1300 left a comment

		)


		def has_primitive_filesystem_ocfs2(cib: lxml.etree.Element) -> bool:

		sys.stdout.write(' Good to migrate to SLES 16.\n\n')


		def check(args: typing.Sequence[str]) -> int:

[crmsh-4.6] Dev: pre-migration: implement pre-migration checks for SLES 16 (jsc#PED-11808) #1629

Are you sure you want to change the base?

[crmsh-4.6] Dev: pre-migration: implement pre-migration checks for SLES 16 (jsc#PED-11808) #1629

Conversation

nicholasyang2022 commented Dec 13, 2024 • edited Loading

Use Cases

crm cluster health sles16

crm cluster health sles16 --fix

When some of the nodes are offline

codecov bot commented Dec 16, 2024 • edited Loading

Codecov Report

liangxin1300 Jan 1, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liangxin1300 commented Jan 3, 2025 • edited Loading

Suggestion:

1. Add completer for crm cluster health, now hawk2 can be completed, so do sles16.

2. Add success info

3. Add failure error

4. Add above output into /var/log/crmsh/crmsh.log log file, too

5. Keep the color of node line consistency

liangxin1300 commented Jan 3, 2025 • edited Loading

6. How about there are Failed Resource Actions?

7. crm report collect pre-migration info

8. Inaccurate usage?

9. Fence agents checking

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liangxin1300 commented Jan 13, 2025

nicholasyang2022 commented Jan 13, 2025

liangxin1300 Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liangxin1300 Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liangxin1300 left a comment

Choose a reason for hiding this comment

nicholasyang2022 commented Dec 13, 2024 •

edited

Loading

`crm cluster health sles16`

`crm cluster health sles16 --fix`

codecov bot commented Dec 16, 2024 •

edited

Loading

liangxin1300 Jan 1, 2025 •

edited

Loading

liangxin1300 commented Jan 3, 2025 •

edited

Loading

1. Add completer for `crm cluster health`, now `hawk2` can be completed, so do `sles16`.

4. Add above output into `/var/log/crmsh/crmsh.log` log file, too

liangxin1300 commented Jan 3, 2025 •

edited

Loading

liangxin1300 Jan 13, 2025 •

edited

Loading

liangxin1300 Jan 13, 2025 •

edited

Loading