Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

updater: make UpdateListIter a proper iterator #238

Merged
merged 1 commit into from
Jan 5, 2025

Conversation

DemiMarie
Copy link

@DemiMarie DemiMarie commented Dec 22, 2024

Python iterators are required to implement an __iter__() that returns
self 1. CPython doesn't check this consistently, but the requirement
is still there, so the existing code is buggy. Python 3.13 started
checking this in list comprehensions, resulting in exceptions being
thrown. Fix the bug by having __iter__() return self, as required by
the iterator protocol.

This worked on Python 3.13 and below, but broke in 3.13.1 2.

Copy link

codecov bot commented Dec 22, 2024

Codecov Report

Attention: Patch coverage is 50.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 93.17%. Comparing base (2b12854) to head (1231efc).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
qui/updater/utils.py 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #238      +/-   ##
==========================================
- Coverage   93.18%   93.17%   -0.01%     
==========================================
  Files          58       58              
  Lines       11046    11048       +2     
==========================================
+ Hits        10293    10294       +1     
- Misses        753      754       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@marmarek
Copy link
Member

If that's a CPython bug, please report it properly (and then put bug reference in the commit message). But it's more likely buggy usage somewhere (possibly in this repo). Do you have the call trace of problematic situation?

@DemiMarie
Copy link
Author

DemiMarie commented Dec 23, 2024

If that's a CPython bug, please report it properly (and then put bug reference in the commit message).

I would, if I was able to reproduce it. The bug went away when I reinstalled the package, so it seemed like something weird (stale .pyc file?). The package I built is from my own branch, but the changes in it are not relevant as the code they alter is not used in the updater.

But it's more likely buggy usage somewhere (possibly in this repo). Do you have the call trace of problematic situation?

I don’t have the full stack trace, but I do remember that the error happened here:

selected_num = sum(row.selected for row in self.list_store)

Python complained that an UpdatesListIter was not iterable. However, the following worked:

    selected_num = 0
    for row in self.list_store:
        selected_num += i.selected

Furthermore, I found that self.list_store was a ListWrapper, not an UpdatesListIter.

@DemiMarie DemiMarie force-pushed the python-bug-workaround branch 2 times, most recently from c920daf to 84fb5ac Compare December 24, 2024 09:31
@marmarek
Copy link
Member

openQA reproduced the issue:

Dec 28 17:01:24.186710 dom0 widget-wrapper[9380]: Traceback (most recent call last):
Dec 28 17:01:24.188175 dom0 widget-wrapper[9380]:   File "/usr/lib/python3.13/site-packages/qui/updater/utils.py", line 48, in wrapper
Dec 28 17:01:24.188175 dom0 widget-wrapper[9380]:     func(self, *args, **kwargs)
Dec 28 17:01:24.188175 dom0 widget-wrapper[9380]:     ~~~~^^^^^^^^^^^^^^^^^^^^^^^
Dec 28 17:01:24.188175 dom0 widget-wrapper[9380]:   File "/usr/lib/python3.13/site-packages/qui/updater/intro_page.py", line 166, in on_checkbox_toggled
Dec 28 17:01:24.188175 dom0 widget-wrapper[9380]:     selected_num = sum(row.selected for row in self.list_store)
Dec 28 17:01:24.188175 dom0 widget-wrapper[9380]:   File "/usr/lib/python3.13/site-packages/qui/updater/intro_page.py", line 166, in <genexpr>
Dec 28 17:01:24.188175 dom0 widget-wrapper[9380]:     selected_num = sum(row.selected for row in self.list_store)
Dec 28 17:01:24.188175 dom0 widget-wrapper[9380]:                                                ^^^^^^^^^^^^^^^
Dec 28 17:01:24.188906 dom0 widget-wrapper[9380]: TypeError: 'UpdateListIter' object is not iterable
Dec 28 17:01:27.597130 dom0 widget-wrapper[9380]: Traceback (most recent call last):
Dec 28 17:01:27.597815 dom0 widget-wrapper[9380]:   File "/usr/lib/python3.13/site-packages/qui/updater/updater.py", line 215, in next_clicked
Dec 28 17:01:27.597815 dom0 widget-wrapper[9380]:     vms_to_update = self.intro_page.get_vms_to_update()
Dec 28 17:01:27.597815 dom0 widget-wrapper[9380]:   File "/usr/lib/python3.13/site-packages/qui/updater/intro_page.py", line 147, in get_vms_to_update
Dec 28 17:01:27.597815 dom0 widget-wrapper[9380]:     return self.list_store.get_selected()
Dec 28 17:01:27.597815 dom0 widget-wrapper[9380]:            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
Dec 28 17:01:27.597815 dom0 widget-wrapper[9380]:   File "/usr/lib/python3.13/site-packages/qui/updater/utils.py", line 306, in get_selected
Dec 28 17:01:27.597815 dom0 widget-wrapper[9380]:     selected_rows = [row for row in self if row.selected]
Dec 28 17:01:27.597815 dom0 widget-wrapper[9380]:                                     ^^^^
Dec 28 17:01:27.597815 dom0 widget-wrapper[9380]: TypeError: 'UpdateListIter' object is not iterable

https://openqa.qubes-os.org/tests/123911#investigation says the python3 version in dom0 didn't change since last good run, but the desktop-linux-manager version did change, so it's most likely a regression in this repo - maybe in the other PR here?

@DemiMarie
Copy link
Author

DemiMarie commented Dec 29, 2024

https://openqa.qubes-os.org/tests/123911#investigation says the python3 version in dom0 didn't change since last good run, but the desktop-linux-manager version did change, so it's most likely a regression in this repo - maybe in the other PR here?

Possibly. From what I can tell, the existing code relies on undefined behavior, so it could break at any time.

Did your merge scripts start including the PR message? In the past I have let that get stale: for instance, this PR is not a workaround for a CPython bug, but rather a bug fix for relying on undocumented behavior in CPython. Also, the PR message is meant to be viewed as markdown, rather than as plain text, so it won’t look as good when incorporated into a commit mesage.

@DemiMarie DemiMarie changed the title updater: workaround possible CPython bug updater: make UpdateListIter a proper iterator Dec 31, 2024
@DemiMarie DemiMarie force-pushed the python-bug-workaround branch from 84fb5ac to 41ba839 Compare December 31, 2024 07:10
@qubesos-bot
Copy link

qubesos-bot commented Dec 31, 2024

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025010404-4.3&flavor=pull-requests

Test run included the following:

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2024111705-4.3&flavor=update

  • system_tests_whonix@hw7

    • startup: Failed (test died)
      # Test died: command 'qvm-run --nogui -u root sys-firewall qvm-sync...
  • system_tests_gui_interactive@hw1

    • screenlocker_leaks: Failed (test died)
      # Test died: command 'qvm-run -p -u root work "dnf -y install mate-...
  • system_tests_suspend@hw1

    • suspend: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...
  • system_tests_basic_vm_qrexec_gui@hw7

    • startup: Failed (test died)
      # Test died: command 'qvm-run --nogui -u root sys-firewall qvm-sync...
  • system_tests_gui_tools@hw7

    • startup: Failed (test died)
      # Test died: command 'qvm-run --nogui -u root sys-firewall qvm-sync...
  • system_tests_whonix

    • whonixcheck: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...
  • system_tests_gui_tools

    • desktop_linux_manager_policy_edit: unnamed test (unknown)
    • desktop_linux_manager_policy_edit: Failed (test died)
      # Test died: no candidate needle with tag(s) 'qubes-policy-editor-g...
  • system_tests_suspend

    • suspend: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...
  • system_tests_gui_interactive

    • screenlocker_leaks: Failed (test died)
      # Test died: command 'qvm-run -p -u root work "dnf -y install mate-...
  • system_tests_guivm_gui_interactive

    • update_guivm: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...
  • system_tests_backupdispvm

    • startup: wait_serial (wait serial expected + timed out)
      # wait_serial expected: "# "...

    • startup: wait_serial (wait serial expected)
      # wait_serial expected: qr/xE1Ra-\d+-/...

    • startup: Failed (test died + timed out)
      # Test died: command 'export TERM=dumb; stty cols 2048 rows 25' tim...

    • startup: wait_serial (wait serial expected)
      # wait_serial expected: "# "...

    • startup: wait_serial (wait serial expected)
      # wait_serial expected: qr/2E8vz-\d+-/...

  • system_tests_kde_gui_interactive

    • kde_install: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...
  • system_tests_guivm_vnc_gui_interactive

    • update_guivm: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...
  • system_tests_basic_vm_qrexec_gui_btrfs

    • switch_pool: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...
  • system_tests_basic_vm_qrexec_gui_ext4

    • switch_pool: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...
  • system_tests_basic_vm_qrexec_gui_xfs

    • switch_pool: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...

Failed tests

22 failures
  • system_tests_whonix@hw7

    • startup: Failed (test died)
      # Test died: command 'qvm-run --nogui -u root sys-firewall qvm-sync...
  • system_tests_gui_interactive@hw1

    • screenlocker_leaks: Failed (test died)
      # Test died: command 'qvm-run -p -u root work "dnf -y install mate-...
  • system_tests_suspend@hw1

    • suspend: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...
  • system_tests_basic_vm_qrexec_gui@hw7

    • startup: Failed (test died)
      # Test died: command 'qvm-run --nogui -u root sys-firewall qvm-sync...
  • system_tests_gui_tools@hw7

    • startup: Failed (test died)
      # Test died: command 'qvm-run --nogui -u root sys-firewall qvm-sync...
  • system_tests_whonix

    • whonixcheck: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...
  • system_tests_gui_tools

    • desktop_linux_manager_policy_edit: unnamed test (unknown)
    • desktop_linux_manager_policy_edit: Failed (test died)
      # Test died: no candidate needle with tag(s) 'qubes-policy-editor-g...
  • system_tests_suspend

    • suspend: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...
  • system_tests_gui_interactive

    • screenlocker_leaks: Failed (test died)
      # Test died: command 'qvm-run -p -u root work "dnf -y install mate-...
  • system_tests_guivm_gui_interactive

    • update_guivm: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...
  • system_tests_backupdispvm

    • startup: wait_serial (wait serial expected + timed out)
      # wait_serial expected: "# "...

    • startup: wait_serial (wait serial expected)
      # wait_serial expected: qr/xE1Ra-\d+-/...

    • startup: Failed (test died + timed out)
      # Test died: command 'export TERM=dumb; stty cols 2048 rows 25' tim...

    • startup: wait_serial (wait serial expected)
      # wait_serial expected: "# "...

    • startup: wait_serial (wait serial expected)
      # wait_serial expected: qr/2E8vz-\d+-/...

  • system_tests_kde_gui_interactive

    • kde_install: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...
  • system_tests_basic_vm_qrexec_gui_zfs

    • switch_pool: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...
  • system_tests_guivm_vnc_gui_interactive

    • update_guivm: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...
  • system_tests_basic_vm_qrexec_gui_btrfs

    • switch_pool: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...
  • system_tests_basic_vm_qrexec_gui_ext4

    • switch_pool: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...
  • system_tests_basic_vm_qrexec_gui_xfs

    • switch_pool: Failed (test died)
      # Test died: command 'qvm-run --no-gui -p -u root sys-net "command ...

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/119126#dependencies

3 fixed
  • system_tests_audio@hw1

  • system_tests_extra

    • TC_00_QVCTest_whonix-gateway-17: test_010_screenshare (failure)
      ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 0 == 0
  • system_tests_kde_gui_interactive

    • gui_keyboard_layout: Failed (test died)
      # Test died: command 'test "$(cd ~user;ls e1*)" = "$(qvm-run -p wor...

Unstable tests

  • system_tests_update@hw1

    update2/Failed (1/5 times with errors)
    • job 121711 # Test died: command '(set -o pipefail; qubesctl --show-output stat...
  • system_tests_update@hw7

    update2/Failed (1/5 times with errors)
    • job 121711 # Test died: command '(set -o pipefail; qubesctl --show-output stat...
  • system_tests_update

    update2/Failed (1/5 times with errors)
    • job 121711 # Test died: command '(set -o pipefail; qubesctl --show-output stat...

Copy link
Member

@marmarek marmarek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ugh, in the meantime black formatting is enforced and it doesn't like this: https://gitlab.com/QubesOS/qubes-desktop-linux-manager/-/jobs/8763230803

Python iterators are required to implement an __iter__() that returns
self [1].  CPython doesn't check this consistently, but the requirement
is still there, so the existing code is buggy.  Python 3.13 started
checking this in list comprehensions, resulting in exceptions being
thrown.  Fix the bug by having __iter__() return self, as required by
the iterator protocol.

This worked on Python 3.13 and below, but broke in 3.13.1 [2].

[1]: https://docs.python.org/3/glossary.html#term-iterator
[2]: python/cpython#128211
@marmarek marmarek merged commit 1231efc into QubesOS:main Jan 5, 2025
3 of 5 checks passed
@DemiMarie DemiMarie deleted the python-bug-workaround branch January 6, 2025 01:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants