Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ResourceWarning in urllib tests #128734

Closed
serhiy-storchaka opened this issue Jan 11, 2025 · 1 comment
Closed

ResourceWarning in urllib tests #128734

serhiy-storchaka opened this issue Jan 11, 2025 · 1 comment
Labels
3.12 bugs and security fixes 3.13 bugs and security fixes 3.14 new features, bugs and security fixes tests Tests in the Lib/test dir

Comments

@serhiy-storchaka
Copy link
Member

serhiy-storchaka commented Jan 11, 2025 •
edited by bedevere-app bot
Loading

Many urllib tests produce resource warnings. For example (there are much more cases):

$ ./python -We -m test -vuall test_urllib test_urllib2 test_urllib2_localnet test_urllib_response test_urllibnet
...
test_geturl (test.test_urllib.urlopen_DataTests.test_geturl) ... ok
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697425919168 whose fp = <_io.BytesIO object at 0x7ff6abc15400>>
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697425378528 whose fp = <_io.BytesIO object at 0x7ff6abc15390>>
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697425378176 whose fp = <_io.BytesIO object at 0x7ff6ac21be70>>
test_info (test.test_urllib.urlopen_DataTests.test_info) ... Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697420160224 whose fp = <_io.BytesIO object at 0x7ff6abc154e0>>
ok
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697420342752 whose fp = <_io.BytesIO object at 0x7ff6abc15400>>
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697420343392 whose fp = <_io.BytesIO object at 0x7ff6abc15390>>
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697425919168 whose fp = <_io.BytesIO object at 0x7ff6ac21be70>>
test_interface (test.test_urllib.urlopen_DataTests.test_interface) ... ok
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697426455808 whose fp = <_io.BytesIO object at 0x7ff6abc15400>>
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697426462784 whose fp = <_io.BytesIO object at 0x7ff6abc15390>>
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697426467104 whose fp = <_io.BytesIO object at 0x7ff6ac21be70>>
test_invalid_base64_data (test.test_urllib.urlopen_DataTests.test_invalid_base64_data) ... ok
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697426847840 whose fp = <_io.BytesIO object at 0x7ff6abc15400>>
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697426847072 whose fp = <_io.BytesIO object at 0x7ff6abc15390>>
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697426455808 whose fp = <_io.BytesIO object at 0x7ff6ac21be70>>
test_missing_comma (test.test_urllib.urlopen_DataTests.test_missing_comma) ... ok
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697425844896 whose fp = <_io.BytesIO object at 0x7ff6abc15400>>
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697425844000 whose fp = <_io.BytesIO object at 0x7ff6abc15390>>
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697419659664 whose fp = <_io.BytesIO object at 0x7ff6ac21be70>>
test_read_image (test.test_urllib.urlopen_DataTests.test_read_image) ... ok
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697420324640 whose fp = <_io.BytesIO object at 0x7ff6abc15400>>
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697419712848 whose fp = <_io.BytesIO object at 0x7ff6abc15390>>
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697419711184 whose fp = <_io.BytesIO object at 0x7ff6ac21be70>>
test_read_text (test.test_urllib.urlopen_DataTests.test_read_text) ... ok
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697420324832 whose fp = <_io.BytesIO object at 0x7ff6abc15400>>
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697420324448 whose fp = <_io.BytesIO object at 0x7ff6abc15390>>
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697420323104 whose fp = <_io.BytesIO object at 0x7ff6ac21be70>>
test_read_text_base64 (test.test_urllib.urlopen_DataTests.test_read_text_base64) ... ok
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697420324640 whose fp = <_io.BytesIO object at 0x7ff6abc15400>>
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697420325216 whose fp = <_io.BytesIO object at 0x7ff6abc15390>>
Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <addinfourl at 140697420323104 whose fp = <_io.BytesIO object at 0x7ff6ac21be70>>
...
test_invalid_redirect (test.test_urllib.urlopen_HttpTests.test_invalid_redirect) ... Warning -- Unraisable exception
Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x7ff6ac410dd0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/tempfile.py", line 483, in __del__
    _warnings.warn(self.warn_message, ResourceWarning)
ResourceWarning: Implicitly cleaning up <HTTPError 302: "Found - Redirection to url 'file://guidocomputer.athome.com:/python/license' is not allowed">
ok
...
5 tests altered the execution environment (env changed):
    test_urllib test_urllib2 test_urllib2_localnet
    test_urllib_response test_urllibnet
...

The reason is that addinfourl() is a subclass of tempfile._TemporaryFileWrapper. It keeps a reference to an open file and emits a warning in its destructor if was not explicitly closed.

The solution: every result of urllib.request.open() and similar functions and any HTTPError exception should be explicitly closed.

See also #128731.

Linked PRs
@serhiy-storchaka serhiy-storchaka added the tests Tests in the Lib/test dir label Jan 11, 2025
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Jan 11, 2025
@serhiy-storchaka
pythongh-128734: Fix ResourceWarning in urllib tests
d770062
@bedevere-app bedevere-app bot mentioned this issue Jan 11, 2025
Merged
serhiy-storchaka added a commit that referenced this issue Jan 12, 2025
@serhiy-storchaka
gh-128734: Fix ResourceWarning in urllib tests (GH-128735)
5ace717
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Jan 12, 2025
@serhiy-storchaka
[3.13] pythongh-128734: Fix ResourceWarning in urllib tests (pythonGH…
0fe7ba5 
…-128735)

(cherry picked from commit 5ace717)

Co-authored-by: Serhiy Storchaka <[email protected]>
@bedevere-app bedevere-app bot mentioned this issue Jan 12, 2025
Merged
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Jan 12, 2025
@serhiy-storchaka
[3.12] pythongh-128734: Fix ResourceWarning in urllib tests (pythonGH…
092e916 
…-128735)

(cherry picked from commit 5ace717)

Co-authored-by: Serhiy Storchaka <[email protected]>
@bedevere-app bedevere-app bot mentioned this issue Jan 12, 2025
Merged
@serhiy-storchaka
Copy link
Member Author

The warnings are only emitted in the main branch, but it is worth to backport the changes to prevent depending on the garbage collector. It may matter on PyPy and others implementations.

@serhiy-storchaka serhiy-storchaka added 3.12 bugs and security fixes 3.13 bugs and security fixes 3.14 new features, bugs and security fixes labels Jan 12, 2025
serhiy-storchaka added a commit that referenced this issue Jan 12, 2025
@serhiy-storchaka
[3.12] gh-128734: Explicitly close sockets in urllib tests (GH-128735) (
c20c551 
GH-128750)

(cherry picked from commit 5ace717)
serhiy-storchaka added a commit that referenced this issue Jan 12, 2025
@serhiy-storchaka
[3.13] gh-128734: Explicitly close sockets in urllib tests (GH-128735) (
6116e1b 
GH-128748)

(cherry picked from commit 5ace717)
@serhiy-storchaka serhiy-storchaka mentioned this issue Jan 12, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.12 bugs and security fixes 3.13 bugs and security fixes 3.14 new features, bugs and security fixes tests Tests in the Lib/test dir
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@serhiy-storchaka