WIP: Improvements for networkless testing (#2128 revamped) #2174
Conversation
|
These are the remaining TODO items before I'd feel comfortable asking this to be merged:
To generate more cassettes, we need to run |
|
OK, I think I managed -- just needed to insert them into the |
hseg
force-pushed
the
fix-vcrs
branch
6 times, most recently
from
4252057 to
3d8492e
Compare
|
Hey you need help with anything? I have some free time today. Let me know if you need help with something specific, if not I will try to work on the thing you mentioned above. ;) |
|
The one thing I still need help with is figuring out why the yt-dlp invocations aren't frozen by simply using vcrpy and freezegun. Talking to the yt-dlp folks, they suggested this may be because yt-dlp does its own request library dispatch, and recommended building a mock RequestHandler for this, pointing me to yt_dlp.networking and YoutubeDL.urlopen()
I haven't yet taken the time to try the suggestion and won't have time for it today, but if you could check it out it would be appreciated.
El 2 de septiembre de 2024 16:35:38 GMT+03:00, kuba ***@***.***> escribió:
…Hey you need help with anything? I have some free time today. Let me know if you need help with something specific, if not I will try to work on the thing you mentioned above. ;)
--
Reply to this email directly or view it on GitHub:
#2174 (comment)
You are receiving this because you authored the thread.
Message ID: ***@***.***>
|
|
Weirdly enough, the tests now pass. I've changed nothing in the past month, only |
|
@xnetcat do you want me to rebase? |
|
Tried running the tests, but I got a lot of errors.
Cassettes removed themselves, but didn't regenerate |
|
Weird, but at least confirms situation was too good to be true. |
|
... 🤦 the script I was using to run was only checking |
|
On my end, with and on system |
|
OK, the |
|
oops, forgot to squash first |
hseg
force-pushed
the
fix-vcrs
branch
3 times, most recently
from
f83bac0 to
8871741
Compare
|
Of the errors we had:
Progress, though! |
|
BTW, I've noticed the testsuites take a long while to run on CI -- ~5h in the latest run. Any idea what's going on with that? |
|
Hrm. Weirdly enough, on chroot: but on system: |
No idea. I cant even finish our current tests on my machine. They just get stuck forever. But I think I've investigated it once and it could be caused by Spotify api not returning the response and and raising an exception even with the lowest timeout in the spotipy library |
|
Wait, checking the failures from the latest CI run, I don't understand the purpose of the `test-vcr` workflow. Surely we're not testing the recording capabilities of vcrpy? A step generating new cassettes makes sense as part of release engineering or bugfixing, but surely it doesn't need to be run on every CI run?
Incidentally, I notice we're on the deprecated v1 of setup-ffmpeg. Is anything blocking the upgrade?
El 14 de octubre de 2024 19:12:19 GMT+03:00, kuba ***@***.***> escribió:
…> BTW, I've noticed the testsuites take a long while to run on CI -- ~5h in the latest run. Any idea what's going on with that?
No idea. I cant even finish our current tests on my machine. They just get stuck forever.
But I think I've investigated it once and it could be caused by Spotify api not returning the response and and raising an exception even with the lowest timeout in the spotipy library
--
Reply to this email directly or view it on GitHub:
#2174 (comment)
You are receiving this because you authored the thread.
Message ID: ***@***.***>
|
Some tests are run using the vcr and some are run with real networking to verify that the responses didn't change too drastically and that the underlying code still works. |
|
I've decided to leave def genVCRpyRH(vcr):
class VCRpyRH(RequestHandler):
...
def _send(req):
return vcr.play_response(req)
return VCRpyRH
@pytest.mark.vcr()
def test_convert(vcr, ...)
rh = genVCRpyRH(vcr)
register_rh(rh)
...
del _REQUEST_HANDLERS[rh.RH_KEY] |
I guess that's fair. Have you given thought to using |
|
(Just rebased for 4.2.9, I'm busy with other stuff right now) |
Cache network responses to focus on testing our consumption of the API. This makes the tests more robust to API rate-limiting and denoises them from network problems. - tests/console/test_entry_point.py::test_download_song - tests/console/test_entry_point.py::test_preload_song - tests/test_init.py::test_download - tests/test_init.py::test_get_urls - tests/test_matching.py::test_ytmusic_matching - tests/utils/test_ffmpeg.py::test_convert - tests/utils/test_m3u.py::test_create_m3u_content - tests/utils/test_m3u.py::test_create_m3u_file - tests/utils/test_metadata.py::test_embed_metadata - tests/utils/test_search.py::test_parse_query
Having trouble getting yt-dlp invocations to work nicely with vcrpy, suspect this is due to it receiving responses that are time-limited Affected tests: - tests/console/test_entry_point.py::test_download_song - tests/console/test_entry_point.py::test_preload_song - tests/test_init.py::test_get_urls - tests/test_init.py::test_download - tests/utils/test_ffmpeg.py::test_convert - tests/utils/test_metadata.py::test_embed_metadata
This enables them to be picked up by vcrpy
This is necessary to allow testing with --block-network -- just pass -m 'not novcr' as well to disable these tests. Tests affected: - tests/utils/test_ffmpeg.py::test_download_ffmpeg However, I doubt whether this function is even necessary -- shouldn't we be providing binaries with ffmpeg bundled instead for those users who can't install it conveniently on their machines?
To smooth out cassette regeneration workflow -- am constantly getting HTTP 401/429 responses, which is masking the cause of test failures.
Some of these were forgotten -- marked, but never generated. Some would raise errors, indicating cached broken responses. Some of these needed to be regenerated so that the timestamp given to freezegun would be withing tolerable distance of their simulated request times -- given we're using a single freeze time for the entire project, all externally-facing cassettes (in our case, facing yt-dlp) need to be regenerated on each freezegun shift.
Cache network responses more widely
Description
The network-based tests are missing saved API responses in some cases.
This makes running them hit the various API endpoints and suffer the resulting
rate-limiting. While #2142 is tracking the long-term fix, there's no reason not
to pick up this low-hanging fruit.
Where this is impossible, mark the tests so they can more easily be skipped if a
dev wants to avoid hitting the network.
Finally, while I'm generating the cassettes, I added
pytest-vcr-delete-on-failas a dependency. It isn't strictly necessary afterwards, so I will be editing
this PR to remove the relevant commit at the end if so desired.
(Historical note: this PR used to be #2128, but I had made it hard to rebase on
top of
master, so I'm reopening it. Plus, given the scope creep on that PR, Ithought a clean slate might make it easier to follow what's going on)
Newly-cached tests:
tests/console/test_entry_point.py::test_download_songtests/console/test_entry_point.py::test_preload_songtests/test_init.py::test_downloadtests/test_init.py::test_get_urlstests/test_matching.py::test_ytmusic_matchingtests/utils/test_ffmpeg.py::test_converttests/utils/test_m3u.py::test_create_m3u_contenttests/utils/test_m3u.py::test_create_m3u_filetests/utils/test_metadata.py::test_embed_metadatatests/utils/test_search.py::test_parse_queryNewly-marked tests:
tests/utils/test_ffmpeg.py::test_download_ffmpegRelated Issue
Closes: #2127
Closes: #2128
Motivation and Context
See above: this should make the testsuite less dependent on the network, making
it more robust both in the face of rate-limiting and in the face of flaky
networks.
How Has This Been Tested?
I am currently testing this in a clean
systemd-nspawncontainer running ArchLinux with the minimal dependencies needed to build the AUR package.
Screenshots (if appropriate)
Types of Changes
Checklist