Run the test suite in a random order #2287
Conversation
matrss
force-pushed
the
test-in-random-order
branch
3 times, most recently
from
33c548d to
f5c0894
Compare
matrss
force-pushed
the
test-in-random-order
branch
2 times, most recently
from
91df126 to
b420380
Compare
|
I've again tested how reliable this is: https://github.com/Open-MSS/MSS/actions/runs/8376494045/job/22936321858. Most runs passed, but two didn't. One of them ran into the 60 second pytest timeout, the other reached the 20 minute step timeout. I think this is just a symptom of the macOS 13 runners being very much all over the place in terms of performance (runtimes vary between ~8 and ~14 minutes for the jobs that passed as well). I also think that this problem is unrelated to this PR, and could happen on current develop also. This does decrease coverage a bit, because it disables problematic tests that previously passed by chance, when running in normal and reverse order on linux. |
matrss
force-pushed
the
test-in-random-order
branch
2 times, most recently
from
f93367b to
b420380
Compare
| @@ -29,6 +28,7 @@ jobs: | |||
| sed -e "s/menuinst.*//" | | |||
| sed -e "s/.*://" > requirements.tmp.txt | |||
| cat requirements.d/development.txt >> requirements.tmp.txt | |||
| echo "pytest-randomly" >> requirements.tmp.txt | |||
There was a problem hiding this comment.
We maybe should add this to the https://github.com/Open-MSS/MSS/blob/develop/requirements.d/development.txt
or does it once it is installed immeadiatly change to randomly?
There was a problem hiding this comment.
Yes, once it is installed it will randomize the test order by default, unless you pass --randomly-dont-reorganize. That is why I decided to only add it in this workflow, until the test suite fully works with it.
Fixes #2192.
Depends on #2249.