-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add cli for checking CI test flakiness #30
base: main
Are you sure you want to change the base?
Conversation
a2e05cf
to
2e9d594
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it will be good to split repackaging from the flakechecker into 2 PRs.
2e9d594
to
6619bc3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I started reviewing but I'm still struggling to understand the overall intent. Can you please point me at the periodic job, and maybe schedule half an hour to talk about the overall design / idea?
35ee249
to
541e548
Compare
ace9269
to
ba4fd4c
Compare
ba4fd4c
to
f6c595b
Compare
3a7905e
to
8eac3d1
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good overall, but I have a whole bunch of nitpicks.
Also, it would be good to have documentation for this somewhere. Nothing fancy, basically what is in the PR description (after bringing it up to date) plus maybe a couple of sentences on the rationale. |
8eac3d1
to
705a2c1
Compare
705a2c1
to
0dfefc4
Compare
0dfefc4
to
eca6c51
Compare
This PR is introducing a new cli tool for evaluating if tests are known as flaky tests or not.
How it works
Flakechecker will handle information in the following way:
it checks all found failed jobs (logic taken from junit2jira)
for every failed test, it will check if it is defined as a flaky test in provided config
if a test is not found in known flaky tests configuration -> it will fail
if a test is found, but it didn't have 30 executions -> it will fail
if a test is found, but the defined threshold found in DB is above the definition in configuration -> it will fail
only if a test is found and the failure ratio in DB is below the defined threshold in config -> it will be reported as success to the CI pipeline - for that, we also need: chore(CI): Add config and call for flakechecker stackrox#13081