Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cli for checking CI test flakiness #30

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

mtodor
Copy link

@mtodor mtodor commented Oct 22, 2024

This PR is introducing a new cli tool for evaluating if tests are known as flaky tests or not.

How it works

  1. we are collecting successes and failures in our DB
  2. there is a scheduled BigQuery query that populates a table with the failure rate for the last 30 executions
  3. flakechecker uses that information to compare with predefined thresholds in a configuration file

Flakechecker will handle information in the following way:

  • it checks all found failed jobs (logic taken from junit2jira)

  • for every failed test, it will check if it is defined as a flaky test in provided config

  • if a test is not found in known flaky tests configuration -> it will fail

  • if a test is found, but it didn't have 30 executions -> it will fail

  • if a test is found, but the defined threshold found in DB is above the definition in configuration -> it will fail

  • only if a test is found and the failure ratio in DB is below the defined threshold in config -> it will be reported as success to the CI pipeline - for that, we also need: chore(CI): Add config and call for flakechecker stackrox#13081

@mtodor mtodor force-pushed the mtodor/add-flakechecker branch 8 times, most recently from a2e05cf to 2e9d594 Compare November 11, 2024 14:03
@mtodor mtodor marked this pull request as ready for review November 11, 2024 14:47
@mtodor mtodor requested a review from porridge as a code owner November 11, 2024 14:47
@mtodor mtodor requested a review from janisz November 11, 2024 14:48
@mtodor mtodor changed the title WiP: Add cli for checking CI test flakiness Add cli for checking CI test flakiness Nov 11, 2024
Copy link
Collaborator

@janisz janisz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it will be good to split repackaging from the flakechecker into 2 PRs.

cmd/flakechecker/bq_client.go Outdated Show resolved Hide resolved
cmd/flakechecker/bq_client.go Outdated Show resolved Hide resolved
cmd/flakechecker/bq_client.go Outdated Show resolved Hide resolved
cmd/flakechecker/main.go Outdated Show resolved Hide resolved
cmd/flakechecker/main_test.go Outdated Show resolved Hide resolved
cmd/flakechecker/bq_client.go Outdated Show resolved Hide resolved
cmd/flakechecker/flake_config.go Outdated Show resolved Hide resolved
@mtodor mtodor mentioned this pull request Nov 13, 2024
@mtodor mtodor force-pushed the mtodor/add-flakechecker branch from 2e9d594 to 6619bc3 Compare November 13, 2024 14:59
@mtodor mtodor changed the base branch from main to mtodor/add-flakechecker-prefactor November 13, 2024 14:59
Copy link

@porridge porridge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I started reviewing but I'm still struggling to understand the overall intent. Can you please point me at the periodic job, and maybe schedule half an hour to talk about the overall design / idea?

cmd/flakechecker/bq_client.go Outdated Show resolved Hide resolved
cmd/flakechecker/bq_client.go Show resolved Hide resolved
cmd/flakechecker/flake_config.go Outdated Show resolved Hide resolved
@mtodor mtodor force-pushed the mtodor/add-flakechecker-prefactor branch from 35ee249 to 541e548 Compare November 14, 2024 18:35
@mtodor mtodor force-pushed the mtodor/add-flakechecker branch 2 times, most recently from ace9269 to ba4fd4c Compare November 15, 2024 15:09
@mtodor mtodor requested review from porridge and janisz November 15, 2024 15:09
@mtodor mtodor changed the base branch from mtodor/add-flakechecker-prefactor to main November 18, 2024 10:48
@mtodor mtodor force-pushed the mtodor/add-flakechecker branch from ba4fd4c to f6c595b Compare November 18, 2024 10:51
@mtodor mtodor force-pushed the mtodor/add-flakechecker branch 4 times, most recently from 3a7905e to 8eac3d1 Compare December 3, 2024 17:49
Copy link

@porridge porridge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall, but I have a whole bunch of nitpicks.

cmd/flakechecker/bq_client.go Outdated Show resolved Hide resolved
cmd/flakechecker/bq_client.go Outdated Show resolved Hide resolved
cmd/flakechecker/bq_client.go Outdated Show resolved Hide resolved
cmd/flakechecker/flake_config.go Outdated Show resolved Hide resolved
cmd/flakechecker/flake_config.go Outdated Show resolved Hide resolved
cmd/flakechecker/main.go Outdated Show resolved Hide resolved
cmd/flakechecker/main.go Outdated Show resolved Hide resolved
cmd/flakechecker/main.go Outdated Show resolved Hide resolved
cmd/flakechecker/main.go Outdated Show resolved Hide resolved
cmd/flakechecker/main.go Outdated Show resolved Hide resolved
@porridge
Copy link

porridge commented Dec 5, 2024

Also, it would be good to have documentation for this somewhere. Nothing fancy, basically what is in the PR description (after bringing it up to date) plus maybe a couple of sentences on the rationale.

@mtodor mtodor force-pushed the mtodor/add-flakechecker branch from 8eac3d1 to 705a2c1 Compare December 20, 2024 12:10
@mtodor mtodor requested a review from porridge December 20, 2024 12:13
@mtodor mtodor force-pushed the mtodor/add-flakechecker branch from 705a2c1 to 0dfefc4 Compare December 20, 2024 12:14
cmd/flakechecker/flake_config.go Outdated Show resolved Hide resolved
README.md Show resolved Hide resolved
@mtodor mtodor force-pushed the mtodor/add-flakechecker branch from 0dfefc4 to eca6c51 Compare January 10, 2025 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants