Skip to content

Commit

Permalink
Update README
Browse files Browse the repository at this point in the history
  • Loading branch information
mtodor committed Jan 10, 2025
1 parent 94f522b commit eca6c51
Showing 1 changed file with 54 additions and 2 deletions.
56 changes: 54 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# junit2jira

Convert test failures to jira issues
Utility tools for handling test failures

### Build
```shell
Expand All @@ -14,6 +14,16 @@ go test ./...

### Usage

This repo provides two cli tools:
- junit2jira
- flakechecker

### junit2jira

`junit2jira` supports conversion of test failures to jira issues. It also posts Slack messages for new failures and imports test results into DB.

*Usage*

```shell
Usage of junit2jira:
-base-link string
Expand Down Expand Up @@ -51,7 +61,7 @@ Usage of junit2jira:
print version information and exit
```
## Example usage
*Example usage*
```shell
JIRA_TOKEN="..." junit2jira \
-jira-url "https://..." \
Expand All @@ -65,3 +75,45 @@ JIRA_TOKEN="..." junit2jira \
-timestamp $(date --rfc-3339=seconds)
-csv-output -
```
### flakechecker
`flakechecker` helps prevent unnecessary CI pipeline failures by suppressing known flaky tests that are within the allowed failure thresholds.
`flakechecker` relies on several components:
- collected test results from `junit2jira`: we generate a table of flaky tests, including their failure ratios for the last 30 executions.
- flaky test configuration: we define and provide a `flakechecker` configuration with allowed failure ratio thresholds for known flaky tests.
- CI pipeline integration script: `flakechecker` is executed as the last step in a CI pipeline, and provided results allow the CI pipeline script to report success or failure.
The `flakechecker` expects at least one failed test. It will return an error if it is executed on test results without any failures.
`flakechecker` decision making:
- it checks if a failed test in a CI pipeline is listed as flaky in the provided configuration.
- if the test is not found in the flaky tests config -> it will cause the CI pipeline to fail. (test not found)
- if the test is found in the configuration, `flakechecker` will fetch information about the fail ratio for that test from the database. If we have fewer than 30 executions for that test -> it will cause the CI pipeline to fail. (insufficient historical test results)
- if the test's failure ratio in the database exceeds the threshold defined in the config -> it will cause the CI pipeline to fail. (flake ratio is above the allowed threshold)
- if a flaky test's failure ratio is below the defined threshold -> it will report the test as a success in the CI pipeline. (test suppression)
The `flakechecker` will apply this logic for each failed test in the CI pipeline.
*Usage*
```
Usage of flakechecker:
-config-file string
Config file with allowed flakes.
-debug
Enable debug log level.
-job-name string
Name of CI job.
-junit-reports-dir string
Directory containing JUnit report XML files.
-v short alias for -version
-version
print version information and exit
```
*Example usage*
```
flakechecker --config-file flake-config.yml --job-name "${JOB_NAME}" -junit-reports-dir "${ARTIFACT_DIR}"
```

0 comments on commit eca6c51

Please sign in to comment.