Help supporting flaky tests using Gradle

[knsonier]

I have a java project that is using Gradle and "org.gradle.test-retry" for retrying failed tests. From the Gradle documentation it looks like there is an option called mergeReruns (https://docs.gradle.org/7.3/userguide/java_testing.html#mergereruns) that:

When mergeReruns is enabled, if a test fails but is then retried and succeeds, its failures will be recorded as instead of , within one . This is effectively the reporting produced by the surefire plugin of Apache Maven™ when enabling reruns. If your CI server understands this format, it will indicate that the test was flaky. If it does not, it will indicate that the test succeeded as it will ignore the information. If the test does not succeed (i.e. it fails for every retry), it will be indicated as having failed whether your tool understands this format or not.

However, I am still having problems as when the check run is created, it still produces an X and the test fails on a flaky. I referenced this issue here and it seems like this should be supported? One thing I noticed that seemed different was the Job Summary usually had a link in it which would take me to the failure but now it does not.

Before enabling mergeReruns I would get something like this:

And after I enabled mergeReruns I see the "For more details" line disappeared

I can not figure out what I'm doing incorrect here. Any help would be appreciated! Thank you.

[EnricoMi]

For that to understand it would help to look into the relevant bits of the test results XML file. First I would identify the tests that fail, second I would extract the relevant <testcase> elements from all test result files.