Verify only props matching the given TestSelectors

[Duhemm]

Previously, ScalaCheck ignored the selectors that it receives as input in the "root task". This prevented users from running only a subset of properties in a specification by passing their TestSelector to ScalaCheck in a TaskDef.

This patch fixes this by having the root task program the execution of only the requested properties if the TaskDef contains only TestSelectors. ScalaCheck's behavior remains unchanged if the TaskDef contains any other kind of Selector.

[Duhemm]

Hi @satorg! Would you mind taking a look at this PR, please?

[Duhemm]

@satorg Hi! Any concerns with this PR?

[satorg]

Hi @Duhemm ! Thank you for the ping. I'm really sorry that it takes so much time to review your PR.

Honestly, I made a couple of attempts to look into it, but every time was realizing that I'm not really sure what the problem the PR is aimed to solve.

It does not mean that there's anything wrong with your PR, but rather it seems that I've never encountered that kind of issue in my personal experience.

So I would really appreciate if you could elaborate a bit more on the issue the PR is addressing, maybe with some really small examples. I believe it could help to move it forward.

Anyway, thank you for contributing to the project!

[Duhemm]

Hi @satorg, thanks for taking a look and sorry for taking so long to get back to you!

sbt's test-interface API has a concept of Selector which can be passed to a test framework, so that only the selected subset of the entire test suite is run. They make it possible to encode, in a programmatic and framework agnostic way, the command line filter that most test frameworks also support (eg. in ScalaCheck: -propFilter).

For instance, Bloop uses TestSelector to ask test frameworks to run only a selected subset of test cases within a test class: https://github.com/scalacenter/bloop/blob/c505385edf0bbd420e19401ae1beabae8895df8f/frontend/src/main/scala/bloop/engine/tasks/TestTask.scala#L343-L348

In my case, I am working on a plugin that adds test retry in sbt. When a test case ("property", in ScalaCheck) fails, it is automatically retried a few times until it passes so that the build is not blocked by flaky tests. The changes in this PR will make it possible for the plugin to retry only the failing test case instead of the entire test object.

Please let me know if you have other questions or if the benefits of this PR are still unclear.

[Duhemm]

Hi @satorg! Was my message above helpful or do you have other questions and concerns?

[satorg]

@Duhemm , thank you for the clarification and sorry again for the long review.

I'm really trying to wrap my head around the problem.

You mentioned the -propFilter option, and looks it is already working as expected:

sbt:scalacheck> coreJVM/testOnly org.scalacheck.ChooseSpecification -- -propFilter BigInt
[info] + Choose.choose[BigInt]: OK, passed 100 tests.
[info] + Choose.choose[BigInteger]: OK, passed 100 tests.
[info] + Choose.choose(BigInt( 2^(2^18 - 1)), BigInt(-2^(2^18 - 1))): OK, passed 100 tests.
[info] Passed: Total 3, Failed 0, Errors 0, Passed 3
[success] Total time: 1 s, completed Aug 31, 2024, 9:29:11 PM

I.e. only the properties with names containing "BigInt" get executed.

Is there any way to reproduce the issue you're targeting with any existing testing framework or is it specific to the plugin you're working on?