[Feature] Run Ginkgo tests with Ginkgo CLI

[MortalHappiness]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Currently, Ginkgo tests are run with go test, and we cannot run a single test, which is annoying. We should use the Ginkgo CLI to run Ginkgo tests.

Implementation Details

• Please search for Describe( using your IDE to identify which files contain Ginkgo tests.
• Try running the tests locally using the Ginkgo CLI and fix any errors that occur.
• Attempt to run a single test with the Ginkgo CLI; it should pass. Update the test to intentionally make it fail, then run it again with the Ginkgo CLI to ensure it fails as expected.
• Update the configurations in the .github/ and .buildkite/ folders to use the Ginkgo CLI instead of go test for running Ginkgo tests.

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[nadongjun]

@MortalHappiness Hi! I’d like to take this on—please assign it to me if possible. Thanks!

[dentiny]

Does go test -run work here?
Eg: https://stackoverflow.com/questions/26092155/just-run-single-test-instead-of-the-whole-suite

[MortalHappiness]

Does go test -run work here? Eg: https://stackoverflow.com/questions/26092155/just-run-single-test-instead-of-the-whole-suite

No, it only works for regular Golang tests, not Ginkgo tests.

[dentiny]

No, it only works for regular Golang tests, not Ginkgo tests.

Hmm would this work?
https://stackoverflow.com/questions/68008485/how-to-run-a-single-spec-using-ginkgo-test-framework

[nadongjun]

I've tested running specific Ginkgo tests using go test -v -ginkgo.focus "Describe", -ginkgo.skip "Describe", and it works as expected.

Given these benefits, do you think switching from go test to the Ginkgo CLI would be a good move?

[MortalHappiness]

do you think switching from go test to the Ginkgo CLI would be a good move?

Yes. I still think it's better to switch to use Ginkgo CLI, because it has more features.

[andrewsykim]

I forgot about -ginkgo.focus, for a short-term fix can we allow setting ginkgo.focus with make test? That should be a short/simple PR that we can do while implementing make test to use Ginkgo CLI

[nadongjun]

Upon reviewing the issue, it was confirmed that a conflict occurs when Ginkgo tests are mixed with standard Go tests. (When running Ginkgo --focus, all tests defined in the standard go test are also executed.) This happens because, to use Ginkgo with go test, all test code must be written in the Ginkgo format. The flags -ginkgo.focus and -ginkgo.skip only work with tests written in the Ginkgo framework.

Conflicts can arise when Ginkgo tests are mixed with standard Go tests, and to resolve this, all tests need to be standardized to the Ginkgo format. Converting all tests to the Ginkgo format will require considerable time and effort, making it a long-term task. For further reference on similar issues, you can check Ginkgo Issue #358.

[MortalHappiness]

Could we use ginkgo convert as mentioned in that issue?

[andrewsykim]

As a temporary measure, can we introduce a new make target that just runs the ginkgo tests, with support for passing in a focus string? We dont need to run it through CI, it's just for local development

[nadongjun]

@MortalHappiness
Regarding the use of ginkgo convert, it has been deprecated starting from Ginkgo v2.

@andrewsykim
Sure, I've added a new make target that runs the Ginkgo tests with the option to pass in a focus string. This is intended for local development, so there's no need to run it through CI. The target requires the TEST_NAME and the path to the test folder to be specified. If go test is available, it runs the test with the provided TEST_NAME. If not, it falls back to running the Ginkgo tests with the focus string. Here's how it works:

Script:

test-target: WHAT ?= ./controllers/ray/
test-target: TEST_NAME ?= ""
test-target: ENVTEST_K8S_VERSION ?= 1.24.2
test-target: manifests fmt vet envtest
	@go test -run $(TEST_NAME) $(WHAT) -count=1 | tee >(if grep -q "no tests to run"; then \
		echo "No matching Go tests found. Running Ginkgo tests…"; \
		KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) —bin-dir $(LOCALBIN) -p path)" ginkgo —focus "$(TEST_NAME)" $(WHAT); \
	fi) 

For ginkgo test:

make test-target TEST_NAME=RayClusterwithinvalidNumOfHosts WHAT=./controllers/ray/

For go test:

make test-target TEST_NAME=TestCreateRayJobSubmitterIfNeed

If this approach looks reasonable, I will proceed to create a PR.