Include generated input with each test.chuck checking block error

[zjroth]

CIDER includes generated input in reports when the generated input is made available, which happens when using test.chuck and its checking block. Currently, generated input is made available only to the first test condition within a checking block. Unfortunately, this means that the generated input is not reported anywhere in CIDER unless the first test condition fails. As a result, the generated input is (in my experience) typically not reported in CIDER.

This PR addresses that issue by modifying the cider.nrepl.middleware.test/current-report atom selectively based on whether a test.chuck testing context is available, which results in the presence of a non-nil value in the :gen-input key of the report precisely when the report corresponds to an error generated in a checking block.

cc: @daemianmack

[alexander-yakushev]

I don't think I ever used test.chuck (just the regular test.check). Could you please make a small reproducer to make it possible to validate the changes? A cider-nrepl test case would be nice too, but we can start with a reproducer and see if a proper test can be written.

[bbatsov]

@zjroth ping :-)

[zjroth]

Thanks for the response, @alexander-yakushev, and for the ping, @bbatsov. Apologies for the delay in this unfortunately busy season for me.

Could you please make a small reproducer to make it possible to validate the changes? A cider-nrepl test case would be nice too, but we can start with a reproducer and see if a proper test can be written.

I'm not familiar with the term "reproducer" as used here, but it seems to be something like a minimal example of code for which the issue arises. Is that correct?

[zjroth]

As an example of how this situation appears to me, consider the following test:

;; [clojure.test.check.generators :as gen]
;; [com.gfredericks.test.chuck.clojure-test :as chuck :refer [checking]]
(deftest show-input-in-test-chuck-checking-block
  (checking "Generated value shown in CIDER test report (only first test fails)"
    {:seed 1}
    [x (gen/choose 1 10)]

    (is (even? x))
    (is true))

  (checking "Generated value not shown in CIDER test report (first test succeeds but subsequent test fails)"
    {:seed 3}
    [x (gen/choose 1 10)]

    (is true)
    (is (even? x))))

For me, this produces the following CIDER test report results:

Fail in show-input-in-test-chuck-checking-block
Generated value shown in CIDER test report (only first test fails)

expected: (even? x)
  actual: (not (even? 3))          
   input: [{x 3}]

Fail in show-input-in-test-chuck-checking-block
Generated value not shown in CIDER test report (first test succeeds but subsequent test fails)

expected: (even? x)          
  actual: (not (even? 7))      

Note the missing input from the second failure above. (Note also that the value of 7 shows because x is present in the test condition, which may not always be the case.)

With this PR, the test results change to the following:

Fail in show-input-in-test-chuck-checking-block
Generated value shown in CIDER test report (only first test fails)

expected: (even? x)
  actual: (not (even? 3))          
   input: [{x 3}]

Fail in show-input-in-test-chuck-checking-block
Generated value not shown in CIDER test report (first test succeeds but subsequent test fails)

expected: (even? x)          
  actual: (not (even? 7))          
   input: [{x 7}]

Note that each failure now contains an input value, which is not dependent on whether the test condition happens to contain the generated value/symbol.