Why does consult not use automated testing with e.g. ert?

[hab25]

With consult.el having ~5.2k lines, I expect that tests will very quickly overvalue their costs.

I expect that consult has much better things to test, but a trivial example of where tests can help is when validating custom standard values against custom types, which, if done in e.g. a git pre-push hook, could prevent the need for e.g. 8f22fbc .

In (very unlikely, I expect) case that you're not familiar with them @minad, see in this section that they're very simple to write

;;; my-tests.el -*- lexical-binding: t -*-

(defun my/ert-deftest-custom-standard-value-is-of-correct-custom-type
    (variable)
  (eval `(ert-deftest
             ,(intern
               (concat
                (symbol-name variable)
                "-custom-standard-value-is-of-correct-custom-type"))
        ()
        (should
         (widget-apply (widget-convert
                        (get ',variable 'custom-type))
                       :match
                       (custom--standard-value ',variable))))))
(my/ert-deftest-custom-standard-value-is-of-correct-custom-type 'consult-project-function)

;;; alternatively, if you like to avoid a naming things when the amount of details the name would abstract over is low
;;; use this lambda e.g. when mapping over `(custom-group-members 'consult nil)`
;; (lambda (variable)
;;   (let ((body
;;          `(should
;;            (widget-apply (widget-convert
;;                           (get ',variable 'custom-type))
;;                          :match
;;                          (custom--standard-value ',variable)))))
;;     (eval
;;      `(ert-deftest ,(intern (secure-hash 'sha512 (format "%s" body)))
;;           ()
;;         ,body))))

Then, in e.g. a git pre-push hook:

emacs -batch -l ert -l my-tests.el -f ert-run-tests-batch-and-exit

[minad]

hab25 ***@***.***> writes:

With consult.el having ~5.2k lines, I expect that tests will very quickly overvalue their costs. I expect that consult has much better things to test, but a trivial example of where tests can help is when validating custom standard values against custom types, which, if done in e.g. a git pre-push hook, could prevent the need for e.g. 8f22fbc . In (very unlikely, I expect) case that you're not familiar with them @minad, see in this section that they're very simple to write...

I am familiar with ERT. I use it heavily in Compat, see https://github.com/emacs-compat/compat/blob/main/compat-tests.el. There it is absolutely crucial to deliver the required robustness of such a base library, and the tests there are also reasonably easy to write, because we test small, isolated and often pure functions. Multiple reasons why we don't have tests for Consult so far: 1. Interactive UI code is difficult to test. Most of the complexity of Consult comes in fact from interactive and asynchronous code. Trivial tests like one catching 8f22fbc would not help with the actual difficulties. This means we mostly rely on manual testing for now, also for testing integrations with Vertico, Embark, etc. 2. Consult is not a library, but a user-facing package with less precisely specified API contracts. 3. Writing tests is time intensive and so far I've prioritized writing features. I think one reasonable start would be to add regression tests. Another possibility would be to add some kind of play back tests which test commands on a high level, as a black box. This would probably provide the greatest value right now. 4. When refactoring code, I usually try to do it in small steps, applying equivalence transformations which are guaranteed to not break. Of course it would be better to have tests, which would allow for a more relaxed approach to refactoring. But even in the presence of tests I would likely stick with my current refactoring approach. 5. Consult is widely rolled out. If I'd introduce a breakage in some heavily used featured it will get noticed in hours or a day. This is not a justification for not having tests, but an observation. ;) But by all means, if there is interest in adding tests here, this is a welcome addition.

[hab25]

Thanks for the thoughtful and information-rich answer.

There is nothing I disagree with. Thus, the below replies are only additions and two questions (both of which end with "?").

Writing tests is time intensive and so far I've prioritized writing
features.

Such a prioritization makes sense given that Consult has only recently reached 1.0; I assume during most of its prior existence it was in a more "exploratory" state where it was expected that tests would have broken too often due to "specification changes".

But by all means, if there is interest in adding tests here, this is a
welcome addition.

Trivial tests like one catching 8f22fbc would not help with the actual
difficulties.

They should be similarly trivial to maintain. While they will not greatly help with the most important problems, they can, as shown, reduce real toil. Therefore, I expect them to outvalue their ongoing maintenance cost and to be worth merging in if somebody goes through the possibly tedious process of writing them.

Consult is [...] a user-facing package with less precisely specified API contracts.

Even with such unspecified contracts, nonetheless, there may be things that are well-specified enough.

E.g.,

Expect that at least one preview buffer was displayed during the full execution of a Consult command.

Consult's documentation can give inspiration for other such safe-to-assume and unlikely-to-change expectations.

Interactive UI code is difficult to test. Most of the complexity of Consult comes in fact from interactive and asynchronous code.

I understand the use of "asynchronous", but "interactive" is surprising. Could greater support for this be low-hanging fruit that can be added to Emacs itself? Or, perhaps I'm misunderstanding and the difficulty you're referring to here simply inherits from "asynchronous" because user interaction is asynchronous from Emacs's point of view.

test commands on a high level, as a black box. This would probably provide the greatest value right now.

I agree; this seems like a great fit for Consult commands: such "end-to-end" testing provides a substantial advantage (high confidence; coverage of entire user scenarios) while, due to Consult's low reliance ¹ on difficult-to-control external programs (e.g. proprietary or accessed-via-network ones), being impacted little by the typically associated disadvantages:

• slower execution
• flakiness
• additional development costs due to the need for simulating and/or controlling out-of-process dependencies²

However, noting @minad's "Interactive UI code is difficult to test[...]" comment, the last two of the three points above are still an important disadvantage.

When refactoring code, I usually try to do it in small steps

A great approach, as humans are universally woefully limited in the number of things they can keep in mind at once.

When refactoring code, I usually try to do it in small steps, applying equivalence transformations which are guaranteed to not break.

With the usage of "guaranteed", it sounds like you're using some kind of theoretical approach. If so, this is quite intriguing ³, could you share some references or keywords I could search for to learn to use such an approach myself?

But even in the presence of tests I would likely stick with my current refactoring approach.

Understandable⁴⁵.

If I'd introduce a breakage in some heavily used featured it will get noticed in hours or a day.

Very nice, but note that not-heavily used features are not similarly included. This may make a user hesitate to customize atypically.

Consult is not a library,

To add, this leads to another argument against tests in Consult. When functions with reverse-dependencies break, these reverse-dependencies likely inherit the breakage (i.e., they are part of a larger "blast radius"). Being closer to the user, Consult suffers less from this.

However, note that Consult's functions do have some reverse dependencies, the most notable being (other) Consult functions.

Other reverse dependencies include at least 25 projects that build on top of Consult⁶; Consult being part of GNU ELPA, and therefore considered part of Emacs, could be a major factor here.

https://github.com/emacs-compat/compat/blob/main/compat-tests.el. There
it is absolutely crucial to deliver the required robustness of such a
base library

Thank you! I expect I'll be referencing this quite a bit.

Footnotes

1. In fact, in Emacs and its ecosystem, I believe most commands meets this criterion; nice! ↩

2. Async Emacs processes introduce some, I expect it to be much lower than for other processes due to Emacs's very extensive easily-accessible API. ↩

3. More theoretical approaches tend to have advantages in at least two regards. 1) they are less fragile due to less coupling with reality, which consists of tightly inter-coupled, hard-to-distinguish, hard-to-measure, variables. 2) they are more general, e.g., theory can apply to an entire class of inputs, whereas an instance of a test could only apply to one. ↩

4. It sounds like this approach is beneficial even disregarding any testing-adjacent benefits; it allows you to refactor more efficiently because, by relying on guarantees, it reduces the amount of things you have to think about. ↩ ↩²

5. Also, in a sense it is a different "test" that is cheap (perhaps free ⁴) for you, and more tests are always better (i.e., if one disregards cost, noise, performance which seems to be appropriate in this case). ↩

6. Understanding what these projects are assuming about Consult could be a great source of inspiration for tests. ↩

[minad]

I understand the use of "asynchronous", but "interactive" is surprising. Could greater support for this be low-hanging fruit that can be added to Emacs itself?

Maybe. I haven't researched this. There are test frameworks which are made to test interactive Emacs code, not sure if there is one which works particularly well or is widely used. ERT shines in for simple unit testing of small functions.

With the usage of "guaranteed", it sounds like you're using some kind of theoretical approach. If so, this is quite intriguing, could you share some references or keywords I could search for to learn to use such an approach myself?

You can search for "equational reasoning" in the context of functional programming.

[minad]

Other reverse dependencies include at least 25 projects that build on top of Consult; Consult being part of GNU ELPA, and therefore considered part of Emacs, could be a major factor here.

Regarding packages depending on Consult - in the past, before I applied breaking changes, I've inspected these packages and ensured that there is an upgrade path without breakage. This approach is obviously not scaleable. However this is besides the discussion; tests will not be helpful if we introduce breaking changes here.