✅ Add unit testing framework

[sjasonsmith]

Description

This revives and builds upon an effort from @costas-basdekis several years ago, which provided a way to execute unit tests against various Marlin configurations.

This uses Platform IO's unit testing mechanisms to build and execute tests on a Linux machine.

How to use this feature

This is used through the top-level makefile. It likely only works on Linux or a Linux VM (including WSL).
Execute by running make tests-code-all-local or make tests-code-all-local-docker.

Overall summary of strategy:

1. Unique config.ini files describing test configurations are placed in sub-folders beneath test. PlatformIO recognizes this folder structure and will build one test executable for each folder present.

2. extra_script tooling adds extra commands which will apply the config.ini prior to building each test.

3. Tests are automatically discovered from test files inside the Marlin/tests directory. The Marlin configuration is available inside these tests, so typical pre-processor mechanisms can be used to determine whether a configuration is suitable to run a test, or to alter the behavior of a test.

4. Will run automatically with our existing CI tests. This was tested in my developer branch, but will have to be verified as soon as this is merged.

How were the initial tests selected?

Some of these tests were simple sanity checks originally added by Costas when he initially prototyped the feature. The types tests were added to specifically test a recently fixed Boolean casting problem which caused major issues across several Marlin subsystems.

Possible future improvements

• Currently it isn't easy to run a single configuration or apply a test filter. This is supported by platformio, but additional work will be needed to document it and make it work through makefiles.
• I'd like to consider switching to googletest, which is a much more robust testing framework if we will only be using this on Linux. I'm currently prevented from doing this due to name conflicts between TEST macros.

Requirements

Linux development environment. This can be executed inside Windows Subsystem for Linux (I've only tested with docker). It may be possible to get it working on other platforms, but this has not been attempted.

Benefits

Provides a mechanism to write tests which can be easily executed and automated by any developer in the world. These can verify correct behavior, as well as protect against future regression.

Configurations

N/A

Related Issues

N/A

[sjasonsmith]

That appears to have been relatively trivial to get running in a GitHub workflow. I have several temporary commits (which will be reverted) to allow them to run in my personal repo/branch for testing purposes.

Output with all tests passing

Output with the previous types.h fix reverted, causing tests to fail

Error messages that produced:

 linux_native_test:test_default_config_with_paren_comments [FAILED] Took 34.76 seconds 

=================================== SUMMARY ===================================
Environment        Test                                     Status    Duration
-----------------  ---------------------------------------  --------  ------------
linux_native_test  test_default_config_with_paren_comments  FAILED    00:00:34.758

__________ linux_native_test:test_default_config_with_paren_comments __________
test/marlin_tests.cpp:19:XYval_non_const_as_bools:FAIL: Expected FALSE Was TRUE

test/marlin_tests.cpp:37:XYZval_non_const_as_bools:FAIL: Expected FALSE Was TRUE

test/marlin_tests.cpp:55:XYZEval_non_const_as_bools:FAIL: Expected FALSE Was TRUE

[sjasonsmith]

While I tested that the workflow worked on my own repo/branch, I didn't test this github.ref check. I'm attempting to prevent this from running on the 2.1.x branch, since it won't work yet.

If we don't want to gamble whether this will work once merged, we could split this into a new YML file with different branch requirements. I thought it made sense to add it to this file because then it can re-use all the trigger configuration regarding file paths.

[sjasonsmith]

That github.ref syntax came from GitHub Copilot.

[thinkyhead]

We could also move this to its own action, if that helps to make things cleaner.

[thinkyhead]

Generally speaking it looks good. I just have one cleanup commit to push. Let's try replacing all the test_*.cpp files with INI files (based on config.ini). I've been planning to do the same thing for our CI tests, which should make them easier to maintain.

[thinkyhead]

We could also move this to its own action, if that helps to make things cleaner.

[sjasonsmith]

@thinkyhead I believe it will be automatic since I retained his name on commits in the history, but I'd like to be sure that @costas-basdekis is attributed as one of the authors when it goes in.

The requested changes have been completed.

[thinkyhead]

This seems to be in good shape as a starting point for a more complete set of tests. After this is merged we can try asking GitHub Copilot to generate more complete tests for our core data type classes and other things.

[sjasonsmith]

This seems to be in good shape as a starting point for a more complete set of tests. After this is merged we can try asking GitHub Copilot to generate more complete tests for our core data type classes and other things.

Core data types and verifying complex macro behavior are good places to start.

I’d like to see us then expand to some of the more complex logic. I believe I have tests in some experiments branches for bilinear subdivision and classic jerk calculations that I may be able to revive.

[sjasonsmith]

Based on the last comment from @thinkyhead, I m going to go ahead and merge this so that we can start adding additional tests.

[costas-basdekis]

I know this is a very late response, but I only recently saw the notification, and wanted to thank everyone involved in this for their patience and guidance, especially Jason.

I don't do a lot of 3D printing lately, but I could help with testing if that's something you need.

Again, well done on making progress!