Automated tests

[tpansino]

Hi all,
Love this bit of software, enough that I've contributed a few regression fixes over the years.

Are there any automated tests for QA for this project? Or plans to add? I think most of the issues I've fixed could have been caught by a test suite, and this latest bug makes me nervous that there isn't enough QA (testing OR code review) going into new features. (That bug deleted all previous backups for all of my SQLite databases - fortunately I have offsite copies)

Edit: reading this back, I feel like I may come off as unappreciative and I apologize. I recognize this is free software, and everyone has limits on their time. I just wanted to take the opportunity to point out that the usability of the software is being affected by what I perceive to be a lack of automated QA at this time.

[tiredofit]

I caught the original and no harm intended.
To be honest I'd love to see some sort of automated tests in place.
I just don't know where to begin on something like this. I am open to pointers and a bit of a helping hand to get me going.

[tiredofit]

A couple years ago (2020) you had recommended BATS when you were affected by another bug related to Docker secrets. That repository has now gone to archived status and wondering if there is something better. I'm at the point right now where I have so many things on the go that I would really appreciate a base to work with and then I could take it from there. I've got 100+ images that I maintain of varying complexity and have to juggle life in between it, like getting food on the table - this definitely isn't at all paying my bills.

[tpansino]

Ok, great. Let me type up some guidance for you. I'll try to have it to you here in the next 24 hours. 👍

[tpansino]

A couple years ago (2020) you had recommended BATS when you were affected by another bug related to Docker secrets. That repository has now gone to archived status and wondering if there is something better.

It still exists, now it's called bats-core. I haven't used, but just looking at the docs I'd still recommend it. Much of what the tutorial covers includes the guidance I was originally going to give. 😅

Let me talk a bit about what I didn't see covered in the tutorial - going beyond the tutorial 😄 . Once you've learned the basics of the testing framework (or really any testing framework), your next challenge is a bit of brainwork to think about/document what needs tested and prioritize. I'd recommend starting with the "happy path" cases - for example make sure backup and restore work as intended for each type of DB. Then move on to the less happy paths, and then the unhappy paths, the error situations, which are where you are most likely to find hidden bugs.

I like writing test cases using the xUnit style. The BATS framework and tutorial introduce this by example (not by name), but you essentially want to follow the 3 A's:

• Arrange - start your test case by setting up everything you need to do the test. This can be done using setup() functions, or before you call the function or script you want to test, whatever makes sense. If you're going to need to read data from a file, or generate a test SQLite DB to backup, do that first in your test case code
• Act - actually call the function or script and exercise the code pathways
• Assert - the fun part, verify that everything as it should be. It looks like BATS comes with some assertion functions to verify text output is correct, files are created and with the right contents, etc.

Try to test only one "thing" per test case, if that makes sense. Multiple assertions are fine, but each case should be named something like test_backup_sqlite_with_normal_conditions or test_restore_influxdb_with_malformed_backup_data which clearly documents what individual "thing" you're trying to test with each case. And then the assertions inside document what the outcome should be.

The other thing it looks like BATS gives is structure for how to organize your test files, and a command for executing them all and seeing results. That should let you focus on getting into a groove of identifying a gap in testing, whipping up a test case with the 3 A's, and running to see it pass/fail.

Once you have some tests working locally, I'd encourage you to look at adding a required status check for merging PRs to ensure tests are passing via GitHub Actions. It's pretty easy to create a workflow file that triggers on push events, installs BATS, and runs the tests. This way you have the tests run each time a branch is pushed to, and this lets contributors (and you 😉 ) see if they've broken anything prior to merging.

and have to juggle life in between it, like getting food on the table - this definitely isn't at all paying my bills.

I completely understand 🙂 And if all this stuff is new to you, it can be a lot to take in. I'd offer more direct help, but I have similar challenges on my time, and FOSS projects of my own. The least I could do though was try to point you in the right direction 🙏 I hope it was helpful.

---

BONUS: if you get far enough along where you're trying to figure out what's left to test, look at adding measurements for Code Coverage. Coverage tells you which paths/lines of code in a file have not been executed, which can expose the gaps in the test suite. Here is one way to add coverage instrumentation to a GHA pipeline, using CodeCov, which I believe is free for FOSS projects.

Coverage can really help gamify adding tests and even make it an enjoyable experience. But don't feel like you have to try to get a perfect score. Often it's not possible or even helpful to test EVERY single line of code.