NUT CI farm

Why

For over a quarter of a century, the Network UPS Tools project has helped various systems and platforms monitor their UPSes and shut down safely.

In fact, we shyly consider ourselves such a part of critical infrastructure, that wherever NUT worked before, our new releases should still build and work to the best extent possible -- meaning not only chasing cutting-edge releases of already tested operating systems and gaining ground on new ones, but also supporting old boxes confined to whatever OS version they ran when built some 20 years ago (perhaps when they were last rebooted, no less), even if all but abandoned by their vendors. "They don't make 'em like this anymore" ;)

What

This is a big undertaking, primarily taking a toll on the sheer amount of build scenarios needed to cover a few dozen platforms (OSes/distributions with a range of their releases running on different CPU architectures), times the toolkits they have (different compilers, Make implementations, shells) and C/C++ language revisions supported there (including some tricky ones, with half-baked GCC 4.x series support of rapidly evolving language being a prime example), times some variation in third-party dependencies (versions of libusb, nss/openssl, etc.), with and without documentation generation, and further build analysis tools like cppcheck, valgrind and others...

Every PR iteration or master-branch build undergoes a couple hundred scenarios (many of which fan out into several builds for third-party dependencies and distcheck routines), and very regularly succeed. While there is a small portion of environmental issues that plague such a huge build/test matrix once in a while, most of the time failed builds are easy to map to proposed changes -- and build logs help locate and fix the issue quickly.

This also allowed NUT code base to finally ingest numerous contributions which lingered for years as PRs, branches and forks, with nobody including authors having any verifiable idea about the quality of proposed changes and what would they break. Now that builds can and must progress "from green to green", it is much easier to be confident about the syntax nuances and focus on reviewing the logical merit of the proposed features.

How

Over time, NUT did not keep all eggs in one basket, and took advantage of a number of approaches.

• The Fightwarn Effort starting with issue #823 (so many similar later PRs linked to it) and eventually tagged with "refactor/fightwarn" label took care over a couple of intensive years to clean up the NUT codebase (and later stale branches with queued contributions) so that its builds pass "green" with a reasonable selection of compiler analysis settings on a wide range of platforms under hand.
  • NUT recipes eventually included configure script settings like --enable-Werror and --enable-warnings=LEVEL, with default builds enabling quite a few warning categories which we survive (sometimes by analyzing code to fix issues, add sanity-checks and/or mark up specific false-positives with #pragma expressions), and the "harder" and somewhat esoteric selection still waiting for their heroes.
  • In particular, this work un-blocked adding libusb-1.0 support (and eventually a major NUT v2.8.0 release), which had considerable changes to many parts of the codebase, and when builds with any sensitivity to warnings were failing due to hundreds of them, it was impossible to tell if an attempted fix backfired.
  • Later, more tested areas got covered, such as spelling checks, shell script checks, and NUT Integration Tests (NIT) actually running parts of freshly built NUT against each other (and automating such sandbox creation for developer testing).
• Work on issue #823 also bootstrapped the use of Travis CI to run several build scenarios on various platforms they provided (i386/amd64, arm, s390x...) based on lessons learned in the 42ITy fork, and added a "CI" label (and/or token in issue/PR titles) for "non-functional" things related to recipes and testing.
  • This boosted CI ability, started some time earlier with a community member's BuildBot instance occasionally running some builds on different platforms.
  • While the 42ITy fork had a long and fruitful run with Travis, its days were numbered and by issue #869 a new CI farm was needed...
• The idea with a dedicated Jenkins instance won, primarily due to ease of adding random platforms (compared with cloud offerings that only serve popular ones -- Lin/Win/Mac -- most of the time).
  • This was completed with a jenkins-dynamatrix library created to generally manage a dynamic build matrix based on currently connected build agents declaring their OS, CPU architecture, available toolkits, supported versions etc. -- so anyone in the community can contribute (and ensure their favorite platform(s) are well supported), by adding a swarm agent from their system(s).
  • This initially ran with Fosshost, whose team was very helpful and machines were decent... until suddenly its days became numbered as of issue #1729. While things became falling apart soon afterwards, with management services and domain names dissipating, the actual NUT CI VMs slowly ran for a good couple of years afterwards until they disappeared too.
  • The current iteration of NUT CI Jenkins controller and several "always-available" build agents runs on DigitalOcean sponsored hosting for FOSS.
  • Since the NUT CI farm involves a number of operating systems not officially supported by DigitalOcean, it was a fun adventure getting them set up -- as detailed in issue #2192 live comment stream.
• CircleCI was later added for MacOS/Homebrew builds in the cloud
• Appveyor CI was later added for NUT for Windows builds in the cloud
• CodeQL runs a number of scenarios in GitHub Actions, doing builds with various dependencies per scenario and scanning the resulting code
• Several other options, as well as expansion of these, are still in the backlog

A large part of the repetitive success in CI and actual development iterations was introduction of ci_build.sh script (inspired by ZeroMQ and their ZProject generator) which ties together many high-level build scenarios and the environment variables or configure options involved there.

More reading

The docs/config-prereqs.txt document (also seen as a rendered page on NUT website) details tools and dependencies that were added on NUT CI build environments, which now cover many operating systems. This should provide a decent starting point for the build on yours (PRs to update the document are welcome!)

Configuration of specifically Linux containers with numerous distributions, aided by LXC, can be seen in docs/ci-farm-lxc-setup.txt, or several rendered pages on NUT website starting here. Much of that pertains to non-Linux platforms as well, if we want systems set up similarly for ease of farm administration.

If you got to this page looking for information on how NUT was built and tested, so you can troubleshoot or fix your installations, please peruse the links in the Wiki page footer below, notably the pages on "in-place builds", "NUT debug settings" and "nut-driver-enumerator (NDE)".