doc: add Mir's CI documentation (#3434)

doc/sphinx/reference/continuous-integration.md

@@ -0,0 +1,160 @@
+# Mir Continuous Integration
+
+This document outlines the journey of a contribution to Mir through its continuous integration
+pipeline.
+
+## Overview
+
+There are a number of components to this story, and a diagram might make for a good overview:
+
+```{mermaid} continuous-integration.mmd
+```
+
+These are run at different stages in the pipeline, balancing the time it takes to run and the
+breadth of testing. We'll discuss those in more detail below.
+
+## Mir builds
+
+When a pull request is opened, updated or merged into `main`, we validate that the contribution
+is correct by building the code and running our test suite. To facilitate this, we use
+[Spread](https://github.com/snapcore/spread) (or rather, our
+[lightly patched version](https://snapcraft.io/spread-mir-ci)) to build Mir across a number of
+environments. [`spread.yaml`](https://github.com/canonical/mir/blob/main/spread.yaml) holds
+environment definitions, while the actual build tasks are maintained under
+[`spread/build`](https://github.com/canonical/mir/tree/main/spread/build).
+
+Our focus of development is the most recent Ubuntu LTS, and we maintain builds for any more recent,
+supported Ubuntu releases. We also build for stable releases of other Linux distributions that
+we have community interest on.
+
+We also run builds using alternative toolchains (Clang) and development versions of Ubuntu and
+those other distributions.
+
+We also maintain Mir branches for all previous Ubuntu LTS releases under support. These will only
+receive security updates and the occasional bug fix. These will always be
+[`release/` branches on GitHub](https://github.com/canonical/mir/branches/all?query=release%2F),
+and releases published on [the GitHub release pages](https://github.com/canonical/mir/releases).
+
+Opening a pull request, or merging into the `main` or `release/` branches, triggers a
+[build on GitHub](https://github.com/canonical/mir/actions/workflows/spread.yml).
+
+As part of the build, the following sets of tests are run:
+
+### Unit tests
+
+Defined in [`tests/unit-tests`](https://github.com/canonical/mir/tree/main/tests/unit-tests),
+these assert the functionality of individual components in isolation, as is usual practice.
+
+### Acceptance tests
+
+[`tests/acceptance-tests`](https://github.com/canonical/mir/tree/main/tests/acceptance-tests)
+run more high-level tests, ensuring we maintain the contracts with external components as well as
+our own loadable modules' interfaces.
+
+One major part of this is [WLCS](https://github.com/canonical/wlcs) - asserting the behaviour
+of Mir's Wayland frontend is to specification of the protocol.
+
+### Integration tests
+
+In [`tests/integration-tests`](https://github.com/canonical/mir/tree/main/tests/integration-tests)
+lie other high level tests, ensuring we correctly integrate with some system components.
+
+### Performance and smoke tests
+
+Lastly, [`tests/performance-tests`](https://github.com/canonical/mir/tree/main/tests/performance-tests)
+holds a handful of performance-related tests, collecting some metrics about how the system
+performs end-to-end - and verifying that it works in the first place.
+
+We also run these on a number of hardware platforms in our testing lab for every build of `main`.
+
+## Sanitizer runs
+
+In addition to the above, for every merge to `main` we build and run the tests using the following
+sanitizers:
+
+- **Undefined Behaviour**
+  Building with UndefinedBehaviourSanitizer enabled reports no undefined behaviour, it would be a
+  CI failure otherwise.
+- **Address Sanitizer**
+  As we have some fixing to do here, we run those builds - but don't enforce the problems reported.
+- **Thread Sanitizer**
+  Unfortunately due to
+  [incompatibility between GLib and TSan](https://github.com/google/sanitizers/issues/490), we
+  don't currently run the thread sanitizer, as that produces too many false positives.
+
+## ABI checks
+
+For each pull request we
+[check that the exported symbols are as expected](https://github.com/canonical/mir/actions/workflows/symbols-check.yml)
+- and fail if the ABI changed in any way. It doesn't necessarily mean an ABI break - but new symbols
+need to be tracked.
+
+## Coverage measurement
+
+We [track test coverage](https://github.com/canonical/mir/actions/workflows/coverage.yml) for each pull
+request and `main` builds, and the results are visible on
+[Codecov.io](https://app.codecov.io/gh/canonical/mir).
+
+## `.deb` package builds
+
+Merges to `main`, `release/` branches as well as annotated tags are followed by `.deb` package
+builds in mir-team's Launchpad's Personal Package Archives (PPAs):
+- [`~mir-team/dev`](https://launchpad.net/~mir-team/+archive/ubuntu/dev) for the latest development
+  builds
+- [`~mir-team/rc`](https://launchpad.net/~mir-team/+archive/ubuntu/rc) for release candidate and
+  release builds
+
+At release, those get copied to the
+[`~mir-team/release`](https://launchpad.net/~mir-team/+archive/ubuntu/release), general availability
+PPA.
+
+## Downstream snap builds
+
+We [build a subset](https://github.com/canonical/mir/actions/workflows/snap.yml) of the downstream
+snaps on pull requests - these are then available in `edge/mir-pr<number>`
+[Snap channels](https://snapcraft.io/docs/channels) for testing, e.g.:
+
+```shell
+snap install miriway --channel edge/mir-pr1234
+```
+
+When `.deb` packages build in the PPAs above, all the affected downstream snaps get rebuilt through
+the [`~mir-team` snap recipes on Launchpad](https://launchpad.net/~mir-team/+snaps) and made
+available in the `edge` or `beta` channels, as appropriate.
+
+## End-to-end testing
+
+When the snaps get published to the store, we have Jenkins dispatch test runs on a selection of
+hardware, ranging from Raspberry Pis through single-GPU systems all the way up to high performance
+multi-GPU (usually hybrid) ones. We maintain a matrix of test coverage
+[in this spreadsheet](https://docs.google.com/spreadsheets/d/1kUbTSt4zWVpTtgZNJvvxCdugsRUv6C5PK9Xw5dxppCc/edit#gid=893560997).
+
+These tests ultimately verify the full story end-to-end, installing the snaps on a range of
+Ubuntu versions (Core and classic alike) and verify behaviours and visuals through a number of
+scenarios.
+
+It's a somewhat complex web of things happening, so again a diagram might help
+
+```{mermaid} end-to-end-testing.mmd
+```
+
+[checkbox-mir](https://github.com/canonical/checkbox-mir/) is the test orchestrator, collecting
+the different tests (smoke, performance and functional) and packaging them in a way that can be
+consumed by our lab setup. It's using our own [mir-ci](https://github.com/canonical/mir-ci) tests
+along with the [mir-test-tools](https://github.com/canonical/mir-test-tools) snap.
+
+You can find more information about Checkbox itself
+[in its documentation](https://github.com/canonical/checkbox) - it's a system used by our
+certification team, running thousands of tests on hundreds of systems every day.
+
+The Jenkins job definitions go into a private repository, as they contain credentials.
+[jenkins-job-builder](https://pypi.org/project/jenkins-job-builder/) is used to maintain the jobs,
+and at runtime, [testflinger](https://github.com/canonical/testflinger) dispatches them to the
+device under test, while [test-observer](https://github.com/canonical/test_observer) collects the
+results.
+
+## Summary
+
+As shown above, Mir is being tested quite extensively on a large number of environments. We're
+extending the coverage every day, as well. There's a lot of moving pieces involved, but it does
+help us greatly in ensuring the quality level we hold ourselves to.

doc/sphinx/reference/continuous-integration.mmd

@@ -0,0 +1,16 @@
+flowchart TD
+    PR{{Pull Request}} --> Builds(<a href="#mir-builds">Builds and tests</a>
+                                  <a href="#unit-tests">unit</a>
+                                  <a href="#acceptance-tests">acceptance</a>
+                                  <a href="#integration-tests">integration</a>
+                                  <a href="#performance-and-smoke-tests">performance</a>)
+    PR --> Snaps(<a href="#downstream-snap-builds">Downstream snaps</a>)
+    PR --> ABI(<a href="#abi-checks">ABI checks</a>)
+    PR --> Coverage(<a href="#coverage-measurement">Coverage</a>)
+    Merge{{Merge}} --> Builds
+    Merge --> Coverage
+    Merge --> Sanitizers(<a href="#sanitizer-runs">Sanitizers</a>)
+    Merge --> PPA(<a href="#deb-package-builds"><tt>.deb</tt> packages</a>)
+    PPA --> Snaps
+    Snaps --> EndToEnd(<a href="#end-to-end-testing">End-to-end testing</a>)
+

doc/sphinx/reference/end-to-end-testing.mmd

@@ -0,0 +1,24 @@
+flowchart TD
+    Store(Store)-->|new revision| Jenkins
+    Jenkins(Jenkins)-->|submits job| Testflinger
+
+    Testflinger(<a href="https://github.com/canonical/testflinger">Testflinger</a>)
+    Testflinger-->|installs and executes| checkbox-mir
+    Testflinger-->|installs| mtt
+    Testflinger-->|installs| snap
+
+    subgraph Device Under Test
+    checkbox-mir[[<a href="https://snapcraft.io/checkbox-mir/">checkbox-mir</a>]]
+
+    checkbox-mir-->|runs| mtt[[<a href="https://snapcraft.io/mir-test-tools/">mir-test-tools</a>
+    smoke
+    performance
+    ]]
+    checkbox-mir-->|runs| mir-ci[<a href="https://github.com/canonical/mir-ci">mir-ci</a>
+    functional
+    ]
+    mir-ci-->|installs| deps[[Dependencies]]
+    mir-ci-->|tests| snap[[Snap Under Test]]
+    end
+
+    checkbox-mir-->|reports to| TO(<a href="https://github.com/canonical/test_observer">test-observer</a>)

doc/sphinx/reference/index.md

@@ -3,6 +3,7 @@ These pages provide detailed reference to the Mir's programming and other interf
 
 - [API](/api/library_root): Mir API reference
 - [Introducing the MirAL API](introducing_the_miral_api): The primary external interface for compositor authors
+- [Continuous integration](continuous-integration): A detailed guide through Mir's testing infrastructure
 - [DSO Versioning guide](dso_versioning_guide): How is ABI managed in the Mir project
 - [Kernel requirements](kernel_requirements): The kernel features required to run Mir-based compositors
 
@@ -12,5 +13,6 @@ These pages provide detailed reference to the Mir's programming and other interf
 /api/library_root
 introducing_the_miral_api
 dso_versioning_guide
+continuous-integration
 kernel_requirements
 ```