Testing - Containerlab and Mocking / Stubbing

[tardoe]

Opening this issue to begin discussion on how we should handle testing with v2. Currently the unit tests return mocked data and integration tests require containerlab to be spun up.

I propose the following:

• A GitHub action that conditionally spins up a 2-3 node Containerlab network when an srlinux_versions.txt file is update. This file contains supported SRLinux versions and (sequentially) spins up a network, captures the JSON RPC output for each set of paths used in the driver and commits the output to the test/<version>/rpc_<paths_name>.json or the like. These fixtures can then be used "offline" by committers and during regular GH Action runs to unit test against supported SRL versions.
• Using the above captured output, we can stub all _jsonrpc* methods under text fixtures and enable per-getter tests without needing an online network.
• Configuration testing can be performed against a mock JSONRPC endpoint (https://colin-b.github.io/pytest_httpx/) with derived input and output to test load / merge / commit / rollback use-cases.

@hellt thoughts / ideas / comments / discussion?

[jbemmel]

I've always considered the live integration testing with Containerlab an important differentiator for the plugin. After having Containerlab installed and downloading the Docker image, the tests can be performed "offline" and are entirely optional. CI/CD workflows on GitHub spin up Containerlab instances internally

Having a 'snapshot' of JSONRPC output isn't as comprehensive as being able to spin up any release you want, with any node configuration / topology, and then running regression testing against that. I'd expect more bugs to be found in the area of specific (combinations of) config options and/or race conditions.

In short, I'm not sure what problem you'd be solving?

[tardoe]

@jbemmel there would still be the option of running the tests locally, but having the option to run tests easily (e.g. for small fixes) without needing a full lab lowers the barrier of entry.

[hellt]

@tardoe I'd say on a high level I'd split the testing into the following parts:

0. Raw input extraction. This is what you proposed in the first bullet point. The goal is to get the raw json responses per each getter that would be used in unit testing of getters
1. local/remote unit testing of getters based on the previous step. This is where we a) take the raw json input extracted before, apply the getters logic, compare the result with a golden file.
2. unit testing of cfg workflows (merge replace, replace, etc). I think this is where we might always require a real system, granted it is easy to spin up one locally or in actions.
3. gh actions would use all of the above by also introducing a matrix of py versions to the mix. If we extract the raw json responses per each getter we support, there is no real need to perform the proper e2e test for all getters, although it is not a problem to add those, to ensure the real life scenario is working.
   The cfg opertaions though should always run in actions by spinning up a real device, I think.

[jbemmel]

@tardoe there is no barrier thanks to Containerlab and SR Linux. Unlike most other platforms that developers may be familiar with, SR Linux makes an easy-to-obtain free container image available. This makes it trivial to construct CI/CD pipelines such as the one illustrated in this repo - without the user having to lift a finger!

Consequently, there is no need to jump through hoops like you are suggesting. This is the best other platforms can do - but ours (yours) is better than that