Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for protobuf serialization for plugin communication #13120

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Add support for protobuf serialization for plugin communication #13120

wants to merge 5 commits into from

Commits on Sep 13, 2024

  1. packer: pick protobuf/gob for serialisation (#13025) 

    As we're trying to move away from gob for serialising data over the
wire, this commit adds the capability for Packer to pick dynamically
between gob or protobuf for the serialisation format to communicate with
plugins.

As it stands, if all the plugins discovered are compatible with
protobuf, and we have not forced gob usage, protobuf will be the
serialisation format picked.

If any plugin is not compatible with protobuf, gob will be used for
communicating with all the plugins that will be used over the course of
a command.
    @lbajolet-hashicorp
    lbajolet-hashicorp committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    e084431 View commit details
    Browse the repository at this point in the history

  2. packer_test: add build customisation capabilities 

    When building a plugin, we may want some customisation capabilities
beyond changing the version/pre-release/metadata, and instead run
commands or change files on the filesystem.

To do so, we introduce functions under the BuildCustomisation type,
which have two responsabilities: changing the current state of the
plugin's directory, and cleaning up afterwards.
These customisations are passed as parameters to the BuildSimplePlugin
function, and are called one-by-one, deferring their cleanup after the
build process is finished.

A first implementation of such a customisation is added with this
commit, in order to change the version of a module that the plugin
depends on, which we'll use to change the version of the plugin SDK in
order to test how Packer behaves with different versions of the SDK for
a single plugin.
    @lbajolet-hashicorp
    lbajolet-hashicorp committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    8e0728f View commit details
    Browse the repository at this point in the history

  3. packer_test: introduce global compilation queue 

    Compiling plugins was originally intended to be an idempotent operation.
This however starts to change as we introduce build customisations,
which have the unfortunate side-effect of changing the state of the
plugin directory, leading to conflicts between concurrent compilation
jobs.

Therefore to mitigate this problem, this commit changes how compilation
jobs are processed, by introducing a global compilation queue, and
processing plugins' compilation one-by-one from this queue.

This however makes such requests asynchronous, so test suites that
require plugins to be compiled will now have to wait on their completion
before they can start their tests.

To this effect, we introduce one more convenience function that
processes those errors, and automatically fails the test should one
compilation job fail for any reason.
    @lbajolet-hashicorp
    lbajolet-hashicorp committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    3609a18 View commit details
    Browse the repository at this point in the history

  4. packer_test: add gob/pb test suite 

    With the draft to support both gob and protobuf as serialisation formats
for Packer, along with the SDK changes that propel them, we add a series
of tests that make sure the logic that picks which protocol is solid and
functional.

These tests rely on building several versions of the tester plugin, with
and without protobuf support, to then install them in the tests as
needed to test the logic of Packer using packer build with them, and
templates that require multiple plugins.
    @lbajolet-hashicorp
    lbajolet-hashicorp committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    1e7e57b View commit details
    Browse the repository at this point in the history

  5. packer_test: update go directive and tidy

    @lbajolet-hashicorp
    lbajolet-hashicorp committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    7bb4c19 View commit details
    Browse the repository at this point in the history