fix: change k8s job container creation logic to retain existing value…

[cmanzi-bestow]

…s from job manifest

Fixes #9409

Fixes: #9409
Related: Relevant tracking issues, for context
Merge before/after: Dependent or prerequisite PRs

Description

This changes the logic that applies the provided container properties for executing a custom action (when using a custom job manifest) so that only the provided values (if not empty) are overwritten, and the container is not replaced entirely. I have done an initial test and this does now allow the preservation of volumeMounts and other properties that are currently erased.

Using the overrides config does not work here, as the overrides are applied prior to the logic that replaces the container.

This is a proposed solution for the referenced issue. Having looked under the hood while trying to debug this, I'm not exactly sure what the intended behavior was when providing a custom manifest, so I am open to other possible solutions - perhaps if a custom manifest is provided, the containers block should be unnecessary, since theoretically this should already be defined in the manifest?

This is currently requiring us to do a pretty extensive workaround to get this working with Cloud Deploy, as we need to run pre-deploy migrations for most of our services, and we have existing manifests that we need to use (the cut-down Container interface isn't sufficient for our purposes).

User facing changes (remove if N/A)

This would change the current behavior (which appears to be a bug, at least according to what one would expect) when providing a custom job manifest for a Kubernetes custom action. For example:

customActions:
- name: db-migration
  containers:
  - name: db-migrate
    image: db-migration-image:latest
    command:
    - /bin/sh
    - -c
    - run-db-migration
  executionMode:
    kubernetesCluster:
      jobManifestPath: migration-job.yaml

Follow-up Work (remove if N/A)

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[cmanzi-bestow]

I am working on some tests right now, I just wanted some initial feedback on the solution I'm proposing.

It would help if I could better understand what the intended behavior is when using a custom K8s job manifest, as how it's set up right now really doesn't make sense to me.

[mattsanta]

Thanks for taking the time to look into this.

So turns out that the verify implementation, which also supports providing a Job manifest properly handles this by only replacing the container fields that can be configured in the Skaffold Config (i.e. image, command, args, name, and env (appends to this)).

To fix the issue for custom actions we should mimic the behavior that verify currently implements. Could you update the PR with these changes and add some tests?