Need help debugging a failed gruntworks-pipeline

[umm0n]

Hello! Some context: We're using the Gruntworks ref arq and the Gruntworks pipeline, in GHA.

We recently tried to delete our openvpn module and all its dependencies. To do so, we created a big PR that:

• Deletes the entire openvpn directory in .../networking/openvpn
• Deletes all dependency blocks that refer to this module across our entire repo
• Deletes all calls to the dependency block outputs

Upon creating the PR, the plan ran okay. All the Openvpn resources were flagged for deletion.

However, when running the deploy itself, it fails with the following:

The following modules were deleted:
<EDITED>/networking/openvpn-server
Running apply  on each deleted module.
Destroying <EDITED>/networking/openvpn-server
Assuming role arn:aws:iam::************:role/allow-auto-deploy-from-other-accounts. These creds will expire after 3600 seconds.
Success!
Running infrastructure-deployer with args: --aws-region ****** -- terraform-applier infrastructure-deploy-script --ref master --binary terragrunt --command apply --command-args -destroy --deploy-path <EDITED>/networking/openvpn-server --repo https://github.com/*******--force-https true --apply-ref-for-destroy HEAD^
level=info msg="Successfully invoked Lambda function ecs-deploy-runner-invoker to trigger deployment."
time="2022-12-12T19:15:39Z" level=info msg="Waiting for ECS task arn:aws:ecs:*****:***********:task/ecs-deploy-runner/************** to start"
level=info msg="Deployment ECS Task arn:aws:ecs:*****:***********:task/ecs-deploy-runner/**************  has started. Streaming logs."
ERROR: Deployment failed

And then:

Error: Expected path /tmp/tmp753wship/<EDITED>/networking/openvpn-server to exist
exit status 1
ERROR: exit status 1
ERROR: Deployment failed

There's nothing else in the logs and we haven't got a clue what this error means. This is hardly the first time we delete a module via the pipeline. Could somebody lend us a hand?

---

Tracked in ticket #109721

[denis256]

Hi,
error with missing path may indicate that value from --apply-ref-for-destroy is pointing to commit which doesn't have networking/openvpn-server directory, so the pipeline fails to run destroy

Can be attempted to use in --apply-ref-for-destroy fixed git hash or tag pointing to the exact commit that precedes the destroy commit

[umm0n]

@denis256 upon further review, we noticed that the plan part of the pipeline runs against the commit that holds the deleted modules:
Run ./_ci/scripts/deploy-infra.sh "HEAD^" "be8b1d2a02cfd5e87c8b0f3458b83a1776548d01" plan

However, after the plan is finished and the pipeline moves on to deploy, it appears to run against master, which ofc no longer has the modules being deleted:

Run ./_ci/scripts/deploy-infra.sh "HEAD^" "master" apply

This understandably causes the deploy part to fail. My question is why? Why would the pipeline run plan against the commit and then run deploy against master? How can we delete modules via the gruntworks-pipeline if this is how it behaves? I'm almost certain we've deleted modules before and haven't faced this issue.

[umm0n]

Nevermind, we seem to have found the problem. We merged another PR after this one, which finished before the first one. When the first one ran deploy, HEAD^ fetched the last commit that no longer had the modules, which is why it failed.