archive_file produces different results on different OSs

[maxrothman]

Affected Resource(s)

• datasource archive_file
• resource aws_lambda_function

Terraform Configuration Files

data "archive_file" "zip" {
    type        = "zip"
    source_file = "${var.source_code_path}"
    output_path = "zip/${basename(var.source_code_path)}.zip"
}

resource "aws_lambda_function" "fn" {
    filename = "${data.archive_file.zip.output_path}"
    source_code_hash = "${data.archive_file.zip.output_base64sha256}"
    function_name = "${var.function_name}"
    role = "${aws_iam_role.role.arn}"
    handler = "${var.handler_name == "" ? "${replace("${basename(var.source_code_path)}",".py","")}.lambda_handler" : var.handler_name}"
    runtime = "python3.6"
    timeout = "${var.timeout}"

    environment {
        variables = "${var.env_vars}"
    }
}

After applying on Windows, planning on Mac indicates that source_code_hash has changed, even though the content of the code has not changed. This is because archive_file produces equivalent but not identical results on Mac and Windows.

Expected Behavior

The lambda updates IFF the source code changes

Actual Behavior

The lambda updates when the source code changes OR if you run terraform on a different OS

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply on Windows
2. terraform plan on Mac
3. observe that it thinks it needs to update the lambda

[takayamaki]

I had encountered this issue too.
We used Lubuntu 16.04 and macOS 10.13.6.

[bboe]

I noticed that the produced zip file differs depending on the permissions of the source file. Why would the source file permissions be different? One common case is having a different umask between systems. For example, my laptop's default umask is 0022 thus when I clone a repository, the permissions of non-executable files are -rw-r--r--. However, the Amazon Linux 2 AMI has a default umask of 0002 thus newly created (or cloned) files have permissions of -rw-rw-r--.

Prior to realizing that archive_file worked for directories, I wrote a little tool to produce deterministic zip files. To handle permissions it either sets all files in the zip to be -r--r--r-- or -r-xr-xr-x depending on if the file should be executable or not. I could see a use case for also handling the write permission, but I have yet to need that.

The reason for these permissions is, as according to the lambda documentation all files should have global read permissions. While it's not explicitly stated, I interpret that to mean any executables thus need to have global execute permissions. Hence why I think zip files created by this tool should apply consistent permissions to all users (i.e., user, group, other sections).

Edit: I'm going to see I can quickly make a PR for this. I've never worked with go, but it seems straightforward enough.

[quixand]

I came across this issue where my centos 7 VM has different permissions for mounted volumes and was removing the execute bit. Annoyingly, trying to change the permissions doesn't work (see https://superuser.com/questions/1083393/virtualbox-guest-shared-folder-ignoring-umask)

some kind of parameter on the archive_file data source block to set the execute bit would make sense as its generating the zip, but wouldn't work for my scenario.

[quixand]

perhaps a better solution would be to provide a source hash i.e.
source_code_hash = "${data.archive_file.zip.input_base64sha256}"

After all, the source_code_hash element is used to trigger an update on change so inspecting the source rather than the exported binary zip should fulfill the same requirement.

[quixand]

I think you misunderstand. AWS does indeed confirm the hash of the uploaded binary(https://docs.aws.amazon.com/lambda/latest/dg/API_CreateFunction.html) but I expect this is intended to confirm that the binary has uploaded correctly. However, according to the terraform docs the source_code_hash element in aws_lambda_function is "Used to trigger updates". so it would be good to have a trigger that can identify that the code has changed separate from the hash result of the binary.

Of course this doesn't answer the question, why is the binary output different across OS's. And if terraform is verifying the hash of the binary in the background anyway this still needs to be resolved.

[maxrothman]

That only works when you're only archiving a single file. What if you're archiving a directory?

[bboe]

I submitted this PR some time ago but haven't heard anything from maintainers of the project. 🤷‍♂

#41

[Sodki]

This is not an easy or straight forward task (even if it looks like it) and I seriously doubt that Terraform is going to fix it. A couple of years ago a new project called TorrentZip was created by the MAME community to tackle this exact problem (reproducible zip files across multiple operating systems). More information here: http://rescene.wikidot.com/torrentzip

It seems that someone implemented a Go version of it: https://github.com/uwedeportivo/torrentzip

[virgofx]

@appilon @radeksimko Any chance we could get this merged into the module? This issue has been around for awhile and has the most upvotes/comments

[sapher]

Still having the issue today

[xarses]

I can confirm this behavior still exists as the patch from @saveriomiroddi wasn't merged (sadly). With out these changes the archive_file production isn't idempotent. At a minimum this should be reflected in the documentation so that appropriate steps can be taken to avoid the problem

[appilon]

Hello all, apologies for the long time without any word. Unfortunately solving this problem is inherently difficult, we recently setup multi-os testing in #71 to try and have a better chance at catching problems such as this, but this issue specifically isn't something that we can practically catch in CI.

We are evaluating the solutions proposed in #41 and #47, but in this circumstance trying to code against operating system or library defaults just so a common pattern for a different provider is resolved, might not be something we want to set precedent for in an individual provider (our apologies if that is a frustrating stance to take).

With that said, we are still looking at the two proposed PRs, and will be attempting to either come up with a solution, or provide a final word/guidance.

UPDATE: #53 is likely the enhancement we want to make to try and remedy the problem.

[virgofx]

@appilon Thanks so much for the update. Looking forward to a hopeful resolution soon.

[karolkania]

A workaround we applied was to have:

• exact same umask on both platforms
• source_dir for data "archive_file" -- see https://www.terraform.io/docs/providers/archive/d/archive_file.html
• dockerized terraform, ie. docker run --rm -it -v ${HOME}/.aws:/root/.aws -v ${PWD}:/code -w /code -e AWS_PROFILE=${AWS_PROFILE} hashicorp/terraform:0.12.24 $@

The zipinfo command will give you a hint for what is the possible difference between one archive and another

Note (in our case) the first column: -rw-rw-r-- vs -rw-r--r--

Linux - zipinfo output:

Archive:  tmp/archive.zip
Zip file size: 14604 bytes, number of entries: 15
-rw-rw-r--  2.0 unx      412 bl defN 49-Jan-01 00:00 index.js
-rw-rw-r--  2.0 unx     1707 bl defN 49-Jan-01 00:00 node_modules/...
[...]
15 files, 26615 bytes uncompressed, 12122 bytes compressed:  54.5%

MacOS - zipinfo output:

Archive:  tmp/archive.zip
Zip file size: 14604 bytes, number of entries: 15
-rw-r--r--  2.0 unx      412 bl defN 49-Jan-01 00:00 index.js
-rw-r--r--  2.0 unx     1707 bl defN 49-Jan-01 00:00 node_modules/...
[...]
15 files, 26615 bytes uncompressed, 12122 bytes compressed:  54.5%

TF code Example

data "archive_file" "lambda" {
  type        = "zip"
  source_dir = "${path.module}/lambda_source/"
  output_path = "${path.module}/tmp/archive.zip"
}

[nathanielks]

Nicely done, that's great @karolkania 👏👏👏

[timwsuqld]

While this is a 2 year old comment (hashicorp/terraform#18422 (comment)), it suggests that zip files should be created by a build process.
https://www.reddit.com/r/Terraform/comments/aupudn/building_deterministic_zips_to_minimize_lambda/ also handles it by creating zip files in the build process instead of using archive_file

It would be great if Hashicorp would give an official status on if archive_file is the "wrong" way to be doing this. There are a bunch of different ways to produce deterministic zip files in your build process, including strip-nondeterminism from Debian (https://salsa.debian.org/reproducible-builds/strip-nondeterminism) and deterministic_zip https://github.com/bboe/deterministic_zip

[virgofx]

Yeah, I'd love to see some traction on this issue. I don't think this is out of scope of Terraform though. As this thread has mentioned, there are tons of workarounds to get it working. Just need Hashicorp to actually put some resources on this and get it working.

[64kramsystem]

Hi there! I've deleted all of my comments are closed the PR; please don't (manually) CC me.

[kmoe]

There is a proposed workaround for this issue in #90, which adds an optional output_file_mode attribute to force consistency in file modes across OSs.

I would be very grateful if some of you experiencing this could test this fix by running @virgofx's provider locally.

@virgofx has kindly provided binaries from his branch for testing: #90 (comment)

You can find instructions for using a locally built provider here: https://www.terraform.io/docs/cli/config/config-file.html#development-overrides-for-provider-developers

[kmoe]

The workaround in #90 has been released in terraform-provider-archive v2.2.0. If output_file_mode does not solve your problem, please comment on this issue or open a new one.

[chrisbloe]

This didn't fix it for me:

data "archive_file" "lambda_zip" {
  type             = "zip"
  output_file_mode = "0777"
  source_dir       = "${path.module}/files/${var.name}"
  output_path      = "${path.module}/files/${var.name}.zip"
}

However, the zips I were having the problem with were zips created inside a module that's used multiple times... If anyone reading this is also having a problem in this scenario, check out my comment @ #31 (comment)

[davidalger]

Here's something that I've been doing for some time now, and is working reliably cross-platform (local runs on darwin, CI runs on linux):

data "archive_file" "this" {
  type        = "zip"
  output_path = "${path.root}/.terraform/gke-cluster-notification-${random_id.this.hex}.zip"

  dynamic "source" {
    for_each = toset([
      "files/function/function.go",
      "files/function/go.mod",
      "files/function/go.sum",
    ])

    content {
      content  = file("${path.module}/${source.value}")
      filename = basename(source.value)
    }
  }
}

I.e. instead of using source_dir I'm enumerating the files the resulting zip should contain, and since terraform is reading their contents in via the fil() function, there is no opportunity for umask, permissions, etc to cause a difference in the resulting zip on different platforms.

[cristianburca]

worked for me as well. Thanks @davidalger

[digitalkaoz]

@davidalger cam across this one. we are uploading our compiled go code into the zip, but the hashes are unstable (like the whole internet has this problem). sadly file(binary_file) wont work. how you deal with that?

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.