Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor terraform code that automates creation of zip for source files #15

Merged
merged 1 commit into from
Jul 15, 2024
Merged

Refactor terraform code that automates creation of zip for source files #15

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 43 additions & 18 deletions modules/data_pipeline/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,29 +1,54 @@
data "google_project" "project" {
}

locals {
source_files = concat(
# Add paths to files from /cloud_functions's root
[
"${path.module}/../../climateiq-cnn/usl_pipeline/cloud_functions/main.py",
"${path.module}/../../climateiq-cnn/usl_pipeline/cloud_functions/requirements.txt"
],
# Add paths to all .py files from /usl_lib subdirectory
tolist(fileset("${path.module}/../../climateiq-cnn/usl_pipeline/usl_lib/usl_lib/", "**/*.py")),
# Add paths to all .whl files from /cloud_functions's /wheels subdirectory
tolist(fileset("${path.module}/../../climateiq-cnn/usl_pipeline/cloud_functions/wheels/", "*.whl"))
)

usl_lib_dir = "${path.module}/../../climateiq-cnn/usl_pipeline/usl_lib/usl_lib/"
wheels_dir = "${path.module}/../../climateiq-cnn/usl_pipeline/cloud_functions/wheels/"
}

# Read and process the content from source_files
data "template_file" "t_file" {
count = "${length(local.source_files)}"
template = element(local.source_files, count.index)
}

# Copy files to a temporary directory to prep for zip
resource "local_file" "to_temp_dir" {
count = "${length(local.source_files)}"

# Filenames are constructed to preserve source subdirectory structure within /temp
filename = "${path.module}/temp/${
contains(fileset(local.usl_lib_dir, "**/"), element(local.source_files, count.index)) ?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might consider making these three separate 'to_temp_dir' resources rather than one with a contains switch if you think that'd be more readable

"usl_lib/${element(local.source_files, count.index)}" :
contains(fileset(local.wheels_dir, "*"), element(local.source_files, count.index)) ?
"wheels/${element(local.source_files, count.index)}" :
basename(element(local.source_files, count.index))
}"

content = "${element(data.template_file.t_file.*.rendered, count.index)}"
}

# Place the source code for the cloud function into a GCS bucket.
data "archive_file" "source" {
type = "zip"
output_path = "${path.module}/files/cloud_function_source.zip"
source_dir = "${path.module}/temp"

# Add main.py to the root of the zip file.
source {
content = file("{path.module}/../../climateiq-cnn/usl_pipeline/cloud_functions/main.py")
filename = "main.py"
}
# Add requirements.txt to the root of the zip file.
source {
content = file("{path.module}/../../climateiq-cnn/usl_pipeline/cloud_functions/requirements.txt")
filename = "requirements.txt"
}
# Add all the contents of usl_lib to a usl_lib directory inside the zip file.
dynamic "source" {
for_each = fileset("{path.module}/../../climateiq-cnn/usl_pipeline/usl_lib/usl_lib/", "**/*.py")
content {
content = file("{path.module}/../../climateiq-cnn/usl_pipeline/usl_lib/usl_lib/${source.value}")
filename = "usl_lib/${source.value}"
}
}
depends_on = [
local_file.to_temp_dir,
]
}

resource "google_storage_bucket_object" "source" {
Expand Down