Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow artifact path to be located outside the sync root #2128

Merged
merged 14 commits into from
Jan 14, 2025

Conversation

pietern
Copy link
Contributor

@pietern pietern commented Jan 13, 2025

Changes

We perform a check during path translation that the path being referenced is contained in the bundle's sync root. If it isn't, it's not a valid remote reference. However, this doesn't apply to paths that are always local, such as the artifact path. An artifact's build command is executed in its path. Files created by the artifact build (e.g. wheels or JARs) don't need to be in the sync root because they have a dedicated and different upload path into ${workspace.artifact_path}.

Therefore, this check that a path is contained in the bundle's sync root doesn't apply to artifact paths. This change modifies the structure of path translation to allow opting out of this check.

Fixes #1927.

Tests

  • Existing and new tests pass.
  • Manually confirmed that building and using a wheel built outside the sync root path works as expected.
  • No acceptance tests because we don't run build as part of validate.

@pietern pietern temporarily deployed to test-trigger-is January 13, 2025 14:43 — with GitHub Actions Inactive
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dummy file to keep the directory around.

This is needed now that path translation verifies that artifact paths exist.

assert.Equal(t, "artifact_a", relA)
relB, err := filepath.Rel(b.SyncRootPath, b.Config.Artifacts["test_b"].Path)
require.NoError(t, err)
assert.Equal(t, filepath.Join("subfolder", "artifact_b"), relB)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Artifact paths are now absolute.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assert.Equal(t, filepath.Join("subfolder", "artifact_b"), relB)

Artifact paths are now absolute.

What are you referring to? the variables in the test are relative.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplified in latest.

}
}

return dyn.InvalidValue, err
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Denormalize the previous rewriteRelativeTo function into the body for jobs and pipelines.

It is not relevant outside these two resources.

}
}

return dyn.InvalidValue, err
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Denormalize the previous rewriteRelativeTo function into the body for jobs and pipelines.

It is not relevant outside these two resources.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this and below comments be incorporated into source? Seems like valuable context.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's included (vs the previous), here:

  // If we failed to rewrite the path, try to rewrite it relative to the fallback directory.
  // We only do this for jobs and pipelines because of the comment in [gatherFallbackPaths].

From gatherFallbackPaths the rest of the context is clear.

I would like to deprecate it but we don't know how much usage is out there. Another good thing to track when we can capture some telemetry.

} else {
t.remoteRoot = t.b.Config.Workspace.FilePath
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assignment now happens once instead of for every path.

@pietern pietern marked this pull request as ready for review January 13, 2025 14:51
@denik
Copy link
Contributor

denik commented Jan 13, 2025

No acceptance tests because we don't run build as part of validate.

You can also do deploy in acceptance tests. You would just need to implement a few more dummy handlers in server_test.go.

Another option is to extract 'build' into separate command.

@pietern pietern temporarily deployed to test-trigger-is January 13, 2025 16:37 — with GitHub Actions Inactive
@pietern pietern temporarily deployed to test-trigger-is January 13, 2025 19:07 — with GitHub Actions Inactive
@pietern
Copy link
Contributor Author

pietern commented Jan 14, 2025

Manually confirmed with pending acceptance tests that this does not change behavior or output.

@pietern pietern added this pull request to the merge queue Jan 14, 2025
Merged via the queue into main with commit 5d9bc3b Jan 14, 2025
9 checks passed
@pietern pietern deleted the no-sync-root-check-for-artifact-paths branch January 14, 2025 08:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Relative path is not working within artifacts
3 participants