Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

allow s3 commands to work from non commercial (ie us-gov) environments #1718

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

allow s3 commands to work from non commercial (ie us-gov) environments #1718

wants to merge 6 commits into from

Commits on Apr 1, 2024

  1. allow s3 commands to work from non commercial environments 

    when running under something like us-gov-east-1, the latest-binaries.sh
script fails b/c it tries to access the us-west-2 bucket from the wrong
endpoints.

this can be avoided by setting AWS_ENDPOINT_URL_S3 to point to
us-west-2, but you still end up trying to use the gov-cloud creds in the
request which would fail with:

$ ./hack/latest-binaries.sh 1.29

An error occurred (InvalidToken) when calling the ListObjectsV2 operation:
The provided token is malformed or otherwise invalid.

so, specify to perform an unauthenticated s3 api request b/c the
govcloud creds wouldn't work against the commercial cloud endpoints.

in other places in the install-worker.sh script, there are 'aws s3'
commands that would fail if running under something like the
us-gov-east-1 environment.

similar to the changes to the latest-binaries.sh script, update the
'aws' cli calls to ensure the requests are unsinged (to avoid trying
to use us-gov creds against a non-gov endpoint).

and plumb through using the user-specified AWS_ENDPOINT_URL_S3 env var
into the install-worker.sh script so that the alternative endpoints can
be used instead of the us-govcloud ones when running in a govcloud
environment.
    @joelddiaz
    joelddiaz committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    204d272 View commit details
    Browse the repository at this point in the history

  2. update al2023 to also use unsigned s3 requests 

    and re-order the al2 variable for aws_endpoint_url_s3 to be sorted
alphabetically with the rest of the variables.
    @joelddiaz
    joelddiaz committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    af1b71c View commit details
    Browse the repository at this point in the history

  3. add doc entries for aws_endpoint_url_s3

    @joelddiaz
    joelddiaz committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    89c9ba4 View commit details
    Browse the repository at this point in the history

  4. add explicit region for s3 commands

    @joelddiaz
    joelddiaz committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    92aacc9 View commit details
    Browse the repository at this point in the history

  5. remove aws_endpoint_url_s3 

    it isn't necessary now that we are providing the region for the s3
bucket
    @joelddiaz
    joelddiaz committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    a733644 View commit details
    Browse the repository at this point in the history

  6. add comment about wishing to use $binary_bucket_region 

    instead of hardcoding us-west-2
    @joelddiaz
    joelddiaz committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    9466e72 View commit details
    Browse the repository at this point in the history