Support for AWS S3 Express One Zone directory buckets

[hveiga]

Hi,

I would like to know if AWS S3 Express One Zone is supported out of the box in nginx-s3-gateway. I have using the following configuration, but still had no luck:

S3_BUCKET_NAME={{MY-DIRECTORY-BUCKET-NAME}}
AWS_ACCESS_KEY_ID="REDACTED"
AWS_SECRET_ACCESS_KEY="REDACTED"
AWS_SESSION_TOKEN="REDACTED"
S3_SERVER=s3express-usw2-az1.us-west-2.amazonaws.com
S3_SERVER_PORT=443
S3_SERVER_PROTO=https
S3_REGION=us-west-2
S3_STYLE=virtual
DEBUG=true
AWS_SIGS_VERSION=4
ALLOW_DIRECTORY_LIST=true
PROVIDE_INDEX_PAGE=false
APPEND_SLASH_FOR_POSSIBLE_DIRECTORY=false

The main motivation to use nginx-s3-gateway in combination with a AWS S3 Express One Zone buckets is the caching capabilities to minimize the number of HEAD/GET requests.

If not supported at the moment, would it be possible to add support for it?

Any hints are appreciated,

Thanks!

[4141done]

Hello,

Thank you for bringing this up. I just took a look at some of the documentation for S3 Express One Zone. Based on my reading I think it's possible that we may not be compatible due to some mentioned changes to authentication.

Would you be able to provide some more details such as:

• The request you're making and the error response
• Logs from the s3 gateway if you have them

I would be interested in taking a look at what it would take to support this s3 feature.
I've created #225 to track this feature request.

References:
Differences between Express One Zone and S3
Create Session syntax

[hveiga]

Using those configuration settings I am able to start nginx:

➜ docker run -it --env-file ./setting-s3-express -p 80:80 ghcr.io/nginxinc/nginx-s3-gateway/nginx-oss-s3-gateway:latest
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/00-check-for-required-env.sh
S3 Session token specified - not using IMDS for credentials
S3 Backend Environment
Access Key ID: "REDACTED"
Origin: https://MY-DIRECTORY-BUCKET-NAME.s3express-usw2-az1.us-west-2.amazonaws.com:443
Region: us-west-2
Addressing Style: virtual
AWS Signatures Version: v4
DNS Resolvers:  192.168.65.7
Directory Listing Enabled: true
Directory Listing Path Prefix: ""
Provide Index Pages Enabled: false
Append slash for directory enabled: false
Stripping the following headers from responses: x-amz-;
CORS Enabled: 0
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Ignoring /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/cache.conf.template to /etc/nginx/conf.d/cache.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/s3_location.conf.template to /etc/nginx/conf.d/gateway/s3_location.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/cors.conf.template to /etc/nginx/conf.d/gateway/cors.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v2_js_vars.conf.template to /etc/nginx/conf.d/gateway/v2_js_vars.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/js_fetch_trusted_certificate.conf.template to /etc/nginx/conf.d/gateway/js_fetch_trusted_certificate.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v2_headers.conf.template to /etc/nginx/conf.d/gateway/v2_headers.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v4_headers.conf.template to /etc/nginx/conf.d/gateway/v4_headers.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/s3listing_location.conf.template to /etc/nginx/conf.d/gateway/s3listing_location.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v4_js_vars.conf.template to /etc/nginx/conf.d/gateway/v4_js_vars.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/s3_server.conf.template to /etc/nginx/conf.d/gateway/s3_server.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/upstreams.conf.template to /etc/nginx/conf.d/upstreams.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/22-enable_js_fetch_trusted_certificate.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2024/03/14 22:08:39 [emerg] 1#1: host not found in upstream "s3express-usw2-az1.us-west-2.amazonaws.com:443" in /etc/nginx/conf.d/upstreams.conf:10
nginx: [emerg] host not found in upstream "s3express-usw2-az1.us-west-2.amazonaws.com:443" in /etc/nginx/conf.d/upstreams.conf:10

After some investigation, the main reason is that s3express-usw2-az1.us-west-2.amazonaws.com:443 is basically not an endpoint that can be reached. On the contrary, if using regular S3, you would use something like s3-us-west-2.amazonaws.com which in fact can be reached.

I have tried doing some tricks, for example setting S3_BUCKET_NAME as empty and S3_SERVER with the full endpoint info:

S3_BUCKET_NAME=
S3_SERVER=MY-DIRECTORY-BUCKET-NAME.s3express-usw2-az1.us-west-2.amazonaws.com

In this case, nginx starts but I see an extra . in the Origin log:

➜  Downloads docker run -it --env-file ./setting-s3-express -p 80:80 ghcr.io/nginxinc/nginx-s3-gateway/nginx-oss-s3-gateway:latest
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/00-check-for-required-env.sh
S3 Session token specified - not using IMDS for credentials
S3 Backend Environment
Access Key ID: "REDACTED"
Origin: https://.MY-DIRECTORY-BUCKET-NAME.s3express-usw2-az1.us-west-2.amazonaws.com:443
Region: us-west-2
Addressing Style: virtual
AWS Signatures Version: v4
DNS Resolvers:  192.168.65.7
Directory Listing Enabled: true
Directory Listing Path Prefix: ""
Provide Index Pages Enabled: false
Append slash for directory enabled: false
Stripping the following headers from responses: x-amz-;
CORS Enabled: 0
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Ignoring /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/cache.conf.template to /etc/nginx/conf.d/cache.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/s3_location.conf.template to /etc/nginx/conf.d/gateway/s3_location.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/cors.conf.template to /etc/nginx/conf.d/gateway/cors.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v2_js_vars.conf.template to /etc/nginx/conf.d/gateway/v2_js_vars.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/js_fetch_trusted_certificate.conf.template to /etc/nginx/conf.d/gateway/js_fetch_trusted_certificate.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v2_headers.conf.template to /etc/nginx/conf.d/gateway/v2_headers.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v4_headers.conf.template to /etc/nginx/conf.d/gateway/v4_headers.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/s3listing_location.conf.template to /etc/nginx/conf.d/gateway/s3listing_location.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v4_js_vars.conf.template to /etc/nginx/conf.d/gateway/v4_js_vars.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/s3_server.conf.template to /etc/nginx/conf.d/gateway/s3_server.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/upstreams.conf.template to /etc/nginx/conf.d/upstreams.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/22-enable_js_fetch_trusted_certificate.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2024/03/14 22:14:04 [notice] 1#1: using the "epoll" event method
2024/03/14 22:14:04 [notice] 1#1: nginx/1.25.3
2024/03/14 22:14:04 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14)
2024/03/14 22:14:04 [notice] 1#1: OS: Linux 6.6.12-linuxkit
2024/03/14 22:14:04 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2024/03/14 22:14:04 [notice] 1#1: start worker processes
2024/03/14 22:14:04 [notice] 1#1: start worker process 74
2024/03/14 22:14:04 [notice] 1#1: start cache manager process 75
2024/03/14 22:14:04 [notice] 1#1: start cache loader process 76

If I try to do a get in an object I get 404 and these are the logs I can see from nginx. Notice host also has the leading .:

2024/03/14 22:14:23 [info] 74#74: *1 js: S3 Request URI: GET /my-file.json
2024/03/14 22:14:23 [info] 74#74: *1 js: S3 Request URI: GET /my-file.json
2024/03/14 22:14:23 [info] 74#74: *1 js: AWS v4 Auth Canonical Request: [GET
/my-file.json

host:.MY-DIRECTORY-BUCKET-NAME.s3express-usw2-az1.us-west-2.amazonaws.com
x-amz-content-sha256:REDACTED
x-amz-date:20240314T221423Z
x-amz-security-token:"REDACTED"

host;x-amz-content-sha256;x-amz-date;x-amz-security-token
REDACTED]
2024/03/14 22:14:23 [info] 74#74: *1 js: AWS v4 Auth Canonical Request Hash: [REDACTED]
2024/03/14 22:14:23 [info] 74#74: *1 js: AWS v4 Auth Signing String: [AWS4-HMAC-SHA256
20240314T221423Z
20240314/us-west-2/s3/aws4_request
REDACTED]
2024/03/14 22:14:23 [info] 74#74: *1 js: AWS v4 Signing Key Hash: [REDACTED]
2024/03/14 22:14:23 [info] 74#74: *1 js: AWS v4 Authorization Header: [REDACTED]
2024/03/14 22:14:23 [info] 74#74: *1 js: AWS v4 Auth header: [AWS4-HMAC-SHA256 Credential="REDACTED"/20240314/us-west-2/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token,Signature=REDACTED]
192.168.65.1 - - [14/Mar/2024:22:14:23 +0000] "GET /my-file.json HTTP/1.1" 404 146 "-" "Wget/1.21.4" "-"

Regarding auth, my understanding is that the session-based auth for S3 Express One Zone is an option to make requests faster but the regular S3 auth should still work.

I hope this information helps.

[4141done]

Thanks to @hveiga 's contribution we were able to implement compatibility #229

For usage instructions see this section of the readme: https://github.com/nginxinc/nginx-s3-gateway/blob/master/docs/getting_started.md#user-content-choosing-a-s3_style-setting

The session-based auth is not implemented. I've added an issue for it here #233