S3 endpoint compatibility

[CEikermann]

Is your feature request related to a problem? Please describe.
Right now you can not use nginx-s3-gateway as a s3 endpoint compatible proxy.
It is not possible to use any S3 library with overwriting the s3 endpoint towards the nginx-s3-gateway.
Background usecase: we want to cache files locally when using the AWS SDK

Describe the solution you'd like
It would be great if the nginx could can be adjust to support this use case. See more details in the additional context

Additional context

• 1.) Run the nginx-s3-gateway on localhost:8000

• 2.) aws --endpoint-url http://localhost:8000 --debug s3 cp s3://any-bucket-name/any-file-path.txt .

• nginx-s3-gateway is not support this case, as its calls the following url: http://localhost:8000/any-bucket-name/any-file-path.txt

• This way also the AWS SDK is emitting S3 request when using s3 endpoint url overwrite

• It would be great, if we can support this case, maybe but adding a location part into the nginx conf which listens on /{S3_BUCKET_NAME} - this should solve the problem already

Looking forward to your reply on this

[dekobon]

Hi Ceikermann,

I've converted this issue to a discussion because I believe it is a better format.

The S3 Gateway was intentionally designed for the use case of using S3 like a backend filesystem allowing someone to still use NGINX as a webserver albeit backed by S3.

What you are asking about is a separate use case. I do think much of the plumbing to do your use case is in place, but it would require many changes and forked logic. Honestly, I believe that such a thing would be better as a separate project. If you wanted to create a project, I would be happy to support it.

[balbok0]

I think docs could state the lack of support for S3 API on the proxy side more clearly. Just personally, I've just spent a good chunk of my day trying to setup nginx-s3-gateway to learn that S3 API is not supported.

[balbok0]

Picking this up to see if anyone knows if there has been any movement on this or similar projects. So far I am aware of long discontinued MinIO Gateway, but it's way too old to use due to security concerns. Unlike author here, I am specifically interested in get/list operations (although put would be a nice bonus as well).

Also, shouldn't it be possible to do with a regular nginx? I am not sure how to always force authentication with the proxied S3 endpoint, but after that ranges could be cached.