Nothing to write: null body - when using Flux<ByteBuffer> and S3CrtAsyncClient

[campidelli]

Hi, me again.

In my application, I am using Spring WebFlux for the asynchronous APIs (https://github.com/campidelli/file-storage-service/blob/main/src/main/java/campidelli/file/storage/controller/AsyncFileStorageController.java) and it works well when I am using a real S3 bucket.

However, when I use S3Mock, it seems that no data is getting to the container as the following log show:

2023-06-21 04:27:48.611 DEBUG 1 --- [tp1227264471-22] o.s.web.servlet.DispatcherServlet        : PUT "/the-return-of-sherlock-holmes.pdf", parameters={}
2023-06-21 04:27:48.616 DEBUG 1 --- [tp1227264471-22] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to com.adobe.testing.s3mock.BucketController#createBucket(String, boolean)
2023-06-21 04:27:48.637 DEBUG 1 --- [tp1227264471-22] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Using 'application/x-www-form-urlencoded', given [application/x-www-form-urlencoded, application/xml] and supported [application/json, application/*+json, application/json, application/*+json, application/xml, application/x-www-form-urlencoded, application/octet-stream, application/xml;charset=UTF-8, text/xml;charset=UTF-8, application/*+xml;charset=UTF-8]
2023-06-21 04:27:48.637 DEBUG 1 --- [tp1227264471-22] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Nothing to write: null body

What am I missing? I think now it is easy to reproduce it there, you only need to execute https://github.com/campidelli/file-storage-service/blob/main/src/test/java/campidelli/file/storage/FileStorageApplicationTests.java

Another tip I would like to receive is: how can I enable the container to write to an actual folder on my host? I have tried to use withEnv("root", "/Users/campidelli/temp") but it times out.

Many thanks!

[afranken]

The only thing I really had to fix was that the S3Client must use "path style access" with S3Mock. The default subdomain style access does not work locally because localhost is a special domain. I have yet to come up with a solution to make this work.

See campidelli/file-storage-service#1

See #144

[campidelli]

Wow! Many thanks @afranken that was brilliant, your level of commitment is outstanding!

It is working perfectly now!

Have a great day sir!

[campidelli]

Hey, sorry (again).

I noticed a difference between the uploaded file and the original one:

2023-06-22T11:32:58.238+12:00  INFO 2973 --- [           main] c.f.storage.FileStorageApplicationTests  : Actual body length 1318787
2023-06-22T11:32:58.242+12:00  INFO 2973 --- [           main] c.f.storage.FileStorageApplicationTests  : Expected body length 1318741

And when I tried to open the uploaded PDF, it doesn't work, it says it is corrupted.

I added a new test to check that.

[afranken]

found the problem.

Seems like the CRT client automatically sends a checksum to S3.
The S3 SDK puts the checksum into the body of the request instead of a header, and S3Mock currently completely ignores the checksum.

When running a test with the CRT client, our sample file:
https://github.com/adobe/S3Mock/blob/main/integration-tests/src/test/resources/sampleFile.txt
looks like this:

24
## sample test file ##

demo=content
0
x-amz-checksum-crc32:I6zdvg==

See #1123

The workaround is to disable checksum validation when using S3Mock:
campidelli/file-storage-service#2

[campidelli]

Amazing thanks a lot!

I think it is worth to add this to the README file so people can be aware.

Have a great day!