Default value for AWS_S3_MAX_MEMORY_SIZE is dangerous for large files

[swiatekm-side]

AWS_S3_MAX_MEMORY_SIZE sets the max size for a SpooledTemporaryFile, which the S3Boto3StorageFile uses for temporary storage. The default of 0 means "never rollover, keep contents only in memory", which causes issues with large files. In particular, even if we only want to read the first X bytes of the file, the current implementation (

django-storages/storages/backends/s3boto3.py

Line 133 in b9bec8b

self.obj.download_fileobj(self._file)

) will still download the whole thing and keep it in memory.

I think this is an unpleasantly surprising default - that users don't expect their files to be kept wholly in memory whenever their contents are accessed, irrespective of their size. The current default behaviour of "keep the buffer in memory always" should be an opt-in rather than an opt-out.

Let me know if I'm missing something here. Given that I've used storages for a while and haven't ever noticed this, I assume the typical use cases don't care. I don't think changing the default would really impact them in a negative way, though.

[jschneier]

Closed in #1359