HTTP-S3 Streaming Download Plugin

[Padeanu]

As a user I would want a download plugin that directly forwards HTTP response to a given S3 bucket without using local file system.

Chunk size, target url, source s3 bucket and key, credentials should be configurable.

Example using boto3:

with requests.get(stream_url, stream=True, auth=auth) as response:
    response.raise_for_status()  # Raise an error for bad responses (4xx and 5xx)
    chunk_size = 64 * 1024  # 64kb
    with response.raw as data_stream:
        boto3_client.upload_fileobj(data_stream, bucket, key, Config=boto3.s3.transfer.TransferConfig(multipart_threshold=chunk_size * 2))