Ping-ing a file to get dimensions whilst streaming it elsewhere

[ozziepeeps]

Hi Magick.NET community. Thank you in advance for a great library and for helping out 👍

I have a .NET Stream that I am streaming up to AWS S3 via the AWSSDK. The benefit of this is I don't need to have the entire file (represented by the Stream) in memory at once so it is memory efficient.

I'd also like to be able to leverage MagickImage's Ping method (which takes a Stream) to extract the image dimensions. Is there an approach I can take to stream the file up to S3 and also end up being able to get the dimensions from Ping in a single pass?

I have to assume that the Stream is not seekable (since it is coming in from an HTTP request), and don't want to copy it into a MemoryStream as that will spike my memory usage and some of my files can be very large.

I'm looking for perhaps a "decorator" approach where I can wrap the Stream and stream it up to S3 and in the process also have MagickImage process the Stream in order to Ping for the dimensions.

Any ideas would be much appreciated! Thank you!

[dlemstra]

This feels like a premature optimization and there is no build in method to do this. You will need to create a stream that would read a byte buffer from disk give it to ImageMagick and then write that output buffer to your AWS stream in another thread.

[ozziepeeps]

Thank you @dlemstra. I ended up taking a approach similar to what you suggested.

[ozziepeeps]

For anyone interested in how I solved this, I made use of System.IO.Pipelines. It is essential to know the incoming stream length up front, which you can read from the incoming Content-Length HTTP header. I construct a new System.IO.Pipelines.Pipe, and using a wrapper / decorator around the incoming Stream, it gets copied to the PipeWriter as it is read. The PipeReader's Stream on the other end of the Pipe is wrapped in another wrapper / decorator that conveys the Content-Length that we read up front, since S3 needs to know this to ingest the Stream.

Here is some pseudo code:

Stream inbound = ; // Stream from ASP.NET Core
long contentLength = ; // Value of incoming Content-Length header

var pipe = new Pipe();

// Decorator copies bytes from wrapped inbound stream to the PipeWriter as data is read.
using var multiplexer = new Multiplexer(wrapped: inbound, pipe.Writer);

// Another decorator conveys the length of the stream being ingested since S3 needs to know that.
using var outbound = new KnownLengthStream(wrapped: pipe.Reader.AsStream(), contentLength);

// Commence the S3 upload of the outbound stream.
var putObjectResponseTask = s3.PutObjectAsync(outbound); // Note that we are NOT awaiting here.
					
// In parallel with the upload, "ping" the image on the multiplexing stream.
// This causes Magick.NET to start to walk the stream, which fills the pipe and makes data available to S3.
using var image = new MagickImage();
image.Ping(multiplexer);

var width = image.Wiidth;
var height = image.Height;

if (multiplexer.Position < contentLength)
{
    // Do a sanity check to ensure that we've walked the entire stream. If we haven't (if Magick.NET has been able to 
    // extract dimensions without walking the entire stream), then it's important to drain the rest of the incoming stream
    // into the multiplexer so that all bytes get written to S3.
    await multiplexer.DrainAsync(...).ConfigureAwait(false);
}

// Await the S3 upload finishing.
await putObjectResponseTask.ConfigureAwait(false);