impose additional limit on storage upload buffer size (#32907)

limit the buffers for multipart uploads to 100MiB, to protect against OOMs.
Note this means the maximum file size is 10000 * 100MiB which is 1TiB

GitOrigin-RevId: 591233fbd92a9b056bc71b85ba72bf1da748cafb

crates/common/src/knobs.rs

@@ -1242,3 +1242,16 @@ pub static DISABLE_FUZZY_TEXT_SEARCH: LazyLock<bool> =
 /// Conductors that are colocated.
 pub static INSTANCE_LOADER_CONCURRENCY: LazyLock<usize> =
     LazyLock::new(|| env_config("INSTANCE_LOADER_CONCURRENCY", 16));
+
+/// The maximum number of bytes to buffer in an multipart upload.
+/// There may be stricter limits imposed by the storage provider, but this is
+/// the target max size for the buffer to protect against memory exhaustion.
+/// The maximum number of parts is 10000, so this imposes a max file size, which
+/// defaults to 10000 * 100MiB = 1TB. When reducing this knob, make sure the
+/// maximum file size can fit a snapshot export of the instance.
+/// Storage uploads can run in parallel, and the buffers get cloned during
+/// upload, so make sure this knob times ~8 can fit in memory.
+///
+/// Defaults to 100MiB.
+pub static STORAGE_MAX_INTERMEDIATE_PART_SIZE: LazyLock<usize> =
+    LazyLock::new(|| env_config("STORAGE_MAX_INTERMEDIATE_PART_SIZE", 100 * (1 << 20)));