Zarr Tile Sink

[annehaley]

This PR makes changes to the Zarr tile source to allow the user to write new N-dimensional files. This includes the addition of new, addTile, and write methods.

This PR also rewrites test/test_sink to copy data from TestFileTileSource rather than randomly generate data.

TODOs:

• Improve band bookkeeping
• Allow user to crop data upon write

Future Work:

• Expand large-image-converter output types #1454
• Generate lower resolution tiles (In zarr sink, save lower resolution tiles #1486)
• Multiprocessing support (Support multiprocessing through the zarr sink #1487)
• Add support for setting mm_x, mm_y, possibly minHeight and minWidth. It would be nice to also include in the list of properties we write to output files magnification and some form of image description (which could also just be an additional json record of metadata -- when generating data via algorithms, it is useful to store the parameters that were used). (In zarr sink, set mm_x, mm_y #1482, In zarr sink, set image description #1483)
• Add a method of prioritizing sinks so large_image.new() picks the most preferred one installed

[manthey]

I was playing with this a little, trying to replace a custom data collector with this branch. I was expecting the class properties sizeX and sizeY properties (and frames, bandCount, dtype) to update based on the data collected. Also, adding data is slow, because (a) we are using np.pad and (b) we are calling create_dataset on every change. If create_dataset is being called just to rechunk things, we can pick a chunking of 512 in x and y, a largish value in samples (bands) like 16, and 1 for all other axes.

[manthey]

I haven't checked ome zarr docs for version information. Are we more conformant to a dev version than the latest official version?

[annehaley]

I was just following what I found here: https://ngff.openmicroscopy.org/latest/#multiscale-md, but I'm not entirely confident that this is all correct.

[manthey]

With #1477, there are now some tests that can be enabled. Uncomment the two parametered tests on lines
https://github.com/girder/large_image/blob/master/test/test_examples.py#L90-L91 ; they should pass for this branch before we consider it mergeable.

[manthey]

It occurred to me this could be done without an explicit while loop. I think the equivalent would be:

if len(tile.shape) < len(axes):
    tile = np.expand_dims(tile, axis=tuple(range(len(axes) - len(tile.shape))))
if mask is not None and len(mask.shape) < len(axes):
    mask = np.expand_dims(mask, axis=tuple(range(len(axes) - len(mask.shape))))

I'm not sure that is clearer, though.

[manthey]

We should probably add self._levels = [] here and a test that just does

sink = large_image_source_zarr.new()
assert sink.metadata['levels'] == 0
assert sink.getRegion(format=TILE_FORMAT_NUMPY)[0].shape[:2] == (0, 0)

[annehaley]

Ok, added in 157c731

[manthey]

This needs to be outdented (not in the if axes[-1] clause).

[annehaley]

Thanks. Moved in 62e6572

[manthey]

Let's add a mask test:

def testAddTileWithMask():
    sink = large_image_source_zarr.new()
    tile0 = np.random.random((10, 10))
    sink.addTile(tile0, 0, 0)
    orig = sink.getRegion(format='numpy')[0]
    tile1 = np.random.random((10, 10))
    sink.addTile(tile1, 0, 0, mask=np.random.random((10, 10)) > 0.5)
    cur = sink.getRegion(format='numpy')[0]
    assert (tile0 == orig[:, :, 0]).all()
    assert not (tile1 == orig[:, :, 0]).all()
    assert not (tile0 == cur[:, :, 0]).all()
    assert not (tile1 == cur[:, :, 0]).all()

This will fail because of caching. In the vips source we have
https://github.com/girder/large_image/blob/master/sources/vips/large_image_source_vips/__init__.py#L125

and do self._cacheValue = str(uuid.uuid4()) as part of _initNew and when we invalidate the image (when we set self._levels = None).