Add support for SpatialData

[pennycuda]

Hello! I opened another PR #587 a little over a week ago to support zipped SpatialData zarr stores. After some conversation with @will-moore and @jo-mueller about the image class overhaul, I decided to wait on Jo's PR to be merged and redo my work after.

Jo's PR was merged yesterday, so I am excited to move forward with my changes. This will reintroduce SpatialData support to ome-zarr-py by adding handling to accept the 0.5-dev-spatialdata ome version in SpatialData metadata.

I tested my changes by using SpatialData's read_zarr() method on zipped and unzipped SpatialData zarr stores. The version of read_zarr() that I used is a combination of changes from the soon to be merged PR scverse/spatialdata#1107 and my branch of that PR that accepts ZipStores and works in the new ome-zarr-py image class changes.

Penny Cuda, she/her, HuBMAP Consortium

[jo-mueller]

So...first thought is that I'm not sure whether I'd like to see the version written by the spatial data projectspecial-cased over here.

From what I see the added code

• adds parsing for the coordinateaTransformations` field - I'm on my phone rn so I can't tell whether that's new (will check later)
• Parses some additional metadata into a spatialdata_transforms field? That I find more problematic because the metadata will essentially be unvalidated on both read and write

A way forward I could imagine:

• Replace the if version == "0.5:" further up with if "0.5" in version:. This will make ome-zarr-py accept a version field such as the 0.5-dev-spatialdata
• Remove reading in spatialdata-transforms because that's something an implementation that uses ome-zarr-py can always do outside the from_one_zarr method after reading.

[pennycuda]

Thank you for the feedback @jo-mueller ! I had the same thoughts about the spatialdata version right after I submitted this PR. 🤦🏻‍♀️ I will take in your feedback and push another commit soon. I'm at a conference this week, so I might be delayed in my responses and commits.

[jo-mueller]

Is that conference ELMI, by any chance? ^^

[pennycuda]

No actually ! Today is HuBMAP Consortium's final meeting, and the next two days are Spatial Biology - The New Frontier. Both in Rockville, MD.

[jo-mueller]

Ok. would have been funny :)

[LucaMarconato]

Thanks @pennycuda for the push. I'll write a few notes on the roadmap, on the meaning of the string 0.5-dev-spatialdata (and 0.4-dev-spatialdata), and how to proceed.

spatialdata was created when OME-Zarr 0.4 was out, while needing the support for coordinate transformations (what today we call RFC-5). This was a few years back, and RFC-5 was supposed to be part of the specs within a few months, so we already started building on top of what should have been the 0.5 format (i.e. 0.4 + RFC-5).

Since 0.5 was requiring more time, we started using the string 0.4-dev-spatialdata. The effort for RFC-5 was turning out to be larger than anticipated: when OME-Zarr 0.5 was released it still could not support RFC-5. From our side, we added support for 0.5 in spatialdata, but still kept using RFC-5 (therefore bumping 0.4-dev-spatialdata to 0.5-dev-spatialdata).

Things were moving slowly, but were moving. And from the last year code and specs have started moving much faster! Thanks to @jo-mueller and collaborators the spec is now close to include RFC-5 into OME-Zarr 0.6. The plan from the spatialdata side is to switch to 0.6 as soon as possible. This will cleanup things on the file format and code, and make interoperability easier.

Until then (or to support earlier dataset), what is possible (this is what we implement), is that we read and write the OME-Zarr store as if it was 0.5, and then we manually add the transformations part. RFC-5 has evolved since our implementation, so there are some differences on how the coordinate systems are encoded in 0.5-spatialdata-dev and the current RFC-5. I therefore endorse the suggredstion from Johannes to keep the avoid having project specific (i.e. the string spatialdata) in a diff, but allow proceeding with 0.5 parsing on specialized versions strings that include 0.5. The rest of the parsing logic ideally would live outside the ome-zarr-py.

Some few useful pointers.

• Here you can find a schema for 0.5-dev-spatialdata from @Bisaloo https://github.com/Huber-group-EMBL/SpatialData.validate/tree/main/inst/extdata
• We recently expanded the spatialdata dev team and we are going to share a public roadmap soon.
• The top items of the roadmap are:
  • Create a new repo to document the details of 0.5-dev-spatialdata and future versions. This repo will be entirely about describing the format, no code. Now source of truth for the spatialdata spec is in format of Python version classes, and therefore difficult to read: https://github.com/scverse/spatialdata/blob/d024c0591bb925c81642702db781f113102b9ece/src/spatialdata/_io/format.py.
  • Implement 0.6 support in spatialdata (this is actually what we are working on now) in preparation to the official OME-Zarr 0.6 release, which is imminent.

@pennycuda can you please share the timeline of HuBMAP, hopefully they are aligned and happy to support from the spatialdata side.