disallow arrays or groups named "zarr.json"

[d-v-b]

Practically, there are 2 ways to identify an array or group: by name, or by <name>/zarr.json. The former is the "hierarchy name", and the latter is the actual storage key that defines the presence of the array or group.

The hierarchy name is generally not a stored object, but rather a prefix. On local storage, this means <name> is a directory, and on object storage, <name> does not resolve to anything, it has no content type, etc. By contrast, <name>/zarr.json is a stored object.

This means there are some advantages to identifying a Zarr array or group by its stored representation, i.e. <name>/zarr,json:

• As a file, zarr,json can easily support drag + drop behavior for GUI applications that work with conventional single-file data formats.
• as a resource, zarr.json can be associated with a JSON schema and a content type.

However, to support these two forms of declaration (<name> or <name>/zarr.json), a client would need a way to distinguish them structurally. Is <name>/zarr.json the name of a group called zarr.json (i.e., should the client check if <name>/zarr.json/zarr.json exists?), or the key of a metadata document for the group called <name>? On object storage both could even be true -- zarr.json and zarr.json/zarr.json can both be the keys for metadata documents.

To allow clients to treat <name> and <name>/zarr.json as valid identifiers for the array or group <name>, we need to exclude the possibility that <name>/zarr.json is the name of an array or group. IMO the best way to do this is by stipulating that arrays and groups MUST not be named zarr.json. If we add this restriction, then users can safely pass strings like foo OR foo/zarr.json into zarr-aware applications without ambiguity, and Zarr data can be declared in an object-stored-friendly way.

[d-v-b]

xref #123

[jbms]

I see that currently the "node names" section of the spec (https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html#node-names) does not explicitly disallow zarr.json as a node name --- I recall us discussing this issue before, but I guess it didn't end up in the spec. Certainly we should disallow zarr.json as a node name, since that would anyway conflict with the zarr.json metadata file except on object storage systems like s3 that allow a file and "directory" with the same name.

However, I think that only partially addresses your issue, because you are talking about general paths / URLs to a zarr array which falls outside the current scope of the spec, and is more in the realm of ZEP 8 / the url pipeline spec (https://github.com/jbms/url-pipeline) I created. Still I agree that what you describe is a useful heuristic. You could anyway handle it even without a strict requirement by checking for a file or directory, but being able to do the resolution without any I/O has advantages.

[d-v-b]

However, I think that only partially addresses your issue, because you are talking about general paths / URLs to a zarr array which falls outside the current scope of the spec

I'm using the exact same language about node names and storage paths the spec uses. The spec uses this language in the context of a single abstract store, whereas real implementations must use additional information to choose which concrete store to associate with a string declaration of Zarr data. In this context the url pipeline spec looks very promising. But the spec itself effectively gives nodes two identifiers: a hierarchy name, and a storage path to a metadata document.

[TomNicholas]

Strong +1 to this - creating an array named zarr.json currently breaks zarr-python, icechunk, and everything downstream.