`concat_on_disk` with `join='outer'` doesn't retain all `.obsm` fields

[alam-shahul]

Not sure if this is a bug or a missing feature, but ad.concat and ad.concat_on_disk seem to behave differently with join='outer' for .obsm fields that are not shared between all AnnData.

Please make sure these conditions are met

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of anndata.
• (optional) I have confirmed this bug exists on the main branch of anndata.

Report

Code:

# /// script
# requires-python = ">=3.12"
# dependencies = [
#   "anndata@git+https://github.com/scverse/anndata.git",
# ]
# ///

import anndata
import numpy as np
a = ad.AnnData(
        X=np.ones((2, 10)),
        obsm={'X_emb': np.arange(4).astype(np.float64).reshape((2, 2))},
)
b = ad.AnnData(
        X=np.ones((4, 12)),
)

c = ad.concat([a, b], join="outer")
assert 'X_emb' in c.obsm.keys()

a.write_h5ad('a.h5ad')
b.write_h5ad('b.h5ad')

ad.experimental.concat_on_disk(['a.h5ad', 'b.h5ad'], out_file='d.h5ad', join="outer")

d = ad.read_h5ad('d.h5ad')
assert 'X_emb' in d.obsm.keys()

Versions

0.12.10

[ilan-gold]

Hmm I don't see any prima-facie reason why this would be the case but I also don't see any handling of empty elements anywhere in concat_on_disk. I think your problem also surfaces with layers:

# /// script
# requires-python = ">=3.12"
# dependencies = [
#   "anndata@git+https://github.com/scverse/anndata.git",
# ]
# ///

from __future__ import annotations
import anndata as ad
import numpy as np

a = ad.AnnData(
    X=np.ones((2, 10)),
    layers={"X_emb": np.arange(20).astype(np.float64).reshape((2, 10))},
)
b = ad.AnnData(
    X=np.ones((4, 12)),
)

c = ad.concat([a, b], join="outer")
assert "X_emb" in c.layers.keys()

a.write_h5ad("a.h5ad")
b.write_h5ad("b.h5ad")

ad.experimental.concat_on_disk(["a.h5ad", "b.h5ad"], out_file="d.h5ad", join="outer")

d = ad.read_h5ad("d.h5ad")
assert "X_emb" in d.layers.keys()

I'm wondering if this is by design for performance reasons. @selmanozleyen Do you remember why this would be the case?

[selmanozleyen]

I remember not implementing outer join. So it's not for performance reasons. Probably wasn't meant to be supported initially