Lazy loading of Zarr from S3

Help anndata
1

Hi wonder if anyone could help me to understand if this is possible;

ad = anndata.read_zarr('s3://foo/bar/baz.zarr')
qux = ad.varm['qux']

Such that only qux is ever pulled from s3 - and the rest of the data is never moved?

For any arbitrary part of the data structure this is not just varm…

2

You could try using the experimental read_dispatched API Dask + Zarr, but Remote! — anndata 0.12.0.dev42+g3d0105b documentation

I have an example here where I defined a custom read function which omits certain keys when reading the AnnData object compasce/src/compasce/io/zarr_io.py at da925c07494cfedf1b46b250722c336d9111cf53 · keller-mark/compasce · GitHub

3

Alternatively, you could try shadows GitHub - scverse/shadows: Shadow objects for AnnData and MuData (experimental) (I do not have experience with this approach)

4

@josh-gree, I would recommend using the anndata.io.read_elem function here. This would look like:

z = zarr.open("s3://foo/bar/baz.zarr")
qux = anndata.io.read_elem(z["varm"]["qux"])