Skip to content

Allow windowed array to hold a wider window - #3061

Draft
jemmajeffree wants to merge 10 commits into
Parcels-code:mainfrom
jemmajeffree:big_memory_window
Draft

jemmajeffree wants to merge 10 commits into
Parcels-code:mainfrom
jemmajeffree:big_memory_window

Conversation

@jemmajeffree

Copy link
Copy Markdown

Description

Changes class WindowedArray to grab as many timesteps as requested by time_chunk, to allow for dask parallelisation of file reading alongside fast numpy interpolation.
See details in #3060

Checklist

  • This will ideally close Option to speed up #3060, but it will probably need iteration and comments from those more involved with the codebase
  • Tests added — nope, can do so but will need some guidance on what to put here, still relatively new to test driven development
  • This PR targets the correct branch (main for normal development, v3-support for v3 support)

No AI was used in the development/writing of this code

@jemmajeffree

jemmajeffree commented Nov 15, 2026 •

Copy link
Copy Markdown
Author

For what it's worth, this is how I've been defining a fieldset

chunks = {'lat':-1,'lon':-1,'time':24}

ds_fset = parcels.convert.mom6_to_sgrid(
    fields={"U": xr.open_mfdataset(uv_files,engine='netcdf4',chunks=chunks,
                                                                     )["SSU"].sel(time=slice('2009-01-01','2009-07-05')), 
            "V": xr.open_mfdataset(uv_files,engine='netcdf4',chunks=chunks,
                                                                     )["SSV"].sel(time=slice('2009-01-01','2009-07-05')),
            # Engine really shouldn't need to be specified, but it was throwing errors without it
           }, 
    coords=xr.load_dataset(coords_file,engine='netcdf4'),
)

fieldset = parcels.FieldSet.from_sgrid_conventions(ds_fset).to_windowed_arrays(time_chunk=24*28)

The rest of my testing setup was pretty much following the MOM6 tutorial

@erikvansebille

Copy link
Copy Markdown
Member

Thanks so much for this contribution, @jemmajeffree! And of course also all the other context and Issues. Really nice to see someone engaging with our code so much!

I see this PR is still in Draft - do you want a review of it? Or would you prefer to keep working on it?

@jemmajeffree

Copy link
Copy Markdown
Author

Please review it! It's working for my use-case, so I'm not planning on making changes without feedback

I just left it as a draft because I think it might be failing a handful of tests and I don't know if it generalises to all use cases, or if it aligns with how you would like the codebase to fit together

@erikvansebille

Copy link
Copy Markdown
Member

OK, I'll then Approve the workflows to run and will aks @wyatt-fluidnumerics or @fluidnumericsJoe to review this code, as they developed the WIndowedArrays() backend so have more insight into how this PR would work there

@wyatt-fluidnumerics wyatt-fluidnumerics left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @jemmajeffree , thanks for this PR and the associated issue, its especially cool how close to numpy performance you were able to get out of windowed arrays. A couple thoughts/questions which relate to your specific parcels use case, and then to this PR:

  • Are you using a dask.distributed.Client? I think that the speedup you are experiencing may be coming from the larger chunk reads rather than parallelized file reading. The netCDF backend for xarray has a per-process lock that prevents multiple threads within a process from reading from disk at the same time.
    xarray/backends/netCDF4_.py
        try:
            with self.datastore.lock:
                original_array = self.get_array(needs_lock=False)
                array = getitem(original_array, key)

The dask default for multi-core jobs is just to use multiple threads, because of the lock your reading speed with 1 vs 28 cores would be the same. In order to actually parallelize the file reading you'd need to set up a dask.distributed.Client and prescribe multiple single threaded processes by calling something like dask.distribured.Client(n_workers=28, threads_per_worker=1) before running the code for your parcels simulation.

  • I am also wondering if you have tried changing the max_cache_bytes in .to_chunk_cached_arrays(). With your chunking scheme (i.e. only chunking in time), ChunkCachedArrays should (to my knowledge) perform essentially the same as this modified version of WindowedArrays by pulling in large, multi step chunks of model data all at once. However, the default for max_cache_bytes is 600 MB, which would definitely not be enough to hold multiple model time levels, when max_cache_bytes for a variable is exceeded ChunkCachedArrays doesn't save the data to the cache at all, which would of course be very slow.
  • I am concerned this could introduce a bug in the interaction between self._max/max_levels and time_chunk. With these changes, the caching process loads an entire chunk worth of time levels on every read, however, when running into self._max cap, it still removes individual time levels. I worry that this could lead to weird behavior where not all of the time levels from a chunk are removed from the cache, and then the chunk is reloaded, leading to duplicate time levels in the cache.
  • On the point of generalization, I think it depends on how these changes compare to using ChunkCachedArray with comparably large chunks being loaded at once. If it turns out that this is significantly faster than a ChunkCachedArray with chunking of {'lat':-1,'lon':-1,'time':24*28} then I think it is definitely worth getting into the code base for other users, and I'd be happy to help work on a solution to the max_bytes/time_chunk problem.

@jemmajeffree

jemmajeffree commented Nov 16, 2026 •

Copy link
Copy Markdown
Author

Thanks!

  • Yes I'm using a dask.distributed.Client. Sorry, I overlooked it as part of the setup in my description because I'm always using one. Full code is here https://gist.github.com/jemmajeffree/13620b7188705af58d831e3012718dfe. I'm pretty sure I'm getting reading in parallel, from watching the dask task stream, though the task taking most of the time is called concatenate (as usual for reading a lot of data) so I guess it's possible that the thing taking a while isn't the raw reading per say.
  • I'll experiment with max_cache_bytes in chunked arrays. My understanding from the documentation is that this still only pulls one chunk at a time? In which case you would lose the speed-up from having the dask client. Is this correct? Also, does it have a deletion mechanism? In my use case, I know I'm going to need to read the entire dataset eventually, I just don't want to do it all at once.
  • Agreed on potential for a bug with max_levels. Can you explain to me what max_levels should do? It looked to me like max_levels got overrulled if isel asked for more, and like this line would throw out everything except the two levels requested anyway, so I didn't want to touch anything in here while not understanding. In what situation might Parcels ask for non-sequential levels?

@wyatt-fluidnumerics

Copy link
Copy Markdown
Contributor

Awesome!

Good point that you would loose the speedup from the dask client with ChunkCachedArrays. It does have a deletion mechanism, whenever the cache size hits max_cached_bytes the LRU cache purges the least recently used data, so you'd just want to make sure to set max_cached_bytes to be 2 or 3 times the size of your chunks, and it will act like a rolling buffer of data. That being said I think you are right that this loses the speedup from the dask client, so this wouldn't be faster.

max_levels is supposed to represent the greatest number of time levels that WindowedArray should ever hold in memory. This matters any time particles have non-synchronous clocks, which could happen if particles are released at different times that aren't divisible by outputdt, or you use an advection kernel that uses per particle dt (see AdvectionRK45 or #3016). Originally, WindowedArray was only implemented to support synchronous particle clocks, but since #2846 we've been trying to move towards an implementation that supports non-synchronous clocks, especially as we move towards a more generalized backend API structure (see #3005).

@jemmajeffree

Copy link
Copy Markdown
Author

I tested the same setup (2000 particles distributed randomly through space and time for 100 days) with both chunk cached arrays and my suggested windowed arrays backend. Lists of times are from repeat tests

  • Windowed Arrays: 176, 177, 186 s
  • Chunk Cached Arrays (same chunks): 255, 262, 268 s
  • Chunk Cached Arrays (28 times larger chunks): 218, 224 s
  • Chunk Cached Arrays (no dask client): 170, 177 s
  • Windowed Arrays (no dask client): 236 s

Taking the two setups in bold above, trying instead releasing 2000 particles at the start of the simulation rather than randomly distributed through time, I get:

  • Windowed Arrays 143, 145, 146 s
  • Chunked Cached Arrays (no dask client) 160, 178, 193 s

Slight improvement in some situations, not universal.
Which isn't what I would have expected, but if it works it works. I guess either the overhead of the dask client is not worth the parallel boost, or they're both limited by file reading anyway.

I think it'd be possible to persuade max_levels to coordinate with time_chunks (e.g. enforcing that it needed to be an integer multiple of time_chunks, and throwing out timechunks together), but if there isn't any speed-up to be gained then probably not worth introducing the complication.

Looking back at how I started down this rabbit hole; the "squeezing performance" docs implied that windowed arrays were going to be a more intelligent way of loading only some data when particles covered the whole domain. It looks like you've got the cached chunked arrays backend working well enough that it's actually fine in that scenario, as long as the max_cache_bytes reflects the system memory limitations and there's no dask client running.
Maybe the most useful thing to do is to update the docs to say so, and close this pr?

caveats would be:

  • If there's situations in which a dask client is necessary for other reasons it might be nice to have something working well with it, but I'm having trouble coming up with a specific scenario
  • If you're forcibly cooperating with something else using a dask client, like the copernicus marine toolbox?
  • If there's another reason for keeping windowed arrays around and you'd like the performance of these to keep up with chunked cached arrays
  • If it's worth having the small performance boost in some situations

Let me know your thoughts

@fluidnumericsJoe

Copy link
Copy Markdown
Contributor

I'm curious, what kind of file system is holding the NetCDF files for the benchmarks referenced in the discussion here ? Is this lustre, beegfs, vast, something else ??

@jemmajeffree

Copy link
Copy Markdown
Author

lustre
scratch described here https://nci.org.au/infrastructure/storage-systems

I'd be curious to know if people get different comparisons on different systems

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

Option to speed up

4 participants