Repository navigation
Description
What version of Parcels are you running?
main branch
Is your feature request related to a problem?
Firstly, thanks to all who've worked on this codebase and made such an amazing tool. I'm following up on the request here for suggestions on how to make Parcels work faster.
Within my research group, the following scenario is fairly common.
- You want to track particles through ocean model output for several years
- You're working on a HPC system, because that's where the ocean model output is stored. A common setup would be 120GB RAM and 28 cores.
- Therefore, fitting the entire fieldset into memory is not possible, but fitting a month's worth of data into memory is easy.
In this situation, completely loading the data and using numpy is not possible. Using windowed_array sounds great in theory, but if you're loading a timestep at once with dask then the chunks are far too small to be optimal, and waste the fact that you have 28 cores you could be using. If you load 28 reasonably-sized time chunks into memory at once, then loading should be (at least) 28 times faster. (28x from the cores, and probably another factor of two from not having inefficient single timestep chunks)
Describe the solution you'd like
Adapt windowed array to have larger windows, so it can handle many chunks at once. I've had a stab at this, loading in the next time_chunk timesteps when isel triggers a request on one, rather than only the next one timestep.
For the tests I've done, I've found this implementation uses ~4-5 times the memory of the original windowed_arrays approach, and takes about 50% longer than a pure numpy approach. For me, this was 13 GB of memory and integrating for 30 days in 5 minutes. On the other hand, it's ~10 times faster than the windowed arrays approach, while still using way less memory than numpy would. (Also, I didn't count the data loading step in the pure numpy benchmarking, so it's probably a conservative estimate.) Overall, this is nearly numpy-like speeds with constant memory use (i.e. not increasing with run length).
Describe alternatives you've considered
No response
Additional context
This works for my use case, but I've probably made some assumptions about how Parcels is used, and I'd welcome feedback on thoughts on how to generalise this solution, or whether it generalises at all.
I have also assumed that the time axes of U and V are monotonic.
And I think this implementation needs to be used with max_times=None, though I was having trouble understanding how max_times threw stuff out, given the windowing documentation implies it's only keeping 2 times anyway.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsBacklog
Activity