You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
WindowedArrays and ChunkCachedArrays not performing as expected when streaming data from copernicusmarine #2756
As described in Parcels-code/parcels-benchmarks#49, using to_windowed_arrays() when streaming data directly from the copernicusmarine store is approximately 25x slower than temporarily storing the files to disk.
As streaming the data from CopernicusMarine is the preferred way (no need for temporary storage, and able to scale to much larger datasets than required here), this slowdown is not acceptable.
This issue is actually not fixed yet. We have found that even ChunkCachedArrays can be much slower in streaming mode than downloading the data first. A simulation by @fleurrothstegge took more than 60 hours in streaming mode, and only an hour or so in when first downloading the data.
changed the title [-]WindowedArrays not performing as expected when streaming data from copernicusmarine[/-][+]WindowedArrays and ChunkCachedArrays not performing as expected when streaming data from copernicusmarine[/+]on Nov 5, 2026
I think this might be being caused by the same chunking problem addressed in #3061
I get the impression that the copernicus marine uses dask on the backend, (e.g. mercator-ocean/copernicus-marine-toolbox#409) and the speed limitation on handing over data is the same chunk/graph type of problems you'd encounter reading things with dask locally. Given WindowedArrays ask for single-timestep chunks, even a tiny overhead on grabbing each of those chunks could blow out the simulation. You might be able to win over that with spatial subsetting in chunk-cached-arrays for a single particle, but probably not for a big simulation.
Either way, I get (slower on the internet connection through to Australia, and more noisy because I've been copy-pasting my password in because I haven't set up copernicus marine to login automatically)
streaming with chunk cached arrays (default) 153 s
using —load_mode as_file97 s
streaming with windowed arrays and time_chunk = 10100 s
streaming with windowed arrays and time_chunk=1 (original windowed arrays) 176 s
Activity
erikvansebille commentedon Sep 14, 2026
Well, it seems that the
CunkCachedArrayBackend fixed this - we now get much better performance with streaming: see Parcels-code/parcels-benchmarks#49 (comment)erikvansebille commentedon Nov 5, 2026
This issue is actually not fixed yet. We have found that even
ChunkCachedArrayscan be much slower in streaming mode than downloading the data first. A simulation by @fleurrothstegge took more than 60 hours in streaming mode, and only an hour or so in when first downloading the data.[-]WindowedArrays not performing as expected when streaming data from copernicusmarine[/-][+]WindowedArrays and ChunkCachedArrays not performing as expected when streaming data from copernicusmarine[/+]jemmajeffree commentedon Nov 16, 2026
I think this might be being caused by the same chunking problem addressed in #3061
I get the impression that the copernicus marine uses dask on the backend, (e.g. mercator-ocean/copernicus-marine-toolbox#409) and the speed limitation on handing over data is the same chunk/graph type of problems you'd encounter reading things with dask locally. Given WindowedArrays ask for single-timestep chunks, even a tiny overhead on grabbing each of those chunks could blow out the simulation. You might be able to win over that with spatial subsetting in chunk-cached-arrays for a single particle, but probably not for a big simulation.
Either way, I get (slower on the internet connection through to Australia, and more noisy because I've been copy-pasting my password in because I haven't set up copernicus marine to login automatically)
—load_mode as_file97 stime_chunk = 10100 stime_chunk=1(original windowed arrays) 176 s