Skip to content

Copernicusmarine streaming performance - #49

Merged
erikvansebille merged 12 commits into
mainfrom
Copernicus_streaming
Nov 3, 2026
Merged

erikvansebille merged 12 commits into
mainfrom
Copernicus_streaming

Conversation

@erikvansebille

@erikvansebille erikvansebille commented Jul 17, 2026 •

Copy link
Copy Markdown
Member

UPDATE: to_chunk_cached_arrays() seems to really help with performance of copernicusmarine streaming! See #49 (comment) below

This PR adds a script to test the performance of directly streaming from the copernicusmarine() store. As also found in Parcels-code/virtualship#357, directly accessing the data from there is much slower than temporarily saving the data to local disk

A quick performance overview of the script here on my MacBook:

  • Running with --load_mode as_file takes 21 seconds. In this case, the full DataSet is stored to a local netcdf file, and then loaded again. Note that it automatically loads as NumPy (probably because the file is only 256MB in size)
  • Running with --load_mode ds_load takes 20 seconds. In this case, the full DataSet is not stored on disk but loaded directly in memory with a ds.load()
  • Running with --load_mode streaming takes 518 seconds, even when we use to_windowed_arrays(). As this is the preferred way to stream data from CopernicusMarine (no need for temporary storage, and able to scale to much larger datasets than required here), the 25x slowdown is not acceptable.

Note 1: I've been able to reduce the --load_mode streaming time to 142 seconds in 7c5a0ac, by setting the options copernicusmarine.open_dataset(..., service="arco-geo-series", chunk_size_limit=1). This is inspired by a discussion at mercator-ocean/copernicus-marine-toolbox#267

Note 2: I can get --load_mode streaming even faster (equivalent to --load_mode ds_load) by setting chunk_size_limit=0, but this disables dask altogether so will have a major memory footprint (as it bypasses the WindowedArrays)

@erikvansebille

Copy link
Copy Markdown
Member Author

An update on these performances:

I have moved offices and my internet is apparently slower, because I now get

  • Running with --load_mode as_file takes 70 seconds. In this case, the full DataSet is stored to a local netcdf file, and then loaded again. Note that it automatically loads as NumPy (probably because the file is only 256MB in size)
  • Running with --load_mode ds_load takes 62 seconds. In this case, the full DataSet is not stored on disk but loaded directly in memory with a ds.load()
  • Running with --load_mode streaming takes 116 seconds, with to_windowed_arrays() and using the service="arco-geo-series", chunk_size_limit=1 in the copernicusmarine call
  • Running with --load_mode streaming takes 33 seconds, with to_chunk_cached_arrays() without the extra service and chunk parameters (see 6c2d69c).

So it seems that using CunkCachedArray Backend can be faster than loading the whole dataset into memory (which is expected for this one single particle, but still a nice result)!

@erikvansebille
erikvansebille merged commit 63bb61b into main Nov 3, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant