Skip to content

Check and fix chararray string dimension names - #10395

Merged
kmuehlbauer merged 13 commits into
pydata:mainfrom
kmuehlbauer:fix-chararray-strings
Jun 20, 2025
Merged

kmuehlbauer merged 13 commits into
pydata:mainfrom
kmuehlbauer:fix-chararray-strings

Conversation

@kmuehlbauer

@kmuehlbauer kmuehlbauer commented Jun 4, 2025 •

Copy link
Copy Markdown
Contributor

@kmuehlbauer kmuehlbauer changed the title Fix chararray strings Check and fix chararray string dimension names Jun 4, 2025
@kmuehlbauer

Copy link
Copy Markdown
Contributor Author

This is ready for review.

In case of character arrays when encoding the PR aims to:

  1. check encoding["char_dim_name"] for something like string10 -> which gets splitted into string (prefix) and 10 (strlen)
  2. if the size of the last dimension of the character array isn't equal to strlen, a warning for possible dimension naming clash is issued and a new dimension name is created with the actual array size
  3. if no trailing number is found, we check if encoding["original_shape"] is set, if it doesn't corresponds to the actual string length, a warning for possible dimension naming clash is issued and a new dimension name is created with the actual array size
  4. if encoding["char_dim_name"] isn't set, we continue creating new dim names with stringN naming scheme.

At this place each variable is handled separately so we do not have the full picture of all needed character array dimensions. Since we now issue warnings and create fitting dimensions names to prevent breakages there are chances we get false positives (eg in the case of several variables with the same size which have all received the same change in size and have encoding["char_dim_name"] = "somestring" without numbering and also original_shape set). I think that additional warnings in those cases might help users in identifying issues and are acceptable.

@kmuehlbauer

kmuehlbauer commented Jun 12, 2025 •

Copy link
Copy Markdown
Contributor Author

Pydap test failure in all-but-numba CI run might be real. Need to investigate.

Update: Fails in other PR, too. Pydap backend errors are most likely related to recent changes in numpy (2.2.6 works, 2.3.0 fails)

@kmuehlbauer

Copy link
Copy Markdown
Contributor Author

Great, works now. I'll mark this as planned form merge and go ahead with this next week. Nevertheless it would be great to have another set of eyes on this one.

@kmuehlbauer kmuehlbauer added plan to merge Final call for comments labels Jun 13, 2025
@kmuehlbauer

Copy link
Copy Markdown
Contributor Author

This fixes inconsistencies which lead to breakages in the past and warns users about that. Given the additional benefit for users and as it doesn't change functionality of non-breaking code, I'm going to merge now.

@kmuehlbauer
kmuehlbauer merged commit c19aa4f into pydata:main Jun 20, 2025
@kmuehlbauer
kmuehlbauer deleted the fix-chararray-strings branch June 20, 2025 07:08
LemonCondor pushed a commit to LemonCondor/xarray that referenced this pull request Oct 19, 2026
Restore the previous rule that the last number in char_dim_name is the
string length, so the clash protection from pydata#10395 keeps working, and keep
the characters after that number. A mismatched "n4_chars" now becomes
"n3_chars" instead of "n3" (main) or "n4_chars3" (previous commit).
headtr1ck pushed a commit that referenced this pull request Oct 21, 2026
* Keep the full char_dim_name when it contains a non-trailing number

validate_char_dim_name matched the last digit run anywhere in the name and
rebuilt the name as prefix + strlen, dropping everything after the digits.
A char dimension like "n3_chars" was therefore re-encoded as "n3", so a
decode/encode roundtrip silently renamed it. Only treat a trailing number
as the string length.

Co-authored-by: Claude <noreply@anthropic.com>

* Add PR number to whats-new entry

* Keep the number as the length and preserve the suffix in char_dim_name

Restore the previous rule that the last number in char_dim_name is the
string length, so the clash protection from #10395 keeps working, and keep
the characters after that number. A mismatched "n4_chars" now becomes
"n3_chars" instead of "n3" (main) or "n4_chars3" (previous commit).

---------

Co-authored-by: Guillaume Marty <condor020@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: root <root@localhost.localdomain>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

io plan to merge Final call for comments topic-backends topic-documentation topic-zarr Related to zarr storage library

Projects

None yet

1 participant