Repository navigation
Conversation
raulEspaa
force-pushed
the
fix/inmemory-memory-combining-marks
branch
from
November 16, 2026 09:42
90e3df2 to
52a9ace
Compare
…earch The in-memory service extracted words with \w+, which does not match combining marks, so Thai and Devanagari words were split at their vowel and tone marks. The pieces were then matched as substrings, and a search for ข้าว also returned unrelated Thai text. Tokenize the same way SqliteMemoryService does: combining marks are word characters, and a mark with no letter before it is dropped. Fixes google#8168
…arch Adds cases to test_search_memory_non_latin for Bengali and Tamil vowel signs and virama, a Latin word inside Thai text, and the keycap emoji 1 + U+FE0F + U+20E3. The Bengali, Tamil and Latin+Thai negative cases fail without the combining-mark fix.
raulEspaa
force-pushed
the
fix/inmemory-memory-combining-marks
branch
from
November 16, 2026 09:52
52a9ace to
05987f9
Compare
This was referenced Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
1. Link to an existing issue (if applicable):
Problem:
InMemoryMemoryServiceextracted words withre.findall(r'\w+', ...).\wdoesn't match combining marks, so Thai and Devanagari words were split at their vowel and tone marks (ข้าว became ข and าว). Non-ASCII query words are also matched as substrings of the event text, so those pieces matched unrelated memories: a search for ข้าว returned ขอบคุณครับ, and a search for हिंदी returned a sentence about Hyderabad.Solution:
Tokenize the same way
SqliteMemoryServicedoes (_is_token_char/_extract_tokens): combining marks count as word characters, and a mark with no letter before it is dropped, so the emoji variation selector U+FE0F doesn't become a word. Both_extract_words_lowerand_extract_searchable_wordsnow use a small_find_wordshelper instead of\w+. When a word is split into Latin and non-Latin runs, a run that is only a combining mark is dropped too.reis no longer used in the module, so I removed the import. Text without combining marks gives the same words as before.Testing Plan
Unit Tests:
I added cases to
test_search_memory_non_latinintests/unittests/memory/test_in_memory_memory_service.py: Thai and Hindi queries that should and shouldn't match, and an emoji with a variation selector. The two "shouldn't match" cases fail without the fix and pass with it.pre-commit run --filespasses on the changed files, andmypyreports no issues for the changed module.Manual End-to-End (E2E) Tests:
I ran the reproduction script from #8168, which stores four Thai and Hindi sentences in
InMemoryMemoryServiceand searches for ข้าว and हिंदी.Before:
After (the same results
SqliteMemoryServicegives for this data):Checklist
Additional context
#8141 also changes
in_memory_memory_service.py, but in a different part of the file (rewind handling), so the two changes shouldn't conflict.