Skip to content

fix(memory): keep combining marks in words in InMemoryMemoryService search - #8169

Open
raulEspaa wants to merge 2 commits into
google:mainfrom
raulEspaa:fix/inmemory-memory-combining-marks
Open

raulEspaa wants to merge 2 commits into
google:mainfrom
raulEspaa:fix/inmemory-memory-combining-marks

Conversation

@raulEspaa

Copy link
Copy Markdown

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

Problem:
InMemoryMemoryService extracted words with re.findall(r'\w+', ...). \w doesn't match combining marks, so Thai and Devanagari words were split at their vowel and tone marks (ข้าว became ข and าว). Non-ASCII query words are also matched as substrings of the event text, so those pieces matched unrelated memories: a search for ข้าว returned ขอบคุณครับ, and a search for हिंदी returned a sentence about Hyderabad.

Solution:
Tokenize the same way SqliteMemoryService does (_is_token_char / _extract_tokens): combining marks count as word characters, and a mark with no letter before it is dropped, so the emoji variation selector U+FE0F doesn't become a word. Both _extract_words_lower and _extract_searchable_words now use a small _find_words helper instead of \w+. When a word is split into Latin and non-Latin runs, a run that is only a combining mark is dropped too.

re is no longer used in the module, so I removed the import. Text without combining marks gives the same words as before.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

I added cases to test_search_memory_non_latin in tests/unittests/memory/test_in_memory_memory_service.py: Thai and Hindi queries that should and shouldn't match, and an emoji with a variation selector. The two "shouldn't match" cases fail without the fix and pass with it.

$ pytest tests/unittests/memory/test_in_memory_memory_service.py
69 passed in 12.25s

$ pytest tests/unittests/memory
235 passed, 10 warnings in 7.65s

pre-commit run --files passes on the changed files, and mypy reports no issues for the changed module.

Manual End-to-End (E2E) Tests:

I ran the reproduction script from #8168, which stores four Thai and Hindi sentences in InMemoryMemoryService and searches for ข้าว and हिंदी.

Before:

ข้าว ['ข้าวผัดอร่อยมาก', 'ขอบคุณครับ']
हिंदी ['मुझे हिंदी पसंद है', 'मैं हैदराबाद में रहता हूँ']

After (the same results SqliteMemoryService gives for this data):

ข้าว ['ข้าวผัดอร่อยมาก']
हिंदी ['मुझे हिंदी पसंद है']

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

#8141 also changes in_memory_memory_service.py, but in a different part of the file (rewind handling), so the two changes shouldn't conflict.

…earch

The in-memory service extracted words with \w+, which does not match
combining marks, so Thai and Devanagari words were split at their vowel
and tone marks. The pieces were then matched as substrings, and a search
for ข้าว also returned unrelated Thai text.

Tokenize the same way SqliteMemoryService does: combining marks are word
characters, and a mark with no letter before it is dropped.

Fixes google#8168
…arch

Adds cases to test_search_memory_non_latin for Bengali and Tamil vowel
signs and virama, a Latin word inside Thai text, and the keycap emoji
1 + U+FE0F + U+20E3. The Bengali, Tamil and Latin+Thai negative cases
fail without the combining-mark fix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

InMemoryMemoryService search matches unrelated Thai and Hindi text because \w splits words at combining marks

2 participants