Skip to content

fix(memory): keep zero-width joiners inside words in memory search - #8182

Open
raulEspaa wants to merge 3 commits into
google:mainfrom
raulEspaa:fix/memory-zwj-zwnj
Open

raulEspaa wants to merge 3 commits into
google:mainfrom
raulEspaa:fix/memory-zwj-zwnj

Conversation

@raulEspaa

@raulEspaa raulEspaa commented Nov 16, 2026 •

Copy link
Copy Markdown

Fixes #8181. Follow-up to #8168 / #8169, and stacked on #8169: until that is merged, the diff here also shows its two commits.

Problem

InMemoryMemoryService and SqliteMemoryService split a word at a zero-width joiner (ZWJ, U+200D) or non-joiner (ZWNJ, U+200C). These are format characters (category Cf), not marks, so the mark-aware tokenizers don't keep them in a word. Bengali, Hindi and Persian use them inside words, and the pieces then match unrelated text: a search for Bengali র‍্যাব becomes the tokens র and যাব, and র also matches রাজ.

Change

  • _is_word_char (in-memory) and _is_token_char (SQLite) also accept ZWJ and ZWNJ.
  • A joiner at the start or end of a word is dropped, the same way a leading mark already is, so the ZWJs of an emoji sequence such as 👨‍👩‍👧 don't become a token that matches every emoji sequence.
  • The trimming lives in _trim_word (in-memory, used by _find_words and the script-run split in _extract_searchable_words) and _trim_token (SQLite), next to the existing mark handling.

Behaviour

Same data, before and after, for both services and both SQLite modes:

Query Before After
Bengali র‍্যাব (ZWJ) also রাজ only র‍্যাব
Hindi क्‌ष (ZWNJ) also क्षमा only क्‌ष
Persian می‌خواهم (ZWNJ) in-memory and LIKE: also میخواهم (no ZWNJ) only می‌خواهم, in every mode
Persian کتاب‌ها ("books", ZWNJ) also مرد‌ها ("men"), via the shared suffix ها only کتاب‌ها
Persian خواهم, می unchanged unchanged
emoji 👨‍👩‍👧, Family unchanged unchanged

Trade-off: in the in-memory service and in SQLite's LIKE mode, a word written with a joiner no longer matches the same word written without it or with a space (می‌خواهم vs میخواهم or می خواهم). Before, those matched only through the fragments the joiner left (می, خواهم), the same mechanism that produced the false matches above. FTS mode still matches the spaced form, and র‍্যাব with র্যাব, because unicode61 splits the indexed text at the joiner (and at Bengali marks) and the query word is matched as a phrase. It didn't match میخواهم before this change either. If you'd rather have joiner-insensitive matching in the other two paths too, I can strip the joiners before comparing instead (with a REPLACE on search_text for LIKE), but that also makes क्‌ष match क्षमा in the in-memory service, through its substring matching for non-ASCII words.

Testing

  • Added ZWJ/ZWNJ cases (Bengali, Hindi, Persian, emoji, a joiner at the start or end of a query word) to test_search_memory_non_latin, and a new test_search_keeps_joiners_in_query_word for SQLite in both FTS and LIKE modes.
  • pytest tests/unittests/memory: 266 passed. Without the change, the 9 Bengali, Hindi and Persian negative cases fail. Dropping only the leading-joiner trim or only the trailing-joiner trim fails 2 cases each.
  • pyink, isort and mypy are clean.

…earch

The in-memory service extracted words with \w+, which does not match
combining marks, so Thai and Devanagari words were split at their vowel
and tone marks. The pieces were then matched as substrings, and a search
for ข้าว also returned unrelated Thai text.

Tokenize the same way SqliteMemoryService does: combining marks are word
characters, and a mark with no letter before it is dropped.

Fixes google#8168
…arch

Adds cases to test_search_memory_non_latin for Bengali and Tamil vowel
signs and virama, a Latin word inside Thai text, and the keycap emoji
1 + U+FE0F + U+20E3. The Bengali, Tamil and Latin+Thai negative cases
fail without the combining-mark fix.
Both memory services split a word at a zero-width joiner (U+200D) or
non-joiner (U+200C), because they are format characters and not marks.
Bengali, Hindi and Persian use them inside words, so a search for the
Bengali র‍্যাব became the tokens র and যাব and also returned unrelated
text such as রাজ.

Treat the joiners as word characters in both tokenizers. A joiner at the
start or end of a word is dropped, like a leading mark, so the joiners of
an emoji sequence are not words of their own.

Fixes google#8181
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Memory search splits words at ZWJ/ZWNJ and matches unrelated Bengali and Hindi text

2 participants