Repository navigation
Conversation
…earch The in-memory service extracted words with \w+, which does not match combining marks, so Thai and Devanagari words were split at their vowel and tone marks. The pieces were then matched as substrings, and a search for ข้าว also returned unrelated Thai text. Tokenize the same way SqliteMemoryService does: combining marks are word characters, and a mark with no letter before it is dropped. Fixes google#8168
…arch Adds cases to test_search_memory_non_latin for Bengali and Tamil vowel signs and virama, a Latin word inside Thai text, and the keycap emoji 1 + U+FE0F + U+20E3. The Bengali, Tamil and Latin+Thai negative cases fail without the combining-mark fix.
Both memory services split a word at a zero-width joiner (U+200D) or non-joiner (U+200C), because they are format characters and not marks. Bengali, Hindi and Persian use them inside words, so a search for the Bengali র্যাব became the tokens র and যাব and also returned unrelated text such as রাজ. Treat the joiners as word characters in both tokenizers. A joiner at the start or end of a word is dropped, like a leading mark, so the joiners of an emoji sequence are not words of their own. Fixes google#8181
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #8181. Follow-up to #8168 / #8169, and stacked on #8169: until that is merged, the diff here also shows its two commits.
Problem
InMemoryMemoryServiceandSqliteMemoryServicesplit a word at a zero-width joiner (ZWJ, U+200D) or non-joiner (ZWNJ, U+200C). These are format characters (category Cf), not marks, so the mark-aware tokenizers don't keep them in a word. Bengali, Hindi and Persian use them inside words, and the pieces then match unrelated text: a search for Bengali র্যাব becomes the tokensরandযাব, andরalso matches রাজ.Change
_is_word_char(in-memory) and_is_token_char(SQLite) also accept ZWJ and ZWNJ._trim_word(in-memory, used by_find_wordsand the script-run split in_extract_searchable_words) and_trim_token(SQLite), next to the existing mark handling.Behaviour
Same data, before and after, for both services and both SQLite modes:
FamilyTrade-off: in the in-memory service and in SQLite's LIKE mode, a word written with a joiner no longer matches the same word written without it or with a space (میخواهم vs میخواهم or می خواهم). Before, those matched only through the fragments the joiner left (
می,خواهم), the same mechanism that produced the false matches above. FTS mode still matches the spaced form, and র্যাব with র্যাব, becauseunicode61splits the indexed text at the joiner (and at Bengali marks) and the query word is matched as a phrase. It didn't match میخواهم before this change either. If you'd rather have joiner-insensitive matching in the other two paths too, I can strip the joiners before comparing instead (with aREPLACEonsearch_textfor LIKE), but that also makes क्ष match क्षमा in the in-memory service, through its substring matching for non-ASCII words.Testing
test_search_memory_non_latin, and a newtest_search_keeps_joiners_in_query_wordfor SQLite in both FTS and LIKE modes.pytest tests/unittests/memory: 266 passed. Without the change, the 9 Bengali, Hindi and Persian negative cases fail. Dropping only the leading-joiner trim or only the trailing-joiner trim fails 2 cases each.