Skip to content

InMemoryMemoryService search matches unrelated Thai and Hindi text because \w splits words at combining marks #8168

Description

@raulEspaa

🔴 Required Information

Describe the Bug:
InMemoryMemoryService.search_memory returns memories that don't contain the query word when the text uses combining marks: Thai vowel and tone marks, or Devanagari vowel signs (Hindi). A search for ข้าว ("rice") also returns ขอบคุณครับ ("thank you"), and a search for हिंदी ("Hindi") also returns a sentence about Hyderabad.

The words are extracted with re.findall(r'\w+', ...), and \w doesn't match combining marks, so ข้าว is split into ข and าว:

def _extract_words_lower(text: str) -> set[str]:
"""Extracts Unicode-aware tokens from a string in lowercase."""
text = unicodedata.normalize('NFC', text)
return set(word.lower() for word in re.findall(r'\w+', text))

for word in re.findall(r'\w+', unicodedata.normalize('NFC', text)):

Non-ASCII query words are also matched as substrings of the event text, so a piece like ข is found inside unrelated words. SqliteMemoryService already treats combining marks as part of a word, with a comment about this exact example:

# Combining marks count as word characters: \w alone would split a word at
# them, such as the Thai vowel and tone marks in ข้าว.
def _is_token_char(c: str) -> bool:
return c.isalnum() or c in "_%" or _is_mark(c)

The in-memory service doesn't have that change, so the two services return different results for the same data.

Steps to Reproduce:

  1. Install google-adk 2.14.1, or use main at d05e482.
  2. Run the script under "Minimal Reproduction Code". It stores four short Thai and Hindi sentences and searches for ข้าว and हिंदी.

Expected Behavior:
Only the sentence that contains the word, which is what SqliteMemoryService returns for the same data:

ข้าว ['ข้าวผัดอร่อยมาก']
हिंदी ['मुझे हिंदी पसंद है']

Observed Behavior:

ข้าว ['ข้าวผัดอร่อยมาก', 'ขอบคุณครับ']
हिंदी ['मुझे हिंदी पसंद है', 'मैं हैदराबाद में रहता हूँ']

Environment Details:

  • ADK Library Version (pip show google-adk): 2.14.1 (main at d05e482)
  • Desktop OS: Linux
  • Python Version (python -V): 3.13.16

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: N/A, no model is called.

🟡 Optional Information

Regression:
Not that I know of. I only tried 2.14.1 and main.

Additional Context:
The fix I have in mind is to tokenize the same way the SQLite service does: treat combining marks as word characters, and drop a mark that has no letter before it, such as the emoji variation selector U+FE0F.

I'd like to fix this myself. I have the change and the tests ready, and I'll open a PR that links to this issue.

Minimal Reproduction Code:

import asyncio

from google.adk.events.event import Event
from google.adk.memory import InMemoryMemoryService
from google.adk.sessions.session import Session
from google.genai import types

TEXTS = [
    "ข้าวผัดอร่อยมาก",  # "The fried rice is delicious"
    "ขอบคุณครับ",  # "Thank you"
    "मुझे हिंदी पसंद है",  # "I like Hindi"
    "मैं हैदराबाद में रहता हूँ",  # "I live in Hyderabad"
]


async def main():
  service = InMemoryMemoryService()
  session = Session(
      app_name="app",
      user_id="user",
      id="session",
      last_update_time=0,
      events=[
          Event(
              id=f"event-{i}",
              invocation_id="inv",
              author="user",
              timestamp=i,
              content=types.Content(parts=[types.Part(text=text)]),
          )
          for i, text in enumerate(TEXTS)
      ],
  )
  await service.add_session_to_memory(session)
  for query in ["ข้าว", "हिंदी"]:  # "rice", "Hindi"
    result = await service.search_memory(
        app_name="app", user_id="user", query=query
    )
    print(query, [m.content.parts[0].text for m in result.memories])


asyncio.run(main())

How often has this issue occurred?:

  • Always (100%)

Activity

  1. raulEspaa commented on Nov 15, 2026

    @raulEspaa
    Author

    I've opened #8169 with the fix and tests.

  2. surajksharma07 commented on Nov 16, 2026

    @surajksharma07
    Collaborator

    @raulEspaa Reproduced this on 2.14.1 and current main: ข้าว and हिंदी both pull in the unrelated sentence because \w+ drops the combining marks.

    As a workaround, swapping the tokenizer in _extract_words_lower / extract_searchable_words for the same mark-aware grouping SqliteMemoryService uses (isalnum() or '' or category M*, dropping a leading orphan mark) gave the expected results for me. The existing in-memory memory tests still pass.

    That looks like the same approach as #8169. Could you confirm it also holds up on your side for a few more scripts (e.g. Bengali/Tamil, emoji with U+FE0F, mixed Latin+Thai) before we move the PR to review?

  3. raulEspaa commented on Nov 16, 2026

    @raulEspaa
    Author

    @surajksharma07 Thanks for reproducing it. I ran the cases you mentioned on the PR branch, comparing InMemoryMemoryService before the fix (d05e482), with the fix, and SqliteMemoryService on the same data:

    Case Query Before the fix With #8169 SqliteMemoryService
    Bengali ভাত ("rice") also 2 unrelated sentences (ভালো…) only আমি ভাত খাই same as #8169
    Bengali বাংলা ("Bangla") also ভালো আছি only আমি বাংলা ভালোবাসি same as #8169
    Tamil தமிழ் ("Tamil") also an unrelated sentence only தமிழ் எனக்கு பிடிக்கும் same as #8169
    Tamil சாதம் ("rice") also an unrelated sentence only நான் சாதம் சாப்பிடுகிறேன் same as #8169
    Emoji Dogs ❤️ only Dogs are great same same
    Emoji ❤️, 👍🏽 alone nothing nothing nothing
    Emoji keycap 1️⃣ (1 + U+FE0F + U+20E3) only Room 1️⃣ is free same same
    Latin + Thai Python on ฉันใช้Pythonเขียนโปรแกรม matches matches matches
    Latin + Thai thon nothing nothing nothing
    Latin + Thai ข้าว on ฉันใช้Pythonเขียนโปรแกรม matches (via the ข in เขียน) no match no match

    So the fix holds for Bengali and Tamil, and the emoji and mixed Latin+Thai cases behave the same as before or better.

    Two things worth flagging:

    1. A query that mixes Latin and Thai in one word, e.g. Pythonข้าว, no longer finds ข้าวผัดอร่อยมาก. Before the fix it found it only by accident, through the broken fragment าว. With the fix, Thai behaves the way Japanese already does on main: Python寿司 doesn't find 寿司が好きです either, because runs split out of a query word only match whole words. SqliteMemoryService finds both, so this is an existing difference between the two services, not something the PR introduces. I've left it as is.
    2. ZWJ/ZWNJ (U+200D/U+200C) still split words, in all three versions. Examples are Bengali র‍্যাব and Hindi क्‌ष: the query also returns a sentence that only shares the first letter. These are format characters (category Cf), not marks, so neither tokenizer keeps them in a word. Fixing it would need the same change in SqliteMemoryService to keep the two consistent, so I'd rather handle it in a separate issue and PR, if you think it's worth it.

    I've added the Bengali, Tamil, Latin+Thai and keycap cases to test_search_memory_non_latin. pytest tests/unittests/memory gives 244 passed. Run against the code before the fix, the new Bengali, Tamil and Latin+Thai negative cases fail, as do the existing Thai and Hindi ones. pyink, isort and mypy are clean.

  4. surajksharma07 commented on Nov 16, 2026

    @surajksharma07
    Collaborator

    Thanks for running those @raulEspaa. Re-checked on my side and got the same: ভাত, தமிழ் and சாதம் now only return their own sentence, and ข้าว no longer hits via the ข in เขียน.

    Leaving the ข้าว change as is makes sense, since it now lines up with how 寿司 already behaves on main. Agreed that ZWJ/ZWNJ (র‍্যাব still matches রবি) belongs in a separate issue, since SqliteMemoryService needs the same change.

    One thing on #8169: cla/google started failing after 90e3df2, which looks like it’s coming from the Co-authored-by trailer on that commit. Could you amend it to drop the trailer and force-push so the CLA check re-runs?

    Once that’s green it should be ready for a maintainer to approve the workflows and review.

  5. added a commit that references this issue on Nov 16, 2026
    b631d7e
  6. raulEspaa commented on Nov 16, 2026

    @raulEspaa
    Author

    @surajksharma07 Thanks for checking and for the pointer on the CLA. I dropped the trailer and fixed the author email on both commits, then force-pushed. The code is unchanged and cla/google is green now on #8169.

    I'll open a separate issue for ZWJ/ZWNJ.

  7. surajksharma07 commented on Nov 16, 2026

    @surajksharma07
    Collaborator

    @raulEspaa thanks for sorting that out so quickly. cla/google is green on #8169 now and the diff is the same as before, so from my side it’s ready for a maintainer to approve the workflows and review.

    Also thanks for splitting out #8181. I tried the joiner-inside-word approach from #8182 locally and got the same table: র‍্যাব, क्‌ष and کتاب‌ها only return their own sentence, and the 👨‍👩‍👧 / Family cases don’t change.

    The می‌خواهم vs میخواهم trade-off you called out shows up for me too. I’ll leave that call (strict vs joiner-insensitive) to the reviewers on #8182, and let’s keep that discussion over on #8181.

    Since #8182 is stacked, a quick rebase once #8169 lands will keep its diff down to just the ZWJ/ZWNJ commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions