Repository navigation
Description
🔴 Required Information
Describe the Bug:
InMemoryMemoryService.search_memory returns memories that don't contain the query word when the text uses combining marks: Thai vowel and tone marks, or Devanagari vowel signs (Hindi). A search for ข้าว ("rice") also returns ขอบคุณครับ ("thank you"), and a search for हिंदी ("Hindi") also returns a sentence about Hyderabad.
The words are extracted with re.findall(r'\w+', ...), and \w doesn't match combining marks, so ข้าว is split into ข and าว:
adk-python/src/google/adk/memory/in_memory_memory_service.py
Lines 49 to 52 in d05e482
| def _extract_words_lower(text: str) -> set[str]: | |
| """Extracts Unicode-aware tokens from a string in lowercase.""" | |
| text = unicodedata.normalize('NFC', text) | |
| return set(word.lower() for word in re.findall(r'\w+', text)) |
| for word in re.findall(r'\w+', unicodedata.normalize('NFC', text)): |
Non-ASCII query words are also matched as substrings of the event text, so a piece like ข is found inside unrelated words. SqliteMemoryService already treats combining marks as part of a word, with a comment about this exact example:
adk-python/src/google/adk/memory/_sqlite_memory_service.py
Lines 595 to 598 in d05e482
| # Combining marks count as word characters: \w alone would split a word at | |
| # them, such as the Thai vowel and tone marks in ข้าว. | |
| def _is_token_char(c: str) -> bool: | |
| return c.isalnum() or c in "_%" or _is_mark(c) |
The in-memory service doesn't have that change, so the two services return different results for the same data.
Steps to Reproduce:
- Install google-adk 2.14.1, or use main at d05e482.
- Run the script under "Minimal Reproduction Code". It stores four short Thai and Hindi sentences and searches for ข้าว and हिंदी.
Expected Behavior:
Only the sentence that contains the word, which is what SqliteMemoryService returns for the same data:
ข้าว ['ข้าวผัดอร่อยมาก']
हिंदी ['मुझे हिंदी पसंद है']
Observed Behavior:
ข้าว ['ข้าวผัดอร่อยมาก', 'ขอบคุณครับ']
हिंदी ['मुझे हिंदी पसंद है', 'मैं हैदराबाद में रहता हूँ']
Environment Details:
- ADK Library Version (pip show google-adk): 2.14.1 (main at d05e482)
- Desktop OS: Linux
- Python Version (python -V): 3.13.16
Model Information:
- Are you using LiteLLM: No
- Which model is being used: N/A, no model is called.
🟡 Optional Information
Regression:
Not that I know of. I only tried 2.14.1 and main.
Additional Context:
The fix I have in mind is to tokenize the same way the SQLite service does: treat combining marks as word characters, and drop a mark that has no letter before it, such as the emoji variation selector U+FE0F.
I'd like to fix this myself. I have the change and the tests ready, and I'll open a PR that links to this issue.
Minimal Reproduction Code:
import asyncio
from google.adk.events.event import Event
from google.adk.memory import InMemoryMemoryService
from google.adk.sessions.session import Session
from google.genai import types
TEXTS = [
"ข้าวผัดอร่อยมาก", # "The fried rice is delicious"
"ขอบคุณครับ", # "Thank you"
"मुझे हिंदी पसंद है", # "I like Hindi"
"मैं हैदराबाद में रहता हूँ", # "I live in Hyderabad"
]
async def main():
service = InMemoryMemoryService()
session = Session(
app_name="app",
user_id="user",
id="session",
last_update_time=0,
events=[
Event(
id=f"event-{i}",
invocation_id="inv",
author="user",
timestamp=i,
content=types.Content(parts=[types.Part(text=text)]),
)
for i, text in enumerate(TEXTS)
],
)
await service.add_session_to_memory(session)
for query in ["ข้าว", "हिंदी"]: # "rice", "Hindi"
result = await service.search_memory(
app_name="app", user_id="user", query=query
)
print(query, [m.content.parts[0].text for m in result.memories])
asyncio.run(main())How often has this issue occurred?:
- Always (100%)
Activity
raulEspaa commentedon Nov 15, 2026
I've opened #8169 with the fix and tests.
surajksharma07 commentedon Nov 16, 2026
@raulEspaa Reproduced this on 2.14.1 and current main: ข้าว and हिंदी both pull in the unrelated sentence because \w+ drops the combining marks.
As a workaround, swapping the tokenizer in _extract_words_lower / extract_searchable_words for the same mark-aware grouping SqliteMemoryService uses (isalnum() or '' or category M*, dropping a leading orphan mark) gave the expected results for me. The existing in-memory memory tests still pass.
That looks like the same approach as #8169. Could you confirm it also holds up on your side for a few more scripts (e.g. Bengali/Tamil, emoji with U+FE0F, mixed Latin+Thai) before we move the PR to review?
raulEspaa commentedon Nov 16, 2026
@surajksharma07 Thanks for reproducing it. I ran the cases you mentioned on the PR branch, comparing
InMemoryMemoryServicebefore the fix (d05e482), with the fix, andSqliteMemoryServiceon the same data:আমি ভাত খাইভালো আছিআমি বাংলা ভালোবাসিதமிழ் எனக்கு பிடிக்கும்நான் சாதம் சாப்பிடுகிறேன்Dogs ❤️Dogs are great❤️,👍🏽alone1️⃣(1 + U+FE0F + U+20E3)Room 1️⃣ is freePythononฉันใช้Pythonเขียนโปรแกรมthonฉันใช้PythonเขียนโปรแกรมSo the fix holds for Bengali and Tamil, and the emoji and mixed Latin+Thai cases behave the same as before or better.
Two things worth flagging:
Pythonข้าว, no longer findsข้าวผัดอร่อยมาก. Before the fix it found it only by accident, through the broken fragmentาว. With the fix, Thai behaves the way Japanese already does on main:Python寿司doesn't find寿司が好きですeither, because runs split out of a query word only match whole words.SqliteMemoryServicefinds both, so this is an existing difference between the two services, not something the PR introduces. I've left it as is.SqliteMemoryServiceto keep the two consistent, so I'd rather handle it in a separate issue and PR, if you think it's worth it.I've added the Bengali, Tamil, Latin+Thai and keycap cases to
test_search_memory_non_latin.pytest tests/unittests/memorygives 244 passed. Run against the code before the fix, the new Bengali, Tamil and Latin+Thai negative cases fail, as do the existing Thai and Hindi ones. pyink, isort and mypy are clean.surajksharma07 commentedon Nov 16, 2026
Thanks for running those @raulEspaa. Re-checked on my side and got the same: ভাত, தமிழ் and சாதம் now only return their own sentence, and ข้าว no longer hits via the ข in เขียน.
Leaving the ข้าว change as is makes sense, since it now lines up with how 寿司 already behaves on main. Agreed that ZWJ/ZWNJ (র্যাব still matches রবি) belongs in a separate issue, since SqliteMemoryService needs the same change.
One thing on #8169: cla/google started failing after 90e3df2, which looks like it’s coming from the Co-authored-by trailer on that commit. Could you amend it to drop the trailer and force-push so the CLA check re-runs?
Once that’s green it should be ready for a maintainer to approve the workflows and review.
fix(memory): keep combining marks in words in InMemoryMemoryService s…
raulEspaa commentedon Nov 16, 2026
@surajksharma07 Thanks for checking and for the pointer on the CLA. I dropped the trailer and fixed the author email on both commits, then force-pushed. The code is unchanged and cla/google is green now on #8169.
I'll open a separate issue for ZWJ/ZWNJ.
surajksharma07 commentedon Nov 16, 2026
@raulEspaa thanks for sorting that out so quickly. cla/google is green on #8169 now and the diff is the same as before, so from my side it’s ready for a maintainer to approve the workflows and review.
Also thanks for splitting out #8181. I tried the joiner-inside-word approach from #8182 locally and got the same table: র্যাব, क्ष and کتابها only return their own sentence, and the 👨👩👧 / Family cases don’t change.
The میخواهم vs میخواهم trade-off you called out shows up for me too. I’ll leave that call (strict vs joiner-insensitive) to the reviewers on #8182, and let’s keep that discussion over on #8181.
Since #8182 is stacked, a quick rebase once #8169 lands will keep its diff down to just the ZWJ/ZWNJ commit.