Repository navigation
Description
🔴 Required Information
Describe the Bug:
Memory search splits a word at a zero-width joiner (ZWJ, U+200D) or zero-width non-joiner (ZWNJ, U+200C), so it also returns text that only shares the first letter of the word. Both SqliteMemoryService and InMemoryMemoryService do this, including with the combining-mark fix in #8169.
Some Indic words use these characters to pick a glyph form. In Bengali, র্যাব (RAB) writes the ra-phala with a ZWJ. In Hindi, क्ष uses a ZWNJ to show the half form instead of the ligature क्ष. A search for র্যাব also returns আমার নাম রাজ ("My name is Raj"), and a search for क्ष also returns मैं क्षमा माँगता हूँ ("I apologise").
ZWJ and ZWNJ are format characters (category Cf), not marks, so the tokenizer does not treat them as word characters:
adk-python/src/google/adk/memory/_sqlite_memory_service.py
Lines 595 to 613 in c6586f1
| # Combining marks count as word characters: \w alone would split a word at | |
| # them, such as the Thai vowel and tone marks in ข้าว. | |
| def _is_token_char(c: str) -> bool: | |
| return c.isalnum() or c in "_%" or _is_mark(c) | |
| def _extract_tokens(query: str) -> list[str]: | |
| normalized = unicodedata.normalize("NFC", query) | |
| normalized = _split_script_runs(normalized) | |
| seen = set() | |
| tokens = [] | |
| for is_token, group in itertools.groupby(normalized, _is_token_char): | |
| if not is_token: | |
| continue | |
| # Marks with no letter before them are dropped: the emoji variation | |
| # selector U+FE0F is a mark, and as a token it would match any emoji. | |
| lowered = "".join(itertools.dropwhile(_is_mark, group)).lower() | |
| if lowered and lowered not in seen: | |
| seen.add(lowered) |
Both services turn র্যাব into the tokens র and যাব (the virama left at the start of the second piece is dropped as an orphan mark), and র alone matches রাজ. क्ष becomes क् and ष, and क् matches क्षमा.
This came up in #8168, where we agreed to track it separately because both services need the same change to stay consistent.
Steps to Reproduce:
- Install google-adk 2.14.1, or use main at c6586f1.
- Run the script under "Minimal Reproduction Code".
Expected Behavior:
Each query returns only the sentence that contains the word:
InMemory 'র্যাব' ['র্যাব এসেছে']
InMemory 'क्ष' ['मैं क्ष लिखता हूँ']
Sqlite 'র্যাব' ['র্যাব এসেছে']
Sqlite 'क्ष' ['मैं क्ष लिखता हूँ']
Observed Behavior:
InMemory 'র্যাব' ['র্যাব এসেছে', 'আমার নাম রাজ']
InMemory 'क्ष' ['मैं क्ष लिखता हूँ', 'मैं क्षमा माँगता हूँ']
Sqlite 'র্যাব' ['র্যাব এসেছে', 'আমার নাম রাজ']
Sqlite 'क्ष' ['मैं क्ष लिखता हूँ', 'मैं क्षमा माँगता हूँ']
The result is the same on main at c6586f1 and on the #8169 branch.
Environment Details:
- ADK Library Version (pip show google-adk): 2.14.1 (main at c6586f1)
- Desktop OS: Linux
- Python Version (python -V): 3.13.16
Model Information:
- Are you using LiteLLM: No
- Which model is being used: N/A, no model is called.
🟡 Optional Information
Regression:
No. Both services have always behaved this way.
Additional Context:
One option is to treat ZWJ and ZWNJ as word characters when they are between two word characters, in both tokenizers. A ZWJ at the start or end of a run would still be dropped, so emoji ZWJ sequences such as 👨👩👧 do not turn into a ZWJ-only token that matches every emoji sequence.
I'm happy to send a PR for both services once #8169 is merged, if this approach sounds right.
Minimal Reproduction Code:
import asyncio
import tempfile
from google.adk.events.event import Event
from google.adk.memory import InMemoryMemoryService
from google.adk.memory import SqliteMemoryService
from google.adk.sessions.session import Session
from google.genai import types
ZWJ, ZWNJ = "", ""
TEXTS = [
f"র{ZWJ}্যাব এসেছে", # Bengali: "RAB has come" (ra-phala written with ZWJ)
"আমার নাম রাজ", # Bengali: "My name is Raj"
f"मैं क्{ZWNJ}ष लिखता हूँ", # Hindi: "I write क्ष" (half form with ZWNJ)
"मैं क्षमा माँगता हूँ", # Hindi: "I apologise"
]
QUERIES = [f"র{ZWJ}্যাব", f"क्{ZWNJ}ष"]
async def main():
session = Session(
app_name="app",
user_id="user",
id="session",
last_update_time=0,
events=[
Event(
id=f"event-{i}",
invocation_id="inv",
author="user",
timestamp=i,
content=types.Content(parts=[types.Part(text=text)]),
)
for i, text in enumerate(TEXTS)
],
)
services = {
"InMemory": InMemoryMemoryService(),
"Sqlite": SqliteMemoryService(f"{tempfile.mkdtemp()}/memory.db"),
}
for name, service in services.items():
await service.add_session_to_memory(session)
for query in QUERIES:
result = await service.search_memory(
app_name="app", user_id="user", query=query
)
print(name, ascii(query), [m.content.parts[0].text for m in result.memories])
await services["Sqlite"].close()
asyncio.run(main())How often has this issue occurred?:
- Always (100%)
Activity
raulEspaa commentedon Nov 16, 2026
On strict vs joiner-insensitive matching: I tried the second option on top of #8169, removing U+200C/U+200D from both the query and the stored text before tokenizing, and compared it with #8182 on the same data in all three paths (in-memory, SQLite FTS and SQLite LIKE).
So removing the joiners keeps the false matches from #8181 out and makes the spellings with and without a joiner find each other, consistently across the three paths. The spaced form (می خواهم) isn't matched in either version, except by FTS in the strict one. The क्षमा match comes from the substring matching for non-ASCII words, and a plain क्ष query already does that today.
This is also what NFKC_Casefold does, since ZWJ and ZWNJ are default-ignorable. My preference would be the joiner-insensitive version, but I'm fine either way. One thing to keep in mind for SQLite: if the folding is done when the text is stored, rows written before the change still contain the joiners, so they'd either need reindexing or a
REPLACEonsearch_textin the LIKE path.I'll leave #8182 as it is until a reviewer picks one.