Skip to content

Memory search splits words at ZWJ/ZWNJ and matches unrelated Bengali and Hindi text #8181

Description

@raulEspaa

🔴 Required Information

Describe the Bug:
Memory search splits a word at a zero-width joiner (ZWJ, U+200D) or zero-width non-joiner (ZWNJ, U+200C), so it also returns text that only shares the first letter of the word. Both SqliteMemoryService and InMemoryMemoryService do this, including with the combining-mark fix in #8169.

Some Indic words use these characters to pick a glyph form. In Bengali, র‍্যাব (RAB) writes the ra-phala with a ZWJ. In Hindi, क्‌ष uses a ZWNJ to show the half form instead of the ligature क्ष. A search for র‍্যাব also returns আমার নাম রাজ ("My name is Raj"), and a search for क्‌ष also returns मैं क्षमा माँगता हूँ ("I apologise").

ZWJ and ZWNJ are format characters (category Cf), not marks, so the tokenizer does not treat them as word characters:

# Combining marks count as word characters: \w alone would split a word at
# them, such as the Thai vowel and tone marks in ข้าว.
def _is_token_char(c: str) -> bool:
return c.isalnum() or c in "_%" or _is_mark(c)
def _extract_tokens(query: str) -> list[str]:
normalized = unicodedata.normalize("NFC", query)
normalized = _split_script_runs(normalized)
seen = set()
tokens = []
for is_token, group in itertools.groupby(normalized, _is_token_char):
if not is_token:
continue
# Marks with no letter before them are dropped: the emoji variation
# selector U+FE0F is a mark, and as a token it would match any emoji.
lowered = "".join(itertools.dropwhile(_is_mark, group)).lower()
if lowered and lowered not in seen:
seen.add(lowered)

Both services turn র‍্যাব into the tokens র and যাব (the virama left at the start of the second piece is dropped as an orphan mark), and র alone matches রাজ. क्‌ष becomes क् and ष, and क् matches क्षमा.

This came up in #8168, where we agreed to track it separately because both services need the same change to stay consistent.

Steps to Reproduce:

  1. Install google-adk 2.14.1, or use main at c6586f1.
  2. Run the script under "Minimal Reproduction Code".

Expected Behavior:
Each query returns only the sentence that contains the word:

InMemory 'র‍্যাব' ['র‍্যাব এসেছে']
InMemory 'क्‌ष' ['मैं क्‌ष लिखता हूँ']
Sqlite 'র‍্যাব' ['র‍্যাব এসেছে']
Sqlite 'क्‌ष' ['मैं क्‌ष लिखता हूँ']

Observed Behavior:

InMemory 'র‍্যাব' ['র‍্যাব এসেছে', 'আমার নাম রাজ']
InMemory 'क्‌ष' ['मैं क्‌ष लिखता हूँ', 'मैं क्षमा माँगता हूँ']
Sqlite 'র‍্যাব' ['র‍্যাব এসেছে', 'আমার নাম রাজ']
Sqlite 'क्‌ष' ['मैं क्‌ष लिखता हूँ', 'मैं क्षमा माँगता हूँ']

The result is the same on main at c6586f1 and on the #8169 branch.

Environment Details:

  • ADK Library Version (pip show google-adk): 2.14.1 (main at c6586f1)
  • Desktop OS: Linux
  • Python Version (python -V): 3.13.16

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: N/A, no model is called.

🟡 Optional Information

Regression:
No. Both services have always behaved this way.

Additional Context:
One option is to treat ZWJ and ZWNJ as word characters when they are between two word characters, in both tokenizers. A ZWJ at the start or end of a run would still be dropped, so emoji ZWJ sequences such as 👨‍👩‍👧 do not turn into a ZWJ-only token that matches every emoji sequence.

I'm happy to send a PR for both services once #8169 is merged, if this approach sounds right.

Minimal Reproduction Code:

import asyncio
import tempfile

from google.adk.events.event import Event
from google.adk.memory import InMemoryMemoryService
from google.adk.memory import SqliteMemoryService
from google.adk.sessions.session import Session
from google.genai import types

ZWJ, ZWNJ = "‍", "‌"
TEXTS = [
    f"র{ZWJ}্যাব এসেছে",  # Bengali: "RAB has come" (ra-phala written with ZWJ)
    "আমার নাম রাজ",  # Bengali: "My name is Raj"
    f"मैं क्{ZWNJ}ष लिखता हूँ",  # Hindi: "I write क्‌ष" (half form with ZWNJ)
    "मैं क्षमा माँगता हूँ",  # Hindi: "I apologise"
]
QUERIES = [f"র{ZWJ}্যাব", f"क्{ZWNJ}ष"]


async def main():
  session = Session(
      app_name="app",
      user_id="user",
      id="session",
      last_update_time=0,
      events=[
          Event(
              id=f"event-{i}",
              invocation_id="inv",
              author="user",
              timestamp=i,
              content=types.Content(parts=[types.Part(text=text)]),
          )
          for i, text in enumerate(TEXTS)
      ],
  )
  services = {
      "InMemory": InMemoryMemoryService(),
      "Sqlite": SqliteMemoryService(f"{tempfile.mkdtemp()}/memory.db"),
  }
  for name, service in services.items():
    await service.add_session_to_memory(session)
    for query in QUERIES:
      result = await service.search_memory(
          app_name="app", user_id="user", query=query
      )
      print(name, ascii(query), [m.content.parts[0].text for m in result.memories])
  await services["Sqlite"].close()


asyncio.run(main())

How often has this issue occurred?:

  • Always (100%)

Activity

  1. raulEspaa commented on Nov 16, 2026

    @raulEspaa
    Author

    On strict vs joiner-insensitive matching: I tried the second option on top of #8169, removing U+200C/U+200D from both the query and the stored text before tokenizing, and compared it with #8182 on the same data in all three paths (in-memory, SQLite FTS and SQLite LIKE).

    Query #8182 (strict) Joiners removed
    র‍্যাব (ZWJ) only র‍্যাব (FTS also র্যাব) র‍্যাব and র্যাব, everywhere
    র্যাব only র্যাব (FTS also র‍্যাব) র‍্যাব and র্যাব, everywhere
    می‌خواهم (ZWNJ) only می‌خواهم (FTS also می خواهم) می‌خواهم and میخواهم, everywhere
    میخواهم only میخواهم میخواهم and می‌خواهم, everywhere
    کتاب‌ها only کتاب‌ها only کتاب‌ها
    False matches (রাজ, مرد‌ها, other emoji sequences) none none
    क्‌ष only क्‌ष in-memory and LIKE also match क्षमा; FTS doesn't

    So removing the joiners keeps the false matches from #8181 out and makes the spellings with and without a joiner find each other, consistently across the three paths. The spaced form (می خواهم) isn't matched in either version, except by FTS in the strict one. The क्षमा match comes from the substring matching for non-ASCII words, and a plain क्ष query already does that today.

    This is also what NFKC_Casefold does, since ZWJ and ZWNJ are default-ignorable. My preference would be the joiner-insensitive version, but I'm fine either way. One thing to keep in mind for SQLite: if the folding is done when the text is stored, rows written before the change still contain the joiners, so they'd either need reindexing or a REPLACE on search_text in the LIKE path.

    I'll leave #8182 as it is until a reviewer picks one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions