fix(memory): tokenize on punctuation and all whitespace in extractWords

[Ajithkumar003-dev]

What

extractWords in memory/inmemory.go now splits text on runs of non-word runes (strings.FieldsFunc) instead of on the literal space character, so punctuation and non-space whitespace no longer become part of an indexed token.

Fixes #569

Why

extractWords builds the keyword index for InMemoryService and also tokenizes the incoming query, so both sides were affected:

1. Punctuation stayed glued to the word. "The agent works great!" indexed the token great!, so SearchMemory with query great returned nothing.
2. Tabs and newlines were never split on. "first line\nsecond\tline" indexed the single token line\nsecond\tline, making every word after the first newline unreachable. Multi-line model responses are the common case here.
3. Punctuated queries failed symmetrically — a query of "Why timed-out?" produced the token timed-out?, which matched nothing.

Implementation note

The issue proposed strings.Fields + strings.TrimFunc. I used strings.FieldsFunc with a non-word separator predicate instead, for two reasons:

• TrimFunc only strips punctuation from the ends of a token, so the comma-separated case the issue lists as affected would still break: regions: us-east1,us-west1 stays one token. FieldsFunc handles it.
• It mirrors adk-python's tokenizer, re.findall(r'\w+', text) in in_memory_memory_service.py, keeping the two implementations aligned per the Alignment with adk-python guidance in CONTRIBUTING.md. The predicate accepts Unicode letters, Unicode digits and _, matching Python 3's \w.

Happy to switch to the TrimFunc approach if maintainers prefer to keep hyphenated and comma-joined tokens intact.

FieldsFunc never yields empty strings, so the old if s == "" { continue } guard is now dead code and was removed.

Testing plan

Three cases added to the existing Test_inMemoryService_SearchMemory table, one per failure mode above:

• find events next to punctuation — covers great! and regions: us-east1,us-west1
• find events separated by non-space whitespace — covers \n and \t
• find events for a query containing punctuation — covers query-side normalization

Each was verified to fail against the unfixed extractWords before the fix was applied, so they are genuine regression tests rather than tests written to match new behavior:

--- FAIL: Test_inMemoryService_SearchMemory
    --- FAIL: Test_inMemoryService_SearchMemory/find_events_next_to_punctuation
    --- FAIL: Test_inMemoryService_SearchMemory/find_events_separated_by_non-space_whitespace
    --- FAIL: Test_inMemoryService_SearchMemory/find_events_for_a_query_containing_punctuation

With the fix applied, the full package passes, including the pre-existing find events and concurrency tests:

--- PASS: Test_inMemoryService_SearchMemory (0.00s)
    --- PASS: Test_inMemoryService_SearchMemory/find_events (0.00s)
    --- PASS: Test_inMemoryService_SearchMemory/find_events_next_to_punctuation (0.00s)
    --- PASS: Test_inMemoryService_SearchMemory/find_events_separated_by_non-space_whitespace (0.00s)
    --- PASS: Test_inMemoryService_SearchMemory/find_events_for_a_query_containing_punctuation (0.00s)
    --- PASS: Test_inMemoryService_SearchMemory/no_leakage_for_different_appName (0.00s)
    --- PASS: Test_inMemoryService_SearchMemory/no_leakage_for_different_user (0.00s)
    --- PASS: Test_inMemoryService_SearchMemory/no_matches (0.00s)
    --- PASS: Test_inMemoryService_SearchMemory/lookup_on_empty_store (0.00s)
--- PASS: Test_inMemoryService_SearchMemory_Concurrent (0.00s)
PASS

Also run locally: go build ./... (whole module, clean), go vet ./memory/ (clean), gofmt (clean). extractWords is unexported and referenced only within the memory package, so the change is self-contained.

I could not run go test -race locally — it requires cgo and there is no C toolchain on my machine. The change introduces no new concurrency, and the existing Test_inMemoryService_SearchMemory_Concurrent passes without race enabled; relying on CI for the race run.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.