Add Tokenizer Mode Support for Message Search by shinaoka · Pull Request #31540 · element-hq/element-web

shinaoka · 2025-12-14T23:13:43Z

Summary

This PR adds a user-facing setting to choose the tokenizer mode for local message search. N-gram tokenization is essential for languages without clear word boundaries such as Japanese, Chinese, and Korean (CJK languages).

Changes

Add tokenizer mode setting (Standard/Japanese) in Message Search preferences
Pass tokenizer mode to Seshat via element-desktop IPC
Add confirmation dialog when changing tokenizer mode (requires reindex)

Dependencies

⚠️ This PR depends on:

Seshat: PR #150: Add N-gram Tokenizer Mode for CJK Language Support

element-desktop: PR #2753: Add Tokenizer Mode Support for Seshat Search Index

Both upstream PRs must be merged before this PR can be merged.

Merge Order

1. Seshat PR #150  →  npm release (e.g., matrix-seshat@4.1.0)
2. element-desktop PR  →  update matrix-seshat dependency
3. element-web (this PR)  →  UI for tokenizer mode selection

Checklist

I have read through review guidelines and CONTRIBUTING.md.
Tests written for new code (and old code if feasible).
New or updated public/exported symbols have accurate TSDoc documentation.
Linter and other CI checks pass.
I have licensed the changes to Element by completing the Contributor License Agreement (CLA)

t3chguy · 2025-12-15T11:40:29Z

@shinaoka seems like you have lint & insufficient test coverage issues

shinaoka · 2025-12-22T05:31:33Z

I have managed to fix the CI errors!

t3chguy · 2026-01-02T10:27:14Z

Blocked on matrix-org/seshat#150

langleyd · 2026-02-04T09:53:56Z

Thanks for the contribution @shinaoka , this seems like a very valuable one.
The product team need a bit of time to consider how we should handle this mode in the UX.
It'll be next week before we have a response.

shinaoka · 2026-02-23T00:38:14Z

Should I proceed? @langleyd

daniellekirkwood · 2026-02-23T11:28:50Z

Looks like this is an improvement for users -- thank you for the contribution, and thank you for your patience.

From a product perspective, we think in the future we might be able to make search a much better experience but I don't want to hold off on this change waiting for that future... So let's move ahead here but know that we may choose to update the Search UI & UX in the future 🙏

Rebased onto latest develop with monorepo directory structure (apps/web/). - Add tokenizer mode setting (Standard/Japanese) in Message Search preferences - Pass tokenizer mode to Seshat via element-desktop IPC - Add confirmation dialog when changing tokenizer mode (requires reindex) - Add tests for tokenizer mode functionality Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Half-Shot · 2026-03-19T12:03:30Z

Converting to draft while element-hq/element-desktop#2753 is outstanding so it doesn't end up in our review queue :)

shinaoka · 2026-03-23T07:49:48Z

Updated this PR with a fix for the tokenizer-change flow in element-web and added a regression test. This is still waiting on the element-desktop and seshat side changes.

shinaoka · 2026-03-30T12:41:42Z

Since element-desktop has been merged into element-web (monorepo), I've prepared a new branch (shinaoka/tokenizer-mode) based on the latest develop that includes both:

the web-side changes from this PR
the desktop-side changes from Add Tokenizer Mode Support for Seshat Search Index element-desktop#2753

Would it be OK to open a new PR from that branch to replace this one? This would keep everything in a single PR and avoid merging ~3500 commits of drift into this branch.

I'll reference this PR and its discussion in the new one.

langleyd · 2026-04-02T09:06:16Z

Sounds good, thanks

shinaoka · 2026-05-05T23:50:48Z

I forgot to make it ready for review...

t3chguy · 2026-05-08T08:46:24Z

@@ -0,0 +1,60 @@
+/*
+Copyright 2024 New Vector Ltd.


This should be your copyright, with the present year

t3chguy · 2026-05-08T08:46:29Z

+Copyright 2025 New Vector Ltd.
+Copyright 2020, 2021 The Matrix.org Foundation C.I.C.


t3chguy · 2026-05-08T08:46:34Z

@@ -0,0 +1,195 @@
+/*
+Copyright 2025 New Vector Ltd.


t3chguy · 2026-05-08T08:46:42Z

+Copyright 2025 New Vector Ltd.
+Copyright 2020, 2021 The Matrix.org Foundation C.I.C.


t3chguy · 2026-05-08T08:46:45Z

+Copyright 2025 New Vector Ltd.
+Copyright 2020, 2021 The Matrix.org Foundation C.I.C.


t3chguy · 2026-05-08T08:46:49Z

+Copyright 2025 New Vector Ltd.
+Copyright 2020, 2021 The Matrix.org Foundation C.I.C.


t3chguy · 2026-05-08T08:47:47Z

    "RoomList.showMessagePreview": IBaseSetting<boolean>;
    "RightPanel.phasesGlobal": IBaseSetting<IRightPanelForRoomStored | null>;
    "RightPanel.phases": IBaseSetting<IRightPanelForRoomStored | null>;
    "enableEventIndexing": IBaseSetting<boolean>;
    "crawlerSleepTime": IBaseSetting<number>;
+    "tokenizerMode": IBaseSetting<string>;


Should this not be "language" | "ngram"?

t3chguy · 2026-05-08T08:48:37Z

Please include some screenshots in the PR of the new UI/modals

shinaoka · 2026-05-16T04:50:54Z

I have posted some screenshots in the new PR:
#33048

Let me close this PR in favor of the new one.

shinaoka requested a review from a team as a code owner December 14, 2025 23:13

shinaoka requested review from MidhunSureshR and t3chguy December 14, 2025 23:13

github-actions Bot added the Z-Community-PR Issue is solved by a community member's PR label Dec 14, 2025

shinaoka force-pushed the shinaoka/ngram_PR branch from 75bba9f to 3ea2396 Compare December 14, 2025 23:15

github-actions Bot deployed to Netlify December 14, 2025 23:20 View deployment

github-actions Bot deployed to EndToEndTests December 14, 2025 23:35 View deployment

t3chguy added the T-Enhancement label Dec 15, 2025

github-actions Bot deployed to EndToEndTests December 15, 2025 11:51 View deployment

github-actions Bot deployed to Netlify December 21, 2025 06:19 View deployment

github-actions Bot deployed to EndToEndTests December 21, 2025 06:32 View deployment

github-actions Bot deployed to Netlify December 22, 2025 04:01 View deployment

github-actions Bot deployed to Netlify December 22, 2025 04:16 View deployment

github-actions Bot deployed to EndToEndTests December 22, 2025 04:30 View deployment

t3chguy added the X-Blocked label Jan 2, 2026

dbkr added the X-Needs-Product More input needed from the Product team label Jan 8, 2026

github-actions Bot deployed to Netlify February 23, 2026 00:39 View deployment

github-actions Bot deployed to EndToEndTests February 23, 2026 00:51 View deployment

langleyd removed X-Needs-Product More input needed from the Product team X-Blocked labels Feb 23, 2026

shinaoka force-pushed the shinaoka/ngram_PR branch from 5c57ad1 to ca56669 Compare February 25, 2026 20:56

github-actions Bot deployed to Netlify February 25, 2026 20:58 View deployment

github-actions Bot deployed to EndToEndTests February 25, 2026 21:09 View deployment

dbkr mentioned this pull request Mar 5, 2026

Add Tokenizer Mode Support for Seshat Search Index element-hq/element-desktop#2753

Closed

4 tasks

Half-Shot marked this pull request as draft March 19, 2026 12:03

fix: handle tokenizer change dialog result

0bfab3a

github-actions Bot deployed to Netlify March 23, 2026 07:51 View deployment

github-actions Bot deployed to EndToEndTests March 23, 2026 08:02 View deployment

shinaoka mentioned this pull request Apr 6, 2026

Add tokenizer mode support for message search #33048

Open

shinaoka marked this pull request as ready for review May 5, 2026 23:50

t3chguy reviewed May 8, 2026

View reviewed changes

shinaoka closed this May 16, 2026

		Copyright 2025 New Vector Ltd.
		Copyright 2020, 2021 The Matrix.org Foundation C.I.C.

Uh oh!

Uh oh!

Conversation

shinaoka commented Dec 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Dependencies

Merge Order

Checklist

Uh oh!

t3chguy commented Dec 15, 2025

Uh oh!

shinaoka commented Dec 22, 2025

Uh oh!

t3chguy commented Jan 2, 2026

Uh oh!

langleyd commented Feb 4, 2026

Uh oh!

shinaoka commented Feb 23, 2026

Uh oh!

daniellekirkwood commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Half-Shot commented Mar 19, 2026

Uh oh!

shinaoka commented Mar 23, 2026

Uh oh!

shinaoka commented Mar 30, 2026

Uh oh!

langleyd commented Apr 2, 2026

Uh oh!

shinaoka commented May 5, 2026

Uh oh!

t3chguy May 8, 2026

Choose a reason for hiding this comment

Uh oh!

t3chguy May 8, 2026

Choose a reason for hiding this comment

Uh oh!

t3chguy May 8, 2026

Choose a reason for hiding this comment

Uh oh!

t3chguy May 8, 2026

Choose a reason for hiding this comment

Uh oh!

t3chguy May 8, 2026

Choose a reason for hiding this comment

Uh oh!

t3chguy May 8, 2026

Choose a reason for hiding this comment

Uh oh!

t3chguy May 8, 2026

Choose a reason for hiding this comment

Uh oh!

t3chguy commented May 8, 2026

Uh oh!

shinaoka commented May 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

shinaoka commented Dec 14, 2025 •

edited

Loading

daniellekirkwood commented Feb 23, 2026 •

edited

Loading