Add tokenizer mode support for message search by shinaoka · Pull Request #33048 · element-hq/element-web

shinaoka · 2026-04-06T05:44:23Z

Summary

This PR ports the tokenizer-mode work to the monorepo layout and combines the previous web and desktop-side changes into a single branch.

It adds a user-facing tokenizer mode setting for local message search, passes the selected mode through the desktop layer to Seshat, and asks for confirmation before reindexing when the mode changes.

Merge Status

matrix-org/seshat#150 is merged, but matrix-org/seshat#157 is still open and matrix-seshat@4.1.0 has not been released yet.

This PR should stay draft for now. Once matrix-seshat@4.1.0 is released, the dependency can be updated and this PR should be ready to merge.

Testing

pnpm --dir apps/desktop test:unit
pnpm --dir apps/web test -- --runTestsByPath test/unit-tests/async-components/dialogs/eventindex/ManageEventIndexDialog-test.tsx

Add user-facing setting to choose tokenizer mode (language-based vs N-gram) for local message search. N-gram tokenization supports CJK languages and mixed-language content. Includes confirmation dialog for mode changes (requires reindex) and automatic checkpoint recovery. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add N-gram tokenizer support to Seshat search index initialization. Refactors initEventIndex into a testable module with dependency injection, supporting both language-based and N-gram tokenization modes. Includes automatic database recreation fallback and unit tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Previously, any Seshat open failure that wasn't a ReindexError would silently delete the EventStore and recreate it, losing the user's search index on transient errors like filesystem locks or passphrase issues. Restore the original behavior: only handle ReindexError with recovery logic, and propagate all other errors to the caller (which shows an error to the user via sendError). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

aik2mlj · 2026-04-14T17:00:57Z

Seshat has released v4.2.0. Perhaps now the dependency can be updated, and this pull request can proceed to be reviewed by the team. Great appreciation from a CJK language user.

t3chguy · 2026-04-16T08:31:46Z

#33168

…er-mode # Conflicts: # apps/desktop/package.json # pnpm-lock.yaml

shinaoka · 2026-05-16T04:45:53Z

The default mode is "language-based."

shinaoka · 2026-06-02T07:15:15Z

@Half-Shot @MidhunSureshR How can I help you?

Half-Shot · 2026-06-11T09:24:15Z

@Half-Shot @MidhunSureshR How can I help you?

Sorry, this is entirely blocked on us finding time to review. Thanks for the patience 🙏

Half-Shot

Sorry for the lateness on this review. I've asked the crypto team to lend a hand with the more technical aspect of the review, but given some pointers on the web side.

Half-Shot · 2026-06-11T09:29:43Z

+ * Create Seshat configuration based on tokenizer mode.
+ *
+ * @param tokenizerMode - The tokenizer mode: "ngram" for N-gram tokenization (CJK languages),
+ *                        or "language" for standard language-based tokenization.
+ * @returns Configuration object for Seshat initialization.
+ */


Your docstring has jumped up a few lines!

Moved the docstring back onto createSeshatConfig so it no longer documents the enum.

Half-Shot · 2026-06-11T09:30:29Z

+    Language = "language",
+}
+
+export function createSeshatConfig(tokenizerMode?: string): {


Should this throw if an unknown tokenizerMode is given?

Updated createSeshatConfig to default only when no tokenizerMode is provided, and to throw for unknown values. Added coverage for that case.

Half-Shot · 2026-06-11T09:43:01Z

                        value={this.state.crawlerSleepTime.toString()}
                        onChange={this.onCrawlerSleepTimeChange}
                    />
+                    <Field


So I think this bit of the UI is a bit clunky and could do with a refactor, how about using a Compound option component https://compound.element.io/?path=/docs/compound-web_form-controls-radio--docs instead which means:

You can separate the labels and descriptions (so you don't need the big block of text below)

Using Compound means it'll fit the rest of our app.

is a trivial example of what I mean.

The other thing I'd do is add a header with the `"settings|security|tokenizer_mode" string

Refactored the tokenizer mode selector to Compound radio controls, added the tokenizer mode heading, and split the labels/descriptions so the explanatory text is attached to each option.

Half-Shot · 2026-06-11T09:45:37Z

+            "strict_encryption": "Only send messages to verified users",
+            "tokenizer_mode": "Search tokenizer mode",
+            "tokenizer_mode_change_warning": "Changing the tokenizer mode requires deleting the existing search index. The index will be rebuilt automatically. Do you want to continue?",
+            "tokenizer_mode_description": "Language-based: Single language support fixed to UI language. Only works with some languages (English, German, etc.). N-gram: Supports all languages including those without word boundaries. Works with mixed languages. Changing this setting will rebuild your search index.",


You probably don't need "Changing this setting will rebuild your search index" if we have a warning too.

Removed the rebuild note from the option description. The destructive/rebuild behavior is now only in the confirmation warning.

Half-Shot · 2026-06-11T09:46:45Z

+    public async initEventIndex(
+        userId: string,
+        deviceId: string,
+        tokenizerMode?: string,


This should probably use the enum, or at the very least:

Suggested change

tokenizerMode?: string,

tokenizerMode?: "ngram"|"language",

Narrowed tokenizerMode through the web indexing manager API using a TokenizerMode enum.

Half-Shot · 2026-06-11T09:47:17Z


-        if (!SettingsStore.getValueAt(SettingLevel.DEVICE, "enableEventIndexing")) {
+        const enableEventIndexing = SettingsStore.getValueAt(SettingLevel.DEVICE, "enableEventIndexing");
+        if (!enableEventIndexing) {


This seems like an unrelated change?

This change is needed to pass the selected tokenizer mode from device settings into the platform index manager. I kept it and narrowed the type so the intent is clearer.

Half-Shot · 2026-06-11T09:49:48Z

+    public async initEventIndex(
+        userId: string,
+        deviceId: string,
+        tokenizerMode?: string,


Also could do with a more narrow string type?

Narrowed this to the same TokenizerMode enum used by the base index manager API.

Half-Shot · 2026-06-11T09:50:09Z

+
+        render(<ConfirmTokenizerChangeDialog onFinished={onFinished} />);
+
+        fireEvent.click(screen.getByRole("button", { name: /ok/i }));


Anything wrong with just "ok"? And I think this needs awaiting?

Updated the test to click the exact "OK" button text and await userEvent.click.

Half-Shot · 2026-06-11T09:51:11Z

+
+import BaseEventIndexManager from "../../../src/indexing/BaseEventIndexManager";
+
+describe("BaseEventIndexManager", () => {


I'm not sure this needs to even be tested to be honest

Removed this test. It only asserted the base unimplemented method behavior and did not add useful coverage for tokenizer mode.

Half-Shot · 2026-06-11T09:51:42Z

+
+    it("passes tokenizerMode to initEventIndex IPC call", async () => {
+        // IPCManager requires window.electron to exist.
+        window.electron = {


I agree with SonarCloud :)

Suggested change

window.electron = {

globalThis.electron = {

Updated the electron stub to use globalThis with a typed cast.

Half-Shot · 2026-06-25T08:00:42Z

@shinaoka Are you able to continue with this PR?

shinaoka · 2026-06-25T08:05:39Z

Oh, I am sorry. I just missed your messages. I will work on them.

…nderer crash auto-recovery (session 14-15, element-hq#33048/element-hq#32038, element-hq#31996, element-hq#32222)

shinaoka · 2026-07-01T13:14:43Z

Addressed the review comments in b3df964:

Moved the Seshat config docstring and made unknown tokenizer modes throw.
Narrowed tokenizerMode through the web indexing path with a TokenizerMode enum.
Refactored the tokenizer mode setting UI to Compound radio controls with separate labels/descriptions.
Removed the duplicated rebuild warning copy from the option description.
Cleaned up the related tests.

Verification run locally:

Desktop vitest for seshat config/index tests: 8 passed
Web jest for event index dialog, SeshatIndexManager, and EventIndexPeg tests: 11 passed
Web and desktop TypeScript checks passed
ESLint passed for touched files
git diff --check passed

Modificator · 2026-07-15T09:14:31Z

Any progress? @Half-Shot

shinaoka and others added 4 commits March 30, 2026 20:26

chore: update lockfile for tokenizer mode branch

940ae09

github-actions Bot added the Z-Community-PR Issue is solved by a community member's PR label Apr 6, 2026

Merge upstream/develop into shinaoka/tokenizer-mode

a27bf89

github-actions Bot deployed to Netlify April 18, 2026 04:20 View deployment

github-actions Bot deployed to EndToEndTests April 18, 2026 04:46 View deployment

shinaoka added 2 commits May 16, 2026 13:33

Fix tokenizer mode CI failures

cbf3e39

Merge remote-tracking branch 'upstream/develop' into shinaoka/tokeniz…

eb6f973

…er-mode # Conflicts: # apps/desktop/package.json # pnpm-lock.yaml

github-actions Bot deployed to Netlify May 16, 2026 04:41 View deployment

shinaoka mentioned this pull request May 16, 2026

Add Tokenizer Mode Support for Message Search #31540

Closed

5 tasks

shinaoka marked this pull request as ready for review May 16, 2026 04:51

shinaoka requested a review from a team as a code owner May 16, 2026 04:51

shinaoka requested review from Half-Shot and MidhunSureshR May 16, 2026 04:51

github-actions Bot deployed to EndToEndTests May 16, 2026 04:58 View deployment

Half-Shot requested changes Jun 11, 2026

View reviewed changes

hayaksi1 pushed a commit to hayaksi1/element-web that referenced this pull request Jun 27, 2026

feat(web,desktop): N-gram search tokenizer, notif-sound throttle & re…

f824c65

…nderer crash auto-recovery (session 14-15, element-hq#33048/element-hq#32038, element-hq#31996, element-hq#32222)

Address tokenizer mode review comments

b3df964

github-actions Bot deployed to Netlify July 1, 2026 14:20 View deployment

Half-Shot self-requested a review July 1, 2026 14:35

github-actions Bot deployed to EndToEndTests July 1, 2026 14:39 View deployment


		render(<ConfirmTokenizerChangeDialog onFinished={onFinished} />);

		fireEvent.click(screen.getByRole("button", { name: /ok/i }));


		import BaseEventIndexManager from "../../../src/indexing/BaseEventIndexManager";

		describe("BaseEventIndexManager", () => {

Uh oh!

Uh oh!

Conversation

shinaoka commented Apr 6, 2026

Summary

Related

Merge Status

Testing

Uh oh!

aik2mlj commented Apr 14, 2026

Uh oh!

t3chguy commented Apr 16, 2026

Uh oh!

shinaoka commented May 16, 2026

Uh oh!

shinaoka commented Jun 2, 2026

Uh oh!

Half-Shot commented Jun 11, 2026

Uh oh!

Half-Shot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Half-Shot commented Jun 25, 2026

Uh oh!

shinaoka commented Jun 25, 2026

Uh oh!

shinaoka commented Jul 1, 2026

Uh oh!

Modificator commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants