Add Tokenizer Mode Support for Seshat Search Index by shinaoka · Pull Request #2753 · element-hq/element-desktop

shinaoka · 2025-12-14T23:11:12Z

element-meta Issue #2883: i18n support for full-text search in encrypted rooms

Summary

This PR adds tokenizer mode support to element-desktop, enabling users to choose between standard (language-based) and N-gram tokenization for the local message search index.

Changes

Add tokenizerMode option to EventIndex initialization
Accept tokenizerMode from element-web via IPC
Support both N-gram and language-based tokenization modes
Default to language-based tokenization for backward compatibility

Files Changed

src/@types/matrix-seshat.d.ts - Add TypeScript types for tokenizer options
src/seshat.ts - Pass tokenizerMode to Seshat, handle IPC from element-web

Dependencies

⚠️ This PR depends on:

Seshat: PR #150: Add N-gram Tokenizer Mode for CJK Language Support

Seshat PR must be merged and a new npm package version released before this PR can be merged.

After Seshat is released, update package.json to use the new matrix-seshat version (e.g., ^4.1.0).

Merge Order

1. Seshat PR #150  →  npm release (e.g., matrix-seshat@4.1.0)
2. element-desktop (this PR)  →  update matrix-seshat dependency
3. element-web  →  UI for tokenizer mode selection

Checklist

Ensure your code works with manual testing.
New or updated public/exported symbols have accurate TSDoc documentation.
Linter and other CI checks pass.
I have licensed the changes to Element by completing the Contributor License Agreement (CLA)

dbkr · 2025-12-15T16:44:20Z

This needs the other two PRs to land first (which will need a look from product).

dbkr · 2026-02-26T10:44:29Z

@@ -0,0 +1,29 @@
+/*
+Copyright 2025 New Vector Ltd.


Copyright should be you

I interpreted your comment as asking for my name in the copyright header. I kept “New Vector Ltd.” because that seems to be the existing convention in this repo (and CONTRIBUTING.md points to the CLA). If you want individual author names on new files instead, I can update them.

Yep, it's New Vector Ltd on most of the files because most of it was written by New Vector employees, but since this is your work, you own the copyright.

dbkr · 2026-02-26T10:49:54Z

+                        // Schema mismatch or other errors - delete and recreate the database
+                        console.warn("Failed to open Seshat database, deleting and recreating:", e);
+                        await deleteContents(eventStorePath);
+                        try {
+                            eventIndex = new Seshat(eventStorePath, { passphrase, ...seshatConfig });
+                            // Return that the database was recreated so element-web can re-add checkpoints
+                            ret = { wasRecreated: true };
+                        } catch (e2) {
+                            sendError(payload.id, <Error>e2);
+                            return;
+                        }


Not sure I understand why this change is relevant to this PR or why it's better

This change is needed because changing the n-gram/tokenizer configuration makes the existing Seshat database incompatible. With this fallback, re-indexing will rebuild the index.

Ah okay. I was going to suggest putting this in a comment but the code seems to be gone now?

dbkr · 2026-02-26T10:53:28Z

@@ -0,0 +1,40 @@
+/*
+Copyright 2025 New Vector Ltd.


Copyright here too

dbkr · 2026-02-26T10:56:29Z

+            },
+        ],
+    },
+};


Unfortunately we're really trying to move away from jest so it would be much better to use vitest if we're introducing unit tests to the desktop package.

I have addressed this issue.

- Add tokenizerMode option to EventIndex initialization - Accept tokenizerMode from element-web via IPC - Support both N-gram and language-based tokenization modes - Default to language-based tokenization for backward compatibility

dbkr · 2026-03-02T11:04:55Z

Also if you could please avoid force pushing: it makes it very hard to review a PR if we can't see what is changing.

dbkr · 2026-03-05T09:46:34Z

What's the status of this PR? It looks like you removed the wasRecreated return value that was added but it's still in element-hq/element-web#31540. The tests also look to have been removed completely and the copyright still needs fixing. That said, I'm also not sure what has changed as you force pushed so the history is now gone.

shinaoka · 2026-03-07T06:28:49Z

I restored the wasRecreated return path in desktop so it matches the flow expected by element-web PR 31540.
I fixed the copyright header in src/seshat-config.ts.
Dependency note: desktop uses hakDependencies with matrix-seshat: ^4.0.1, resolved via hak.
matrix-org/seshat#157 is still open, and npm matrix-seshat@latest is still 4.0.1.

So this desktop PR is backward-compatible and safe to merge now; tokenizerMode=ngram will become effective once a newer matrix-seshat release is available.

One question on tests, so I can align with your preference:
Would you like me to add a minimal Vitest unit test for createSeshatConfig, or cover this at Playwright level instead?

I’ll add follow-up commits only (no force-push).

dbkr · 2026-03-12T09:52:42Z

Thanks for doing that. If it is possible to add a minimal vitest test then that would be ideal - since this previously added a jest test, I assume this shouldn't be too problematic (hopefully?). Much appreciated!

shinaoka · 2026-03-13T13:06:08Z

Added minimal Vitest coverage.

src/seshat.ts changed only to extract the initEventIndex logic into a helper so the tokenizer config path can be unit-tested directly; behavior should be unchanged.

The pnpm-lock.yaml diff is just the result of adding vitest to the dependencies, not a manual change.

dbkr

Thanks for doing this, great to have vitests for desktop too.

… into shinaoka/ngram2 # Conflicts: # pnpm-lock.yaml

t3chguy · 2026-03-25T10:25:07Z

Sorry this repo has been merged into element-web so you will need to merge your changes into element-hq/element-web#31540 apps/desktop

shinaoka requested a review from a team as a code owner December 14, 2025 23:11

shinaoka requested review from dbkr and florianduros December 14, 2025 23:11

github-actions Bot added the Z-Community-PR Issue is solved by a community member's PR label Dec 14, 2025

dosubot Bot added the T-Enhancement label Dec 14, 2025

shinaoka force-pushed the shinaoka/ngram2 branch from 567a3db to edc0d00 Compare December 14, 2025 23:21

shinaoka requested a review from a team as a code owner December 14, 2025 23:21

shinaoka force-pushed the shinaoka/ngram2 branch from edc0d00 to cfa845c Compare December 14, 2025 23:35

shinaoka mentioned this pull request Dec 15, 2025

Add Tokenizer Mode Support for Message Search element-hq/element-web#31540

Closed

5 tasks

dbkr reviewed Feb 26, 2026

View reviewed changes

shinaoka force-pushed the shinaoka/ngram2 branch from cfa845c to 15331c1 Compare March 1, 2026 21:00

Merge branch 'develop' into shinaoka/ngram2

e7a7f27

shinaoka added 2 commits March 6, 2026 16:59

chore: update seshat-config copyright header

3017997

fix: restore wasRecreated return on db recreation

9385234

test: add vitest coverage for seshat init

aa70bd0

github-advanced-security AI found potential problems Mar 13, 2026

View reviewed changes

Comment thread test/unit-tests/seshat-index.test.ts Fixed

Comment thread test/unit-tests/seshat-index.test.ts Fixed

Comment thread test/unit-tests/seshat-index.test.ts Fixed

Comment thread test/unit-tests/seshat-index.test.ts Fixed

test: avoid sonar secret false positive

891c285

dbkr reviewed Mar 16, 2026

View reviewed changes

Comment thread test/unit-tests/seshat-index.test.ts Outdated

shinaoka added 2 commits March 19, 2026 21:11

test: use platform temp dir in seshat index tests

8c68bae

Merge branch 'develop' of https://github.com/element-hq/element-desktop…

8930cba

… into shinaoka/ngram2 # Conflicts: # pnpm-lock.yaml

t3chguy requested a review from dbkr March 20, 2026 15:23

t3chguy closed this Mar 25, 2026

shinaoka mentioned this pull request Apr 6, 2026

Add tokenizer mode support for message search element-hq/element-web#33048

Open

Uh oh!

Conversation

shinaoka commented Dec 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Files Changed

Dependencies

Merge Order

Checklist

Uh oh!

dbkr commented Dec 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shinaoka Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dbkr commented Mar 2, 2026

Uh oh!

dbkr commented Mar 5, 2026

Uh oh!

shinaoka commented Mar 7, 2026

Uh oh!

dbkr commented Mar 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shinaoka commented Mar 13, 2026

Uh oh!

dbkr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

t3chguy commented Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shinaoka commented Dec 14, 2025 •

edited

Loading

shinaoka Mar 1, 2026 •

edited

Loading