feat(csfle): add CSFLE for field messages#14156
Open
diogoserrano wants to merge 12 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds MongoDB Client-Side Field Level Encryption (CSFLE) for sensitive messages fields, including Docker packaging of the crypt_shared library, runtime wiring in the API DB connection, CSFLE setup/migration helpers, and supporting documentation/tests.
Changes:
- Introduces CSFLE utilities (KMS provider setup, DEK bootstrap, schemaMap creation) and a migration manager to backfill existing plaintext
messages.text/messages.content. - Updates the API Mongo connection to enable
autoEncryptionand optionally run startup migrations. - Updates Docker and env/docs to support bundling and configuring
mongo_crypt_v1.so.
Reviewed changes
Copilot reviewed 14 out of 15 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/data-schemas/src/models/message.ts | Disables MeiliSearch indexing for messages when CSFLE is enabled. |
| package-lock.json | Adds mongodb-client-encryption and its transitive dependencies to the lockfile. |
| Dockerfile | Adds a build stage to download/verify mongo_crypt_v1.so and switches runtime base to Debian glibc. |
| api/package.json | Adds mongodb-client-encryption dependency for CSFLE support. |
| api/db/connect.js | Enables Mongo auto-encryption options and runs startup migrations when configured. |
| api/csfle/README.md | Adds end-user documentation for setup, env vars, and migration flow. |
| api/csfle/provider.js | Implements KMS provider selection (GCP or local) and GCP key file parsing. |
| api/csfle/manager.js | Adds migration state tracking and backfill logic for existing messages. |
| api/csfle/init.js | Bootstraps key vault collection/index and creates DEKs idempotently. |
| api/csfle/index.js | Builds/caches autoEncryption options (key vault namespace, schemaMap, extraOptions). |
| api/csfle/fields.js | Defines encrypted fields and constructs the schemaMap for auto-encryption. |
| api/csfle/deks.js | Defines DEK alt-name constants for the key vault. |
| api/csfle/tests/provider.sanity.js | Adds a standalone (non-jest) sanity script for provider behavior. |
| api/csfle/tests/manager.test.js | Adds Jest tests for migration manager state/idempotency/error handling. |
| .env.example | Documents new CSFLE-related environment variables and defaults. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Implements Client-Side Field Level Encryption with versioned policies,
startup-integrated migration manager, and __csfle_migrations state tracking.
## What's included
- api/csfle/policies/v1.js — Wave 1: users (email/openidId/googleId/name),
sessions (refreshTokenHash), keys (value)
- api/csfle/policies/v2.js — Wave 2: messages (text/content),
conversations (title) [requires MeiliSearch disabled]
- api/csfle/policies/v3.js — Wave 3: users (username/avatar),
files (filename/filepath)
- api/csfle/policies/index.js — registry, checksum builder, schema builder
- api/csfle/manager.js — migration manager with __csfle_migrations state,
applied-once idempotency, crash recovery, strict/warn startup policy
- api/csfle/index.js — buildAutoEncryptionOptions() (cached)
- api/csfle/init.js — __keyVault bootstrap + 6 DEK idempotent creation
- api/csfle/provider.js — KMS: GCP (ADC/Workload Identity) or local master key
- api/csfle/deks.js — DEK alt-name constants
- api/csfle/schemas.js — thin wrapper -> buildSchemaFromPolicies()
- api/csfle/__tests__/manager.test.js — Jest tests (mongodb-memory-server)
- api/csfle/README.md — integration contract for nos-gpt-apps
- api/db/connect.js — autoEncryption injection + startup auto-migrate hook
- api/package.json — mongodb-client-encryption@^6.4.0
- packages/data-schemas/src/models/message.ts — skip MeiliSearch when CSFLE_ENABLED
- packages/data-schemas/src/models/convo.ts — skip MeiliSearch when CSFLE_ENABLED
- .env.example — all CSFLE env vars documented
## New env vars
CSFLE_ENABLED activate CSFLE in MongoClient
CSFLE_AUTO_MIGRATE run migration manager at startup
CSFLE_STARTUP_POLICY strict (default) | warn
CSFLE_MIGRATION_TARGET_VERSION limit to policies <= N
MONGO_CSFLE_KEY_VAULT_NAMESPACE defaults to <dbName>.__keyVault
MONGO_CRYPT_SHARED_LIB_PATH path to crypt_shared in container
MONGO_CSFLE_LOCAL_MASTER_KEY 96-byte base64 key (dev/CI only)
GCP_KMS_PROJECT_ID/LOCATION/KEY_RING/KEY_NAME production KMS
## Startup flow
connectDb() -> buildAutoEncryptionOptions() -> mongoose.connect()
-> if CSFLE_AUTO_MIGRATE: runStartupMigration() -> runMigrations()
marks each policy PENDING -> backfills -> marks APPLIED
strict mode: throw on failure; warn mode: log and continue
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
## Lockfile fix - Add mongodb-client-encryption@6.5.0 to root package-lock.json via npm install --package-lock-only so npm ci in Docker build resolves correctly ## Reduced encryption scope Encrypted fields trimmed to only the requested set: sessions.refreshTokenHash — Deterministic (equality lookup) keys.value — Random messages.text — Random messages.content — Random conversations.title — Random Changes: - policies/v1.js: now covers sessions + keys only (removed users.* fields) - policies/v2.js: unchanged (messages + conversations) - policies/v3.js: deleted (users.username/avatar + files.* no longer encrypted) - policies/index.js: POLICIES array trimmed to [v1, v2] - deks.js: removed dek-users and dek-files constants (4 DEKs remain) - README.md: updated policy table and DEK alt-names table - __tests__/manager.test.js: hard-code expected policy count to 2 MeiliSearch guard (messages + conversations) is unchanged and correct. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…encryption
Root cause: ClientEncryption is exported by the mongodb driver package
(mongodb@6.x), not by the mongodb-client-encryption native addon which
only exports MongoCrypt/MongoCryptContextCtor/cryptoCallbacks.
Fix: require('mongodb-client-encryption') -> require('mongodb') in init.js.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Downloads mongo_crypt_v1.so (Automatic Encryption Shared Library) during
image build so CSFLE works without a host-side volume mount.
## Build details
- Stage: alpine:3.21 fetches from downloads.mongodb.com, verifies SHA-256,
then COPYs the .so into the runtime node:alpine stage
- Version: enterprise-ubuntu2204-7.0.21 (compatible with mongodb-client-encryption@6.x)
- Checksums pinned (verified 2026-06-21):
amd64 25190407f7131989fdd9c113ef0aa4c7ef618cccee024b35e8de0aeaf3f74764
arm64 81bbf9120dd00e856a36d5f8234c88b0fdd4c9d6677a327e9047a7483a9e88d0
- gcompat + libgcc added to Alpine to provide glibc ABI compatibility for
the glibc-built .so running on musl
- ENV MONGO_CRYPT_SHARED_LIB_PATH=/app/lib/mongo_crypt_v1.so baked into image
## Operator changes
- No env var or mount changes required — crypt_shared is bundled automatically
- Override possible by volume-mounting a replacement .so at the same path
## Docs updated
- api/csfle/README.md: Mounted Paths section updated (bundled, not mounted)
- .env.example: MONGO_CRYPT_SHARED_LIB_PATH comment updated
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ompat
pthread_cond_clockwait (glibc 2.30+) is not implemented by Alpine's
gcompat shim, causing 'Error relocating mongo_crypt_v1.so: symbol not
found' at container startup.
Fix: replace node:24.16.0-alpine with node:24.16.0-bookworm-slim (Debian
Bookworm, native glibc 2.36) so all glibc symbols in mongo_crypt_v1.so
resolve correctly.
Changes:
- Base image: node:24.16.0-alpine -> node:24.16.0-bookworm-slim
- Package manager: apk -> apt-get (libjemalloc2, python3)
- jemalloc: symlinked to /usr/local/lib/libjemalloc.so.2 (arch-agnostic)
- uv source: uv:0.9.5-python3.12-alpine -> uv:0.9.5-python3.12-bookworm
- Removed: gcompat libgcc apk install (no longer needed)
LD_PRELOAD and MONGO_CRYPT_SHARED_LIB_PATH env vars unchanged.
Verify in running container:
ldd /app/lib/mongo_crypt_v1.so
node -e 'require("mongodb-client-encryption")'
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two bugs prevented existing plaintext documents from being encrypted during startup auto-migration: Bug 1 (Critical): replaceOne with partial projection destroyed documents backfillCollection projected only _id + target fields from the raw client, then called replaceOne() with that sparse doc. replaceOne replaces the ENTIRE document, so all other fields (conversationId, sender, model, createdAt, etc.) were silently deleted. Fix: use updateOne + $set to touch only the encrypted fields. All other document fields are preserved. Bug 2: migrated counter over-counted errors; APPLIED set regardless migrated += batch.length was outside the per-doc try/catch, so every error still incremented migrated and the policy was always marked APPLIED even when every write failed. Fix: increment migrated only on successful updateOne; mark FAILED when errors > 0. Additional improvements: - DB-level not-binData query filter so only plaintext docs are fetched. Empty strings correctly included (they are not null and not binData). Already-encrypted BinData blobs excluded at the DB level. - isEncrypted() checks both sub_type (bson v4) and subType (bson v5+) for driver compatibility. - CSFLE_FORCE_REMIGRATE=true env var to re-run already-APPLIED policies and encrypt docs missed by previous buggy runs (single-use). - Updated tests: spy on updateOne not replaceOne; new tests for empty-string handling, FORCE_REMIGRATE, and error->FAILED status. - .env.example: CSFLE_FORCE_REMIGRATE documented with usage warning. To encrypt existing plaintext docs: Set CSFLE_FORCE_REMIGRATE=true + CSFLE_AUTO_MIGRATE=true, restart once, then remove CSFLE_FORCE_REMIGRATE. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove multi-collection policy versioning and focus CSFLE exclusively
on messages.text and messages.content.
Encrypted fields (final scope):
messages.text - string - Random encryption
messages.content - array - Random encryption
## Files removed
- api/csfle/policies/index.js (policy registry + schema builder)
- api/csfle/policies/v1.js (sessions.refreshTokenHash, keys.value)
- api/csfle/policies/v2.js (messages + conversations.title)
- api/csfle/schemas.js (thin wrapper, no longer needed)
## Files created
- api/csfle/fields.js (static ENCRYPTED_FIELDS + buildSchemaMap)
## Files simplified
- api/csfle/deks.js (one DEK: dek-messages only)
- api/csfle/index.js (use buildSchemaMap from fields.js)
- api/csfle/manager.js (single MESSAGES_MIGRATION, no version registry;
keep __csfle_migrations idempotency, strict/warn,
CSFLE_FORCE_REMIGRATE, updateOne + $set)
- api/csfle/__tests__/manager.test.js (rewritten for simplified manager)
- api/csfle/README.md (updated: local + GCP setup, startup flow, verify steps)
- .env.example (remove CSFLE_MIGRATION_TARGET_VERSION)
## Behaviour changes
- CSFLE_MIGRATION_TARGET_VERSION removed (no version registry to target)
- CSFLE_FORCE_REMIGRATE retained (re-run migration for missed docs)
- CSFLE_STARTUP_POLICY (strict/warn) retained
- conversations.title no longer encrypted; MeiliSearch guard removed from
packages/data-schemas/src/models/convo.ts (guard remains in message.ts)
## Validation
- Node --check passes on all 6 changed JS files and test file
- No dead references to removed modules (grep confirms)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add CSFLE_GCP_SERVICE_ACCOUNT_FILE env var so a GCP service account
JSON key file can be used for CSFLE KMS authentication, enabling local
Docker deployments that cannot use Workload Identity.
Resolution order when GCP_KMS_PROJECT_ID is set:
1. CSFLE_GCP_SERVICE_ACCOUNT_FILE (CSFLE-specific, preferred)
2. GOOGLE_SERVICE_KEY_FILE (shared GCP key used by other integrations)
3. Neither set -> ADC / K8s Workload Identity (unchanged)
Error handling:
- File not found / unreadable -> explicit error naming env var + path
- Invalid JSON -> explicit error naming env var + path
- Missing client_email / private_key -> explicit error
Changes:
api/csfle/provider.js
- Add loadGcpCredentials() helper (exported for testing)
- buildKmsProviders() injects explicit gcp.email/privateKey when file found
- ADC path unchanged (kmsProviders.gcp = {}) when no file configured
api/csfle/__tests__/provider.sanity.js
- Standalone Node.js sanity test (no jest needed): 14 assertions
- Covers all resolution paths, error cases, local mode regression
api/csfle/README.md
- Updated GCP KMS setup section with auth options and docker-compose example
.env.example
- Added CSFLE_GCP_SERVICE_ACCOUNT_FILE with resolution order comment
docker-compose usage example:
environment:
CSFLE_GCP_SERVICE_ACCOUNT_FILE: /run/secrets/csfle-sa.json
volumes:
- ./secrets/csfle-sa.json:/run/secrets/csfle-sa.json:ro
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
GCP service account JSON files ship private_key as a full PEM block:
-----BEGIN RSA PRIVATE KEY-----
<base64 payload>
-----END RSA PRIVATE KEY-----
libmongocrypt kmsProviders.gcp.privateKey expects bare base64 only.
Passing the raw PEM string caused:
"Failed to parse KMS provider gcp: unable to parse base64 from UTF-8 field privateKey"
Fix: add normalisePemToBase64() in provider.js that:
1. Strips BEGIN/END header and footer lines (any key type)
2. Removes all whitespace (newlines, spaces, \r)
3. Validates the result decodes to a non-empty buffer
4. Returns bare base64 payload
Called automatically by loadGcpCredentials() before returning privateKey.
Bare base64 values (no PEM headers) pass through unchanged.
Errors include the env var name and file path for fast diagnosis:
[CSFLE] GCP service account file at /app/data/auth.json (from GOOGLE_SERVICE_KEY_FILE):
private_key is empty after stripping PEM headers
normalisePemToBase64 is exported for direct testing.
Tests: provider.sanity.js extended — 20/20 pass
- PEM strip (RSA + PKCS8 header variants)
- bare base64 passthrough
- JSON literal-backslash-n handling
- empty-after-strip error
- end-to-end: loadGcpCredentials returns stripped key from PEM file
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
pedrojreis
force-pushed
the
diogoserrano/nos-gpt-csfle-implementation
branch
from
d7f1695 to
8ba9650
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces support for MongoDB Client-Side Field Level Encryption (CSFLE) to LibreChat, ensuring sensitive message fields are encrypted transparently at the driver layer. It includes Dockerfile changes to bundle the required cryptographic shared library for CSFLE, a comprehensive README for setup and migration, and a robust test suite for the migration manager. The most important changes are grouped below:
CSFLE Integration and Documentation:
api/csfle/README.mdexplaining CSFLE, encrypted fields, environment variables, migration flow, GCP KMS integration, and operational caveats..env.examplewith new CSFLE-related environment variables, including flags for enabling encryption, migration behavior, key vault configuration, and KMS integration.Docker Support for CSFLE:
Dockerfileto download and verify themongo_crypt_v1.soshared library in a dedicated build stage, and switched the base image to Debian Bookworm (glibc) for compatibility./app/lib/mongo_crypt_v1.soand setMONGO_CRYPT_SHARED_LIB_PATHautomatically, ensuring CSFLE works out-of-the-box in Docker.Migration Manager Testing:
api/csfle/__tests__/manager.test.js) for the CSFLE migration manager, covering migration state tracking, idempotency, forced re-migration, error handling, and startup policies.Example of the field messages using GCP as the KMS:
