Skip to content

[WIP] Downgrade SQLite engine to 2026-05-12 build (012a04ed) to isolate OOM regression#2684

Draft
chiragsalian wants to merge 1 commit into
expensify_prodfrom
chirag/test-sqlite-downgrade-012a04ed
Draft

[WIP] Downgrade SQLite engine to 2026-05-12 build (012a04ed) to isolate OOM regression#2684
chiragsalian wants to merge 1 commit into
expensify_prodfrom
chirag/test-sqlite-downgrade-012a04ed

Conversation

@chiragsalian

@chiragsalian chiragsalian commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Explanation of Change

  • Reverts the SQLite amalgamation (libstuff/sqlite3.c, libstuff/sqlite3.h, libstuff/sqlite3ext.h) from the current build 45f6fccc (2026-07-09) back to 012a04ed (2026-05-12), taken from commit c6d195fd ("new sqlite version from dan"). Code taken from the before this June 5th PR.
  • SQLITE_VERSION is unchanged (3.54.0) — only the hctree/engine build (SOURCE_ID) changes. No other files touched.

Why

  • The monthly "Account Manager Retention Report" — a large read-only SELECT run via readdb/CustomQuery, doing ~12,763 full-table scans under BEGIN CONCURRENT — has run for ~3 years and now aborts with out-of-memory: {SQLITE} Code: 7, Message: statement aborts at ...: out of memory402 Bad query.
  • Fleet-wide CustomQuery out-of-memory events were 0 through June, onset 2026-07-06. App-side Bedrock changes in the window were cleared (logging / rate-limiting / write-path locking). The only memory-relevant change underneath this path is the SQLite/hctree engine, which was bumped 3× between the report's last success (012a04ed, May 12) and failure (69941fdc, Jun 17).
  • 012a04ed is the last engine build the report ran on cleanly, so this reverts to it.

Tests

  • Deploy this branch to a single non-critical follower (not leader / secondary-leader).
  • Re-run the report query on that node.
  • OOM disappears → confirms the engine bump is the cause. OOM persists → engine is exonerated; look elsewhere.

Caveat

  • The amalgamation is reverted ~2 months while keeping current Bedrock C++. Watch CI for any compile incompatibility.

…6-05-12) to isolate OOM regression

Reverts libstuff/sqlite3.{c,h} + sqlite3ext.h to the last engine build the
Account Manager Retention Report ran on without OOM. SQLITE_VERSION unchanged (3.54.0).
@chiragsalian chiragsalian self-assigned this Jul 16, 2026
@chiragsalian chiragsalian changed the title [Test / do not merge] Downgrade SQLite engine to 2026-05-12 build (012a04ed) to isolate OOM regression [WIP] Downgrade SQLite engine to 2026-05-12 build (012a04ed) to isolate OOM regression Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant