Skip to content

wallet: prototype Store-backed SQL runtime - #1299

Closed
yyforyongyu wants to merge 2 commits into
btcsuite:sql-port-stage2-waddrmgrfrom
yyforyongyu:salvage-wallet-adapter-poc
Closed

wallet: prototype Store-backed SQL runtime#1299
yyforyongyu wants to merge 2 commits into
btcsuite:sql-port-stage2-waddrmgrfrom
yyforyongyu:salvage-wallet-adapter-poc

Conversation

@yyforyongyu

Copy link
Copy Markdown
Collaborator

Change Description

This draft PR is stacked on #1296 and evaluates whether btcwallet's existing
wallet facade can operate directly over the backend-neutral SQL Stores.

It follows the compatibility-first approach:

Scope

The prototype covers:

  • Wallet creation, opening, locking, and passphrase lifecycle.
  • Accounts, scopes, address derivation, imports, and watch-only wallets.
  • Transaction ingestion, balances, UTXOs, leases, labels, and notifications.
  • Recovery, rescans, restart behavior, and reorg handling.
  • Transaction creation, signing, publishing, and PSBT operations.
  • SQLite and PostgreSQL manager and transaction Stores.
  • lnd-facing scope, account, key derivation, private-key, and ECDH operations.
  • SQLite contention, failure-injection, and synchronization benchmarks.

The prototype splits several long-running operations into read, compute, and
write phases:

  • Wallet-open KDF and address reconstruction occur after the Store view closes.
  • Chain hash, header, and filter RPCs occur outside Store transactions.
  • Coin selection, signing, script validation, and watcher registration occur
    outside Store transactions.
  • Final writes revalidate sync tips, UTXOs, and address indexes.

This materially improves SQLite lock behavior:

  • The wallet-open Store view at 1,000 keys falls from approximately 145.9 ms to
    3.0 ms.
  • Blocked RPC, strategy, signing, and watcher operations allow unrelated writes
    to finish in approximately 4.2-4.5 ms.
  • A mixed key-allocation/funding workload measures 9.1 ms p50, 20.4 ms p95,
    and 22.3 ms p99.
  • No SQLite busy errors or writer retries occur in the measured workload.

The branch contains 96 files with approximately 21,437 additions and 1,717
deletions.

Issues Found

The experiment demonstrates that adapting the existing wallet requires more
than replacing its storage operations.

An independent Store-callback audit found these remaining issues:

  • Recovery performs derivation inside replayable write callbacks.
  • Funding plan conflicts can replay caller policy, coin selection, signing,
    and script validation.
  • Funding does not revalidate the chain tip used for confirmation and coinbase
    maturity decisions.
  • Several ambiguous-commit checks verify the terminal sync tip without proving
    the complete atomic write set.
  • Passphrase changes and watch-only conversion can commit durably without
    publishing the corresponding live secret or cache state.
  • Retryable reads can mutate address and account caches before the transaction
    succeeds.
  • Forty-six Store callbacks retain derivation, decryption, cache mutation,
    logging, dynamic callback dispatch, or unbounded work.

Correcting these issues requires systematic prepare, commit, and reconcile
semantics across wallet lifecycle, recovery, imports, keys, transactions, and
notifications.

This removes the expected implementation-size and schedule advantage of the
compatibility-first approach.

Conclusion

The prototype proves that the existing wallet can operate over native SQL
storage without using SQLWallet or emulating walletdb buckets.

It also shows that making this architecture safe requires introducing
role-style transaction ownership throughout the legacy wallet. Continuing the
adapter would effectively rebuild the role-based persistence model behind the
existing facade while retaining duplicated KV and Store paths.

The role-based native SQL wallet is therefore the preferred production
direction. This branch is preserved as a decision PoC and as a source of
behavioral, contention, failure-injection, and synchronization acceptance tests.
It is not proposed as the production implementation.

Add an experimental Store-backed wallet path over the SQL manager and transaction stores. Cover lifecycle, recovery, funding, signing, PSBT, lnd-facing key operations, and transaction-boundary failure injection.

This preserves the decision PoC and its benchmarks for review. The adapter remains experimental and is not the selected production direction.
Separate database open, runtime activation, and full-block backlog catch-up. Compare deterministic 100, 1,000, and 10,000 block workloads with equivalent SQLite settings and semantic validation.
@saubyk saubyk added this to lnd v0.22 Jul 26, 2026
@github-project-automation github-project-automation Bot moved this to Backlog in lnd v0.22 Jul 26, 2026
@yyforyongyu yyforyongyu closed this Aug 3, 2026
@github-project-automation github-project-automation Bot moved this from Backlog to Done in lnd v0.22 Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants