NIP-44 encryption is wire-incompatible with nostr-js-sdk (TS) — NIP-17 DMs don't interoperate

[MastaP]

Summary

The Java SDK's NIP-44 encryption (src/main/java/org/unicitylabs/nostr/crypto/NIP44Encryption.java) is wire-incompatible with the TypeScript SDK's (@unicitylabs/nostr-js-sdk, src/crypto/nip44.ts). A NIP-44 payload produced by one cannot be decrypted by the other, so NIP-17 gift-wrapped private DMs (seals + gift wraps) do not interoperate between Java/Android clients and the TypeScript/Sphere ecosystem.

Since sphere-sdk → nostr-js-sdk is the deployed/canonical stack, the practical effect is that Java wallets cannot exchange encrypted DMs with Sphere wallets (and vice-versa).

Where the two agree

Key derivation is identical, so the incompatibility is isolated to the cipher + MAC:

• Conversation key: HKDF-SHA256(IKM=ECDH_x, salt=sorted-xonly-pubkeys(64B), info="nip44-v2", 32B) — Java deriveConversationKey (NIP44Encryption.java:176), TS deriveConversationKey.
• Per-message key: HKDF-SHA256(conversation_key, salt=nonce(24B), info="", 76B) → chacha_key[0:32], chacha_nonce[32:44], [44:76] — Java deriveMessageKey (:283), TS nip44.ts.
• 24-byte random nonce, 16-byte MAC, and identical payload framing version(0x02) || nonce(24) || ciphertext || mac(16).

Where they diverge (the bug)

	TypeScript (nostr-js-sdk, deployed)	Java (this repo)
Cipher	ChaCha20-Poly1305 AEAD (RFC 8439) — chacha20poly1305(key, nonce) (nip44.ts:234)	raw ChaCha20 — BouncyCastle ChaCha7539Engine (NIP44Encryption.java:295, called at :92)
MAC	Poly1305 tag produced by the AEAD (keyed from the ChaCha20 block-0 keystream); messageKey[44:76] is unused	HMAC-SHA256(messageKey[44:76], nonce ‖ ciphertext), truncated to 16 bytes (calculateMac, :307, called at :95)

Two independent consequences, either of which alone breaks interop:

1. Ciphertext bytes differ. RFC 8439 ChaCha20-Poly1305 reserves block counter 0 to generate the Poly1305 key and encrypts the plaintext starting at counter 1. ChaCha7539Engine.processBytes encrypts the plaintext starting at counter 0. So even the ciphertext differs for identical inputs.
2. MAC bytes differ. A Poly1305 tag can never equal a truncated HMAC-SHA256, and they are keyed differently (Poly1305 key derived from the ChaCha20 keystream vs. messageKey[44:76]).

Net result: decrypting a TS-produced payload in Java fails at MAC verification (decryptWithKey, :153–:156); the reverse fails at the TS AEAD Poly1305 check.

Why it went unnoticed

NIP44EncryptionTest.java only does self-round-trip (encrypt then decrypt within the same implementation). There are no cross-implementation test vectors, so neither SDK's tests ever exercise the other's wire format.

Note vs. the official spec

For reference, neither variant matches official NIP-44 v2 (32-byte nonce, conversation_key = hkdf_extract(shared_x, salt="nip44-v2"), raw ChaCha20, full 32-byte HMAC-SHA256(hmac_key, aad=nonce ‖ ciphertext)). The Java construction (ChaCha20 + HMAC) is structurally closer to the official spec than TS's AEAD — but the deployed ecosystem runs the TS variant, so TS is the de-facto wire format that must be matched.

Suggested fix

Align Java to the TS wire format (that is what's deployed):

• Replace chacha20Encrypt + calculateMac with a single ChaCha20-Poly1305 AEAD over the padded plaintext, using key = messageKey[0:32] and nonce = messageKey[32:44]. Drop the separate HMAC and stop consuming messageKey[44:76]; the AEAD's 16-byte Poly1305 tag is appended to the ciphertext, keeping the payload 0x02 || nonce(24) || ct+tag.
• Add cross-implementation golden vectors captured from nostr-js-sdk as a regression gate, so any future drift between the two SDKs fails CI.

Minor: the comments referencing "XChaCha20" (:81) and "Poly1305-style authentication" (:94) are misleading — the code uses IETF ChaCha20 (12-byte nonce) + HMAC-SHA256.