KAFKA-20503: Add integration tests for transaction buffering - #22958
Open
nicktelford wants to merge 1 commit into
Open
KAFKA-20503: Add integration tests for transaction buffering#22958nicktelford wants to merge 1 commit into
nicktelford wants to merge 1 commit into
Conversation
We need to verify that, when transactional state stores are enabled and processing.guarantee is exactly_once_v2, records are buffered in the transaction buffer (invisible to a READ_COMMITTED reader) until the Streams commit cycle completes, and then committed to the store as expected. Reuses EosIntegrationTest, which already owns the EOS invariant machinery: a new shouldBufferStateStoreWritesUntilCommitUnderEos test drives three bursts of writes across a commit boundary and asserts the READ_COMMITTED/READ_UNCOMMITTED store views at each step via a new isolation-aware verifyStateStore/queryStateStore, waiting on the async buffer flush with a new waitForStateStore helper (context.commit() only requests a commit; the buffer flush happens afterwards). The transactional dimension is orthogonal to group protocol and processing-threads, so the new test and existing sparse transactional parameterization are each exercised via a single representative combination rather than the full matrix, to avoid unnecessary integration-test runtime. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
github-actions
Bot
added
streams
tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We need to verify that, when transactional state stores are enabled and
processing.guaranteeisexactly_once_v2, records are buffered in the transaction buffer (invisible to a READ_COMMITTED reader) until the Streams commit cycle completes, and then committed to the store as expected.This reuses
EosIntegrationTest, which already owns the EOS invariant machinery and atransactionalStateStoresparameter. A newshouldBufferStateStoreWritesUntilCommitUnderEostest drives three bursts of writes across a commit boundary and asserts the READ_COMMITTED/READ_UNCOMMITTED store views at each step, via a new isolation-awareverifyStateStore/queryStateStoreand awaitForStateStorehelper that polls for the buffer flush (context.commit()only requests a commit; the actual flush happens asynchronously afterwards).The transactional dimension is orthogonal to group protocol and processing-threads, so — consistent with this file's existing sparse-matrix convention — the new test and the existing transactional parameterization are each exercised via a single representative combination rather than the full matrix, to avoid unnecessary integration-test runtime.