KAFKA-20623: Update RPC for streams group topology description plugin (2/3)#22552
Open
frankvicky wants to merge 2 commits into
Open
KAFKA-20623: Update RPC for streams group topology description plugin (2/3)#22552frankvicky wants to merge 2 commits into
frankvicky wants to merge 2 commits into
Conversation
…on plugin (1/3) Wires the plugin reference into GroupCoordinatorService and adds the heartbeat-path gate that asks streams clients to push their topology description. Plugin lookup is resolved internally from groupCoordinatorConfig (no BrokerServer change).
… (2/3) Adds the StreamsGroupTopologyDescriptionUpdate RPC handler stacked on the heartbeat extension (1/3). The push pipeline runs through TopologyDescriptionManager.handleSetTopology: validate (group, member), convert wire payload, call plugin.setTopology, persist StoredDescriptionTopologyEpoch on success or FailedDescriptionTopologyEpoch on permanent failure, and centralise back-off state mutations in a single whenComplete.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JIRA: KAFKA-20623
This is a part of KIP-1331
This PR shouldn't be merge before #22551
Adds the StreamsGroupTopologyDescriptionUpdate RPC handler stacked on
the heartbeat extension (1/3). The push pipeline runs through
TopologyDescriptionManager.handleSetTopology:validate (group, member), convert wire payload, callplugin.setTopology, persistStoredDescriptionTopologyEpochon success orFailedDescriptionTopologyEpochon permanent failure, and centraliseback-off state mutations in a single
whenComplete.Push pipeline on the manager
TopologyDescriptionManagergains the push entry points:preCheckTopologyDescriptionUpdate— synchronous structuralvalidation that rejects
the request with
UNSUPPORTED_VERSIONwhen no plugin is configured,INVALID_REQUESTfor empty
MemberId/GroupIdor nullTopologyDescription.handleSetTopology— runsvalidateStreamsGroupMemberfirst (so afenced caller gets
UNKNOWN_MEMBER_IDrather than anINVALID_REQUESTfrom a payloadconversion
failure), then converts the wire payload to the broker-side POJO, then
calls the
plugin. Plugin success writes a metadata record advancing
StoredDescriptionTopologyEpoch; aStreamsTopologyDescriptionPermanentFailureException(or a synchronous throw) writes
FailedDescriptionTopologyEpoch; anyother exception
is treated as transient and writes no record.
Back-off mutation point
All back-off mutations on the push path are folded into a single
whenComplete,driven by a
BackoffActionholder populated by each terminal branch.The default is
ARM; only success and permanent-failure branches setCLEAR. Apost-plugin write
failure therefore re-arms the back-off and the next heartbeat sees the
drift and
re-solicits an idempotent re-push, matching the KIP-1331 invariant.
Wire ↔ POJO converter
StreamsGroupTopologyDescriptionConvertertranslatesStreamsGroupTopologyDescriptionUpdateRequestData.TopologyDescriptioninto the
broker-side
StreamsGroupTopologyDescriptionPOJO. Subtopology nodeordering and
string-collection iteration order are preserved via
LinkedHashSet, soa downstream
pretty-printer (e.g. the future
kafka-streams-groups.sh --topologycommand) can
reproduce the source ordering.
Wire-level node types (
SOURCE=1,PROCESSOR=2,SINK=3) map to thesealed
Nodehierarchy.GlobalStoreshape is validated; a malformedpayload throws
InvalidRequestException. The POJO itself dropsSet.copyOfin favour ofCollections.unmodifiableSetto preserve theconverter's insertion order — production callers only ever construct
nodes from a fresh
LinkedHashSetso this is not a defensive-copyregression.
Coordinator interface and service entry
GroupCoordinator.streamsGroupTopologyDescriptionUpdate(context, request)interfacemethod added.
GroupCoordinatorService.streamsGroupTopologyDescriptionUpdate(...)short-circuits
on a non-active coordinator with
COORDINATOR_NOT_AVAILABLE, thendelegates to
topologyDescriptionManager.preCheckTopologyDescriptionUpdate(...)and
handleSetTopology(...). Unhandled exceptions are translated byhandleOperationExceptioninto the wire error response.Coordinator shard + GMM
GroupMetadataManager.streamsGroupSetTopologyDescriptionEpoch(groupId, pushedEpoch, permanentFailure)— writes a
StreamsGroupMetadatarecord advancingStoredDescriptionTopologyEpoch(on success) or
FailedDescriptionTopologyEpoch(on permanentfailure).
GroupCoordinatorShard.streamsGroupSetTopologyDescriptionEpoch(...)exposes the
method to the runtime write-operation scheduler.
topologyDescriptionManager.preCheckTopologyDescriptionUpdate(...)and
handleSetTopology(...). Unhandled exceptions are translated byhandleOperationExceptioninto the wire error response.Coordinator shard + GMM
GroupMetadataManager.streamsGroupSetTopologyDescriptionEpoch(groupId, pushedEpoch, permanentFailure)— writes a
StreamsGroupMetadatarecord advancingStoredDescriptionTopologyEpoch(on success) or
FailedDescriptionTopologyEpoch(on permanentfailure).
GroupCoordinatorShard.streamsGroupSetTopologyDescriptionEpoch(...)exposes the
method to the runtime write-operation scheduler.