KAFKA-20623: Heartbeat extension for streams group topology description plugin (1/3)#22551
Open
frankvicky wants to merge 1 commit into
Open
KAFKA-20623: Heartbeat extension for streams group topology description plugin (1/3)#22551frankvicky wants to merge 1 commit into
frankvicky wants to merge 1 commit into
Conversation
…on plugin (1/3) Wires the plugin reference into GroupCoordinatorService and adds the heartbeat-path gate that asks streams clients to push their topology description. Plugin lookup is resolved internally from groupCoordinatorConfig (no BrokerServer change).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JIRA: KAFKA-20623 This PR is a part of KIP-1331
Wires the plugin reference into
GroupCoordinatorServiceand adds theheartbeat-path gate that asks streams clients to push their topology
description.
Plugin reference on the service
GroupCoordinatorService.Builder.build()resolves the plugininternally via
config.streamsGroupTopologyDescriptionPlugin(Map.of())Optional<StreamsGroupTopologyDescriptionPlugin>andhands it to a new
TopologyDescriptionManagerthat owns the pluginreference and the
per-group push back-off.
Heartbeat post-processing
StreamsGroupHeartbeatResultcarries three epoch fields now —currentTopologyEpoch,storedDescriptionTopologyEpoch,failedDescriptionTopologyEpoch— so the service-layer gate candecide whether to set
TopologyDescriptionRequired=truewithout re-reading the group onevery heartbeat.
GroupMetadataManagerbuilds the heartbeat result with these fieldsat the four
existing
StreamsGroupHeartbeatResultconstruction sites.TopologyDescriptionManager.maybeSetTopologyDescriptionRequired(...)runs in the
.thenApply(...)after the heartbeat write. The flag is set when theplugin is
configured, the response has no error, the current epoch is resolved,
that epoch is
neither stored nor permanently failed at the plugin, the response does
not carry a
STALE_TOPOLOGYstatus, and the per-group back-off window is not ineffect.
Back-off
StreamsGroupTopologyDescriptionBackoffis a broker-level, per-groupexponential
back-off (30 s → 1 h, doubled on each arm at the same topology epoch,
reset on
topology-epoch advance).
armIfNotActivecomputeso twoconcurrent heartbeats for the same group cannot both arm the back-off
and double its
window beyond the intended length.
scratch on broker
restart. Convergence after a restart is driven by the persisted
StoredDescriptionTopologyEpoch/FailedDescriptionTopologyEpochfields on each
streams group.
clear(...)andarmOrExtend(...)sites consumed by the push andDeleteGroups
paths land in the follow-up PRs along with their respective entry
points on
TopologyDescriptionManager.