IGNITE-28671 Describe healthy cluster behavior in general tips guide#13130
Open
w3ll1ngt wants to merge 11 commits into
Open
IGNITE-28671 Describe healthy cluster behavior in general tips guide#13130w3ll1ngt wants to merge 11 commits into
w3ll1ngt wants to merge 11 commits into
Conversation
zstan
reviewed
May 14, 2026
zstan
reviewed
May 14, 2026
zstan
reviewed
May 14, 2026
|
nizhikov
reviewed
Jun 3, 2026
There was a problem hiding this comment.
Pull request overview
This PR updates the general performance tips documentation to include guidance on assessing Apache Ignite cluster health, and adjusts the page title wording to “General Performance Tips”.
Changes:
- Rename the document title from “Generic Performance Tips” to “General Performance Tips”.
- Add a new “How to assess cluster health” section describing practical signals and checks (cluster state/baseline,
idle_verify, rebalance convergence, queues, transactions/queries, persistence checkpoint pressure, and logs).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
zstan
reviewed
Jul 15, 2026
copilot' grammar changes Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
zstan
reviewed
Jul 20, 2026
| * The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state. | ||
| * Baseline topology, when it is used or managed manually, matches the expected data-bearing server nodes. | ||
| * Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle. | ||
| * Expected nodes are present, and no node segmentation or repeated membership churn is reported. |
Contributor
There was a problem hiding this comment.
"repeated membership churn" - i don`t listen this term before, what does it mean ?
Contributor
Author
There was a problem hiding this comment.
churn-change describes sitations that happens in the set of participating nodes participating nodes due to joins, graceful leaves, and failures. For example, a high churn rate can increase costs or decrease service quality. src:
as for this article, i'll replace it with more common wording
| * Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle. | ||
| * Expected nodes are present, and no node segmentation or repeated membership churn is reported. | ||
| * link:configuring-caches/partition-loss-policy[Lost partitions] are absent. | ||
| * Long-running transactions, Partition Map Exchange (PME), rebalancing, checkpointing, and executor queues converge instead of accumulating. |
Contributor
There was a problem hiding this comment.
All this terms need to be covered\documentally extended somehow ? PME - what does it mean ? Long-running transactions - how it can be detected ? and so on ...
Contributor
Author
There was a problem hiding this comment.
there will be detailed explanation of this metrics later on in arictle, but here is the first use of it in text. so i've linked it to their doc origins
Comment on lines
+67
to
+69
| Use several signals together. | ||
| A simple client connection or SQL liveness check can prove only that a particular client or query path is reachable; it does not check user partitions, backup consistency, baseline membership, or all server nodes. | ||
| Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency check, but it is still not a complete health check. |
Contributor
There was a problem hiding this comment.
I misunderstand this section at all ( It gives nothing only confusing a reader, need to be refactored or removed.
Contributor
Author
There was a problem hiding this comment.
last line removed. finally
|
|
||
| [source,text] | ||
| ---- | ||
| LastCheckpointTotalPagesNumber |
Contributor
There was a problem hiding this comment.
plz also show "ignite.log" output for checkpoint operations, how it looks
| * repeated striped-pool starvation warnings; | ||
| * unexpected node restarts. | ||
|
|
||
| Ignite reports critical failures to the configured failure handler; depending on the handler, the result can be node invalidation, failover handling, node stop, or JVM termination. |
Contributor
There was a problem hiding this comment.
Oh ! seems we need to mention failure handler and it`s configurations policies here ? wdyt ?
Contributor
Author
There was a problem hiding this comment.
done with crosslinking
| Monitor membership from all expected monitoring points, and use logs and failure handling to detect segmentation and discovery failures. | ||
|
|
||
| CDC is a feature-specific check, not part of a universal cluster health definition. | ||
| If CDC is enabled, monitor `ignite-cdc.log`, WAL gaps, and consumption lag. |
Contributor
There was a problem hiding this comment.
theme is not covered at all )) may be it`s ok for this doc but if you have more info - plz append it
Co-authored-by: Evgeniy Stanilovskiy <stanilovsky@gmail.com>
zstan
reviewed
Jul 23, 2026
|
|
||
| == How to assess cluster health | ||
|
|
||
| Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention. |
Contributor
There was a problem hiding this comment.
Suggested change
| Cluster health is a complex thing. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention. | |
| Cluster health is a complex set of metrics and states that need to be analyzed together. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention. |
| * Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle. | ||
| * Expected nodes are present, and no unexpected repeated node `JOIN`, `LEFT`, or `FAIL` events or node segmentation are reported. | ||
| * link:configuring-caches/partition-loss-policy[Lost partitions] are absent. | ||
| * The number and age of link:tools/control-script#transaction-management[long-running transactions] do not keep increasing; active link:data-modeling/data-partitioning#partition-map-exchange[Partition Map Exchanges (PMEs)] complete; link:monitoring-metrics/new-metrics-system#monitoring-rebalancing[rebalancing] finishes; link:monitoring-metrics/new-metrics-system#monitoring-checkpointing-operations[checkpoint-related metrics] and link:monitoring-metrics/new-metrics#thread-pools[executor queue sizes] return to their usual ranges after the link:monitoring-metrics/new-metrics-system#monitoring-topology[topology] has stabilized and the application workload has returned to its expected level. |
Contributor
There was a problem hiding this comment.
Suggested change
| * The number and age of link:tools/control-script#transaction-management[long-running transactions] do not keep increasing; active link:data-modeling/data-partitioning#partition-map-exchange[Partition Map Exchanges (PMEs)] complete; link:monitoring-metrics/new-metrics-system#monitoring-rebalancing[rebalancing] finishes; link:monitoring-metrics/new-metrics-system#monitoring-checkpointing-operations[checkpoint-related metrics] and link:monitoring-metrics/new-metrics#thread-pools[executor queue sizes] return to their usual ranges after the link:monitoring-metrics/new-metrics-system#monitoring-topology[topology] has stabilized and the application workload has returned to its expected level. | |
| * The number and timeout of link:tools/control-script#transaction-management[long-running transactions] do not keep increasing; active link:data-modeling/data-partitioning#partition-map-exchange[Partition Map Exchanges (PMEs)] complete; link:monitoring-metrics/new-metrics-system#monitoring-rebalancing[rebalancing] finishes; link:monitoring-metrics/new-metrics-system#monitoring-checkpointing-operations[checkpoint-related metrics] and link:monitoring-metrics/new-metrics#thread-pools[executor queue sizes] return to their usual ranges after the link:monitoring-metrics/new-metrics-system#monitoring-topology[topology] has stabilized and the application workload has returned to its expected level. |
|
|
||
| The command compares partition update counters and partition hashes between primary and backup copies. | ||
| Run it only when data updates are stopped. | ||
| If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated. |
Contributor
There was a problem hiding this comment.
Suggested change
| If updates are active, the command can report false conflicts because copies are changing while hashes are being calculated. | |
| If updates are active, the command can report false positive conflicts because copies are changing while hashes are being calculated. |
| ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID; | ||
| ---- | ||
|
|
||
| In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows. |
Contributor
There was a problem hiding this comment.
Suggested change
| In steady state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows. | |
| In stable cluster state, this query usually should not return `MOVING`, `RENTING`, or `LOST` rows. |
| * `io.datastorage.LastCheckpointTotalPagesNumber` | ||
| * `io.datastorage.LastCheckpointFsyncDuration` | ||
|
|
||
| It looks like this in ignite.log |
Contributor
There was a problem hiding this comment.
Suggested change
| It looks like this in ignite.log | |
| Corresponding records in <SOME_formatting ? bold ? italic ?>ignite.log looks like as follows: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for submitting the pull request to the Apache Ignite.
In order to streamline the review of the contribution
we ask you to ensure the following steps have been taken:
The Contribution Checklist
The description explains WHAT and WHY was made instead of HOW.
The following pattern must be used:
IGNITE-XXXX Change summarywhereXXXX- number of JIRA issue.(see the Maintainers list)
the
green visaattached to the JIRA ticket (see tabPR Checkat TC.Bot - Instance 1 or TC.Bot - Instance 2)Notes
If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com #ignite channel.