Skip to content

Commit 8a0f8db

Browse files
committed
feat(diagnostics): Add Helm upgrade failure triage
Add a HelmRelease diagnostic entrypoint and Helm-aware evidence metadata so agents can distinguish rollout evidence from missing Helm CLI or release manifest evidence. Document the user story for diagnosing Helm upgrade failures when users no longer have Helm CLI output, including validated kind scenarios and acceptance criteria.
1 parent 87e0258 commit 8a0f8db

11 files changed

Lines changed: 683 additions & 24 deletions

File tree

AI_CONTRACT.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -223,6 +223,20 @@ relationships use `provenance.sourceType=label_evidence`,
223223
`provenance.state=inferred`, and a numeric confidence. Treat them as strong or
224224
weak ownership hints until exact Helm release manifest evidence is collected.
225225

226+
For Helm upgrade failures, the diagnostic graph describes current Kubernetes
227+
state plus Helm metadata already visible in the cluster. It cannot observe Helm
228+
CLI stderr, chart rendering, values schema validation, repository/client
229+
authentication, hook execution semantics, or the original failure hidden by
230+
`--atomic` rollback unless the user provides that output. When Helm evidence is
231+
present, agents should expect:
232+
233+
- a `helm_cli_output_not_observed` warning
234+
- a `helm_manifest_evidence_not_collected` warning for label-derived ownership
235+
- `helm_cli_output` and `helm_release_manifest` entries in `degradedSources`
236+
- `HelmOwnershipEvidence` / `HelmChartEvidence` entries in `rankedEvidence`
237+
- `helm_ownership_conflict` conflicts when one resource points at multiple
238+
candidate releases
239+
226240
AI-Agent code should assume these meanings are stable, even if the set of returned nodes varies by policy.
227241

228242
## What AI-Agent code should treat as optional

Makefile

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ DAEMON_CONFIG_OVERRIDES = $(CLI_CONFIG_OVERRIDES) $(call make_arg,SERVER_ADDR,ad
4444
CLI_CONFIG_ARGS = $(if $(CONFIG),--config "$(CONFIG)" $(CLI_CONFIG_OVERRIDES),--kubeconfig "$(KUBECONFIG)" --cluster "$(CLUSTER)" --context-namespaces "$(CONTEXT_NAMESPACES)" --workload-resources "$(WORKLOAD_RESOURCES)" --controller-rules "$(CONTROLLER_RULES)" --bootstrap-timeout "$(BOOTSTRAP_TIMEOUT)")
4545
DAEMON_CONFIG_ARGS = $(if $(CONFIG),--config "$(CONFIG)" $(DAEMON_CONFIG_OVERRIDES),--kubeconfig "$(KUBECONFIG)" --cluster "$(CLUSTER)" --context-namespaces "$(CONTEXT_NAMESPACES)" --workload-resources "$(WORKLOAD_RESOURCES)" --controller-rules "$(CONTROLLER_RULES)" --addr "$(SERVER_ADDR)" --bootstrap-timeout "$(BOOTSTRAP_TIMEOUT)" --poll-interval "$(POLL_INTERVAL)")
4646

47-
.PHONY: build build-daemon build-viewer docker-build owl test verify ci ci-go ci-helm ci-binaries ci-client ci-visualize run serve status status-server list-entities-server get-entity-server list-relations-server neighbors-server expand-node-server collapse-node-graph observe-status diagnose-pod diagnose-workload diagnose-pod-server diagnose-workload-server visualize visualize-go visualize-check live-check verify-live require-kubeconfig require-entry
47+
.PHONY: build build-daemon build-viewer docker-build owl test verify ci ci-go ci-helm ci-binaries ci-client ci-visualize run serve status status-server list-entities-server get-entity-server list-relations-server neighbors-server expand-node-server collapse-node-graph observe-status diagnose-pod diagnose-workload diagnose-helm-release diagnose-pod-server diagnose-workload-server visualize visualize-go visualize-check live-check verify-live require-kubeconfig require-entry
4848

4949
build:
5050
mkdir -p bin
@@ -174,6 +174,15 @@ diagnose-workload: build require-kubeconfig require-entry
174174
--max-depth "$(MAX_DEPTH)" \
175175
--storage-max-depth "$(STORAGE_MAX_DEPTH)" $(make_diagnostic_budget_args)
176176

177+
diagnose-helm-release: build require-kubeconfig require-entry
178+
$(BINARY) \
179+
$(CLI_CONFIG_ARGS) \
180+
--diagnose-helm-release \
181+
--namespace "$(NAMESPACE)" \
182+
--name "$(NAME)" \
183+
--max-depth "$(MAX_DEPTH)" \
184+
--storage-max-depth "$(STORAGE_MAX_DEPTH)" $(make_diagnostic_budget_args)
185+
177186
diagnose-pod-server: build require-entry
178187
$(BINARY) \
179188
--server "$(SERVER_URL)" \

QUICKSTART.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -556,6 +556,22 @@ When resources carry standard Helm metadata, diagnostic graphs can also include
556556
`installs_chart` edges. These edges are label evidence with confidence scores,
557557
not exact Helm manifest membership.
558558

559+
For a failed Helm upgrade where the user does not have CLI output, start from
560+
the release and inspect current cluster evidence:
561+
562+
```bash
563+
kubernetes-ontology \
564+
--server "http://127.0.0.1:18080" \
565+
--diagnose-helm-release \
566+
--namespace default \
567+
--name my-release
568+
```
569+
570+
The response can identify release-owned resources, rollout blockers, Events,
571+
and probable chart ownership. It cannot observe Helm template, values,
572+
repository, client, hook, or `--atomic` rollback errors unless the user provides
573+
the Helm output.
574+
559575
Pod-centered diagnostic queries keep shared nodes bounded by default. For
560576
example, a pod's `ServiceAccount` is shown, but the traversal does not continue
561577
through that ServiceAccount to every other pod using it. Use

README.md

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,32 @@ Resources labeled with standard Helm metadata produce `HelmRelease` and
112112
`installs_chart` edges with `label_evidence` provenance and confidence scores.
113113
These are ownership hints from labels, not exact manifest membership.
114114

115+
### Helm Upgrade Failure Triage
116+
117+
When a user only says "helm upgrade failed" and does not have the Helm CLI
118+
output, `kubernetes-ontology` can still diagnose the current cluster state for
119+
that release:
120+
121+
```bash
122+
kubernetes-ontology \
123+
--server "http://127.0.0.1:18080" \
124+
--diagnose-helm-release \
125+
--namespace default \
126+
--name my-release
127+
```
128+
129+
The response expands the probable release-owned resources and chart evidence.
130+
It also marks the missing Helm-side evidence explicitly:
131+
132+
- `helm_cli_output_not_observed`: template, values, repository, client, hook,
133+
and `--atomic` rollback errors are outside current Kubernetes object state.
134+
- `helm_manifest_evidence_not_collected`: default Helm ownership is label and
135+
annotation evidence, not exact release manifest membership.
136+
137+
For rollout failures that reached the cluster, follow the release graph into
138+
the affected Workload or Pod diagnostic. For render/client failures, ask the
139+
user to paste the `helm upgrade` stderr or `helm status/history` output.
140+
115141
## Agent Onboarding
116142

117143
This repository provides a Codex-style skill:
@@ -390,6 +416,16 @@ Diagnose a pod:
390416
--max-edges 400
391417
```
392418

419+
Diagnose a Helm release after a failed upgrade:
420+
421+
```bash
422+
./bin/kubernetes-ontology \
423+
--server "http://127.0.0.1:18080" \
424+
--diagnose-helm-release \
425+
--namespace default \
426+
--name my-release
427+
```
428+
393429
Diagnostic responses include additive `partial`, `warnings`, `budgets`,
394430
`rankedEvidence`, `degradedSources`, and `conflicts` fields. Agents should use
395431
those fields to distinguish bounded evidence from complete cluster truth.

README.zh-CN.md

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,21 @@ Helm 关系会以 `managed_by_helm_release` 和 `installs_chart` 边呈现,
7070
provenance 是 `label_evidence`,并带 confidence。它们是标签证据下的归属
7171
提示,不等同于精确 manifest 成员关系。
7272

73+
如果用户说“helm upgrade 失败了”但没有 Helm CLI 输出,可以先从当前集群状态
74+
诊断 release:
75+
76+
```bash
77+
kubernetes-ontology \
78+
--server "http://127.0.0.1:18080" \
79+
--diagnose-helm-release \
80+
--namespace default \
81+
--name my-release
82+
```
83+
84+
返回结果会展示 release 可能管理的资源、chart 证据、rollout 阻塞点和事件。
85+
同时也会明确标记:template、values、repo/client、hook、`--atomic` rollback
86+
原始错误需要用户提供 Helm stderr/status/history 才能判断。
87+
7388
运行时支持:
7489

7590
- 启动时全量快照
@@ -313,6 +328,16 @@ OpenKruise,这是正常情况,不需要为此中断启动。
313328
--max-edges 400
314329
```
315330

331+
诊断 Helm release:
332+
333+
```bash
334+
./bin/kubernetes-ontology \
335+
--server "http://127.0.0.1:18080" \
336+
--diagnose-helm-release \
337+
--namespace default \
338+
--name my-release
339+
```
340+
316341
诊断返回会额外包含 `partial`、`warnings`、`budgets`、
317342
`rankedEvidence`、`degradedSources` 和 `conflicts`。Agent 应优先读取
318343
这些字段,区分“有边界的证据图”和“完整集群事实”。

cmd/kubernetes-ontology/main.go

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ func main() {
4949
var statusOnly bool
5050
var diagnosePod bool
5151
var diagnoseWorkload bool
52+
var diagnoseHelmRelease bool
5253
var observeDuration time.Duration
5354
var pollInterval time.Duration
5455
var streamMode string
@@ -92,6 +93,7 @@ func main() {
9293
flag.BoolVar(&statusOnly, "status", false, "Alias for --status-only")
9394
flag.BoolVar(&diagnosePod, "diagnose-pod", false, "Diagnose a Pod by --namespace and --name")
9495
flag.BoolVar(&diagnoseWorkload, "diagnose-workload", false, "Diagnose a Workload by --namespace and --name")
96+
flag.BoolVar(&diagnoseHelmRelease, "diagnose-helm-release", false, "Diagnose a Helm release by --namespace and --name")
9597
flag.DurationVar(&observeDuration, "observe-duration", 0, "Keep observing the cluster for this duration before printing status or query output")
9698
flag.DurationVar(&pollInterval, "poll-interval", 10*time.Second, "Polling interval used with --observe-duration")
9799
flag.StringVar(&streamMode, "stream-mode", string(collectk8s.StreamModePolling), "Continuous update stream mode for --observe-duration: informer or polling")
@@ -140,8 +142,14 @@ func main() {
140142
if terminalKindsDisable {
141143
expandTerminalNodes = true
142144
}
143-
if diagnosePod && diagnoseWorkload {
144-
err := fmt.Errorf("only one of --diagnose-pod or --diagnose-workload may be set")
145+
diagnoseModeCount := 0
146+
for _, enabled := range []bool{diagnosePod, diagnoseWorkload, diagnoseHelmRelease} {
147+
if enabled {
148+
diagnoseModeCount++
149+
}
150+
}
151+
if diagnoseModeCount > 1 {
152+
err := fmt.Errorf("only one of --diagnose-pod, --diagnose-workload, or --diagnose-helm-release may be set")
145153
if machineErrors {
146154
writeMachineError(os.Stderr, err)
147155
} else {
@@ -155,6 +163,9 @@ func main() {
155163
if diagnoseWorkload {
156164
entryKind = "Workload"
157165
}
166+
if diagnoseHelmRelease {
167+
entryKind = "HelmRelease"
168+
}
158169

159170
if collapseNode {
160171
if err := collapseGraphFile(graphFile, entityID); err != nil {

docs/design/README.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,9 @@ Recommended reading order:
1414
2. `current-state-and-next-steps.md` records the current implementation state.
1515
3. `open-source-mvp-plan.md` defines the open-source MVP boundary.
1616
4. `kubernetes-semantic-kernel-evolution.md` gives the long-term architecture.
17-
5. `continuous-runtime-technical-design.md` and
17+
5. `helm-upgrade-failure-user-story.md` standardizes the Helm upgrade failure
18+
scenario where the user no longer has Helm CLI output.
19+
6. `continuous-runtime-technical-design.md` and
1820
`continuous-runtime-progress-snapshot.md` capture runtime design history.
1921

2022
Detailed research notes are kept under `research/` so the design rationale and

0 commit comments

Comments
 (0)