Skip to content

Commit 7dca5ff

Browse files
qu0bqu0b
committed
feat(panda-chat): split panda-server into a privileged sidecar container
Two-container pod: hermes always runs unprivileged (uid 10000, caps dropped) and panda-server + dockerd move to a privileged sidecar that is the only container holding PANDA_BOT_USERNAME/TOKEN — the credential gets its own Secret so hermes' envFrom can no longer expose it to LLM-driven shell execution. Hermes reaches the sidecar on 127.0.0.1:2480 via the shared pod netns; sidecar probes are exec-based (server binds loopback). Resources split: .resources -> hermes, panda.resources -> sidecar.
1 parent 976378d commit 7dca5ff

6 files changed

Lines changed: 115 additions & 68 deletions

File tree

‎charts/panda-chat/README.md‎

Lines changed: 13 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -26,13 +26,17 @@ AI chat for an Ethereum devnet — an Open-WebUI front end backed by a NousResea
2626
`observability/langfuse` plugin — one span per turn, one generation per LLM
2727
call, one observation per tool call; traces tagged with the devnet name.
2828

29-
When `panda.enabled` is true the agent pod runs `panda-server` + `dockerd`
30-
alongside Hermes and is **privileged** (dockerd needs root). The bot identity
31-
for the proxy is an Authentik **service account** (e.g. `panda-chat-svc` with a
29+
When `panda.enabled` is true the pod gains a separate **`panda-server`
30+
sidecar container** (`panda-server` + `dockerd`, privileged — dockerd needs
31+
root); the `hermes` container always runs unprivileged. The bot identity for
32+
the proxy is an Authentik **service account** (e.g. `panda-chat-svc` with a
3233
non-expiring app password) supplied via `credentials.panda.botUsername` /
33-
`credentials.panda.botToken`. panda-server mints proxy access tokens on demand
34-
with the OAuth2 `client_credentials` grant and keeps them in memory only — no
35-
seeded credential files, no refresh-token rotation.
34+
`credentials.panda.botToken` and materialized in a **dedicated Secret that only
35+
the sidecar mounts** — Hermes executes LLM-driven shell commands, so it never
36+
shares an environment with the bot credential. panda-server mints proxy access
37+
tokens on demand with the OAuth2 `client_credentials` grant and keeps them in
38+
memory only — no seeded credential files, no refresh-token rotation. Hermes
39+
reaches the sidecar on `127.0.0.1:2480` (shared pod network namespace).
3640

3741
## Access control
3842

@@ -154,17 +158,18 @@ open-webui:
154158
| open-webui.websocket.enabled | bool | `false` | |
155159
| open-webui.websocket.redis.enabled | bool | `false` | |
156160
| panda.clientId | string | `"panda-proxy"` | OAuth client id at the proxy |
157-
| panda.enabled | bool | `true` | Enable the panda sidecar processes + privileged pod |
161+
| panda.enabled | bool | `true` | Enable the panda-server sidecar container (privileged; the hermes container is not) |
158162
| panda.issuerUrl | string | `"https://authentik.analytics.production.platform.ethpandaops.io/application/o/panda-proxy/"` | Authentik application issuer the bot service account mints client_credentials tokens against (the trailing slash is part of the issuer — keep it) |
159163
| panda.proxyUrl | string | `"https://panda-proxy.analytics.production.platform.ethpandaops.io"` | Hosted panda-proxy URL (analytics data plane) |
164+
| panda.resources | object | `{"limits":{"cpu":"2000m","memory":"4Gi"},"requests":{"cpu":"200m","memory":"512Mi"}}` | Resources for the panda-server sidecar (panda-server + dockerd + sandboxes) |
160165
| panda.sandboxImage | string | `"ethpandaops/panda:sandbox-v0.31.0"` | Sandbox container image panda-server spawns for Python execution |
161166
| panda.storageDriver | string | `"overlay2"` | dockerd storage driver (overlay2; set to vfs if overlayfs is unavailable in-pod) |
162167
| persistence.accessModes | list | `["ReadWriteOnce"]` | Access modes |
163168
| persistence.enabled | bool | `true` | Enable a PVC for /opt/data (Hermes state + panda config/creds/storage) |
164169
| persistence.existingClaim | string | `""` | Use an existing claim instead of creating one |
165170
| persistence.size | string | `"8Gi"` | PVC size |
166171
| persistence.storageClass | string | `""` | Storage class (cluster default when empty) |
167-
| resources | object | `{"limits":{"cpu":"3000m","memory":"6Gi"},"requests":{"cpu":"300m","memory":"1Gi"}}` | Resources for the agent pod (Hermes + panda-server + dockerd + sandboxes) |
172+
| resources | object | `{"limits":{"cpu":"1000m","memory":"2Gi"},"requests":{"cpu":"200m","memory":"768Mi"}}` | Resources for the hermes container (panda-server sidecar sized separately under `panda.resources`) |
168173
| service.port | int | `8642` | Agent service port (Hermes OpenAI-compatible API) |
169174
| service.type | string | `"ClusterIP"` | Agent service type |
170175
| serviceAccount.annotations | object | `{}` | Annotations to add to the service account |

‎charts/panda-chat/README.md.gotmpl‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -23,13 +23,17 @@
2323
`observability/langfuse` plugin — one span per turn, one generation per LLM
2424
call, one observation per tool call; traces tagged with the devnet name.
2525

26-
When `panda.enabled` is true the agent pod runs `panda-server` + `dockerd`
27-
alongside Hermes and is **privileged** (dockerd needs root). The bot identity
28-
for the proxy is an Authentik **service account** (e.g. `panda-chat-svc` with a
26+
When `panda.enabled` is true the pod gains a separate **`panda-server`
27+
sidecar container** (`panda-server` + `dockerd`, privileged — dockerd needs
28+
root); the `hermes` container always runs unprivileged. The bot identity for
29+
the proxy is an Authentik **service account** (e.g. `panda-chat-svc` with a
2930
non-expiring app password) supplied via `credentials.panda.botUsername` /
30-
`credentials.panda.botToken`. panda-server mints proxy access tokens on demand
31-
with the OAuth2 `client_credentials` grant and keeps them in memory only — no
32-
seeded credential files, no refresh-token rotation.
31+
`credentials.panda.botToken` and materialized in a **dedicated Secret that only
32+
the sidecar mounts** — Hermes executes LLM-driven shell commands, so it never
33+
shares an environment with the bot credential. panda-server mints proxy access
34+
tokens on demand with the OAuth2 `client_credentials` grant and keeps them in
35+
memory only — no seeded credential files, no refresh-token rotation. Hermes
36+
reaches the sidecar on `127.0.0.1:2480` (shared pod network namespace).
3337

3438
## Access control
3539

‎charts/panda-chat/templates/_helpers.tpl‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,13 +65,22 @@ Agent image reference. Tag defaults to the chart appVersion.
6565
{{- end }}
6666

6767
{{/*
68-
Secret holding the Hermes bearer (API_SERVER_KEY), the LLM model key and
69-
the panda bot credentials.
68+
Secret holding the Hermes bearer (API_SERVER_KEY) and the LLM model key.
69+
Mounted ONLY into the unprivileged hermes container.
7070
*/}}
7171
{{- define "panda-chat.secretName" -}}
7272
{{- printf "%s-secret" (include "panda-chat.hermesFullname" .) | trunc 63 | trimSuffix "-" }}
7373
{{- end }}
7474

75+
{{/*
76+
Secret holding the panda bot credentials (PANDA_BOT_USERNAME/TOKEN).
77+
Mounted ONLY into the panda-server sidecar — never into hermes, which
78+
executes LLM-driven shell commands and must not be able to read it.
79+
*/}}
80+
{{- define "panda-chat.pandaSecretName" -}}
81+
{{- printf "%s-panda-secret" (include "panda-chat.hermesFullname" .) | trunc 63 | trimSuffix "-" }}
82+
{{- end }}
83+
7584
{{- define "panda-chat.configMapName" -}}
7685
{{- printf "%s-config" (include "panda-chat.hermesFullname" .) | trunc 63 | trimSuffix "-" }}
7786
{{- end }}

‎charts/panda-chat/templates/deployment.yaml‎

Lines changed: 42 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -20,17 +20,11 @@ spec:
2020
{{- with .Values.imagePullSecrets }}
2121
imagePullSecrets: {{- toYaml . | nindent 8 }}
2222
{{- end }}
23-
{{- if .Values.panda.enabled }}
24-
# panda: dockerd needs root, so only fsGroup applies (lets the init containers write the PVC).
23+
# Per-container security contexts below; fsGroup lets every container
24+
# (and the init containers) share the PVC. Hermes is ALWAYS unprivileged
25+
# — only the panda-server sidecar is privileged (dockerd).
2526
securityContext:
2627
fsGroup: 10000
27-
{{- else }}
28-
securityContext:
29-
runAsNonRoot: true
30-
runAsUser: 10000
31-
runAsGroup: 10000
32-
fsGroup: 10000
33-
{{- end }}
3428
initContainers:
3529
# Seed Hermes (and panda) config onto the PVC.
3630
- name: seed-config
@@ -62,19 +56,19 @@ spec:
6256
- name: hermes
6357
image: {{ include "panda-chat.image" . }}
6458
imagePullPolicy: {{ .Values.image.pullPolicy }}
65-
{{- if .Values.panda.enabled }}
66-
# privileged for dockerd; the image ENTRYPOINT starts dockerd + panda-server before Hermes.
67-
securityContext:
68-
privileged: true
69-
{{- else }}
59+
# Unprivileged, always: this container executes LLM-driven shell
60+
# commands. It carries no bot credentials and no docker socket; it
61+
# reaches panda-server over 127.0.0.1 (shared pod netns).
7062
securityContext:
63+
runAsNonRoot: true
64+
runAsUser: 10000
65+
runAsGroup: 10000
7166
allowPrivilegeEscalation: false
7267
readOnlyRootFilesystem: false
7368
capabilities:
7469
drop: [ALL]
75-
command: ["/opt/hermes/docker/entrypoint.sh"]
70+
command: ["/opt/panda/entrypoint.sh"]
7671
args: ["gateway", "run"]
77-
{{- end }}
7872
env:
7973
- {name: API_SERVER_HOST, value: "0.0.0.0"}
8074
- {name: API_SERVER_PORT, value: "8642"}
@@ -88,7 +82,6 @@ spec:
8882
- {name: HERMES_UID, value: "10000"}
8983
- {name: HERMES_GID, value: "10000"}
9084
{{- if .Values.panda.enabled }}
91-
- {name: DOCKERD_STORAGE_DRIVER, value: {{ .Values.panda.storageDriver | quote }}}
9285
- {name: PANDA_SERVER_URL, value: "http://127.0.0.1:2480"}
9386
{{- end }}
9487
# Devnet context for the agent + skills.
@@ -108,30 +101,13 @@ spec:
108101
- {name: HERMES_LANGFUSE_ENV, value: {{ .Values.langfuse.env | default .Values.network | quote }}}
109102
{{- end }}
110103
envFrom:
104+
# Hermes secret only (bearer, LLM key, langfuse) — the panda bot
105+
# credential lives in its own Secret on the sidecar.
111106
- secretRef:
112107
name: {{ include "panda-chat.secretName" . }}
113108
optional: true
114109
ports:
115110
- {name: http, containerPort: 8642, protocol: TCP}
116-
{{- if .Values.panda.enabled }}
117-
# ~5 min budget: dockerd + sandbox image pull + panda-server + hermes.
118-
startupProbe:
119-
exec:
120-
command:
121-
- sh
122-
- -c
123-
- curl -sf http://127.0.0.1:2480/health && curl -sf http://127.0.0.1:8642/health
124-
periodSeconds: 5
125-
failureThreshold: 60
126-
readinessProbe:
127-
httpGet: {path: /health, port: http}
128-
initialDelaySeconds: 60
129-
periodSeconds: 5
130-
livenessProbe:
131-
httpGet: {path: /health, port: http}
132-
initialDelaySeconds: 120
133-
periodSeconds: 30
134-
{{- else }}
135111
readinessProbe:
136112
httpGet: {path: /health, port: http}
137113
initialDelaySeconds: 10
@@ -140,11 +116,40 @@ spec:
140116
httpGet: {path: /health, port: http}
141117
initialDelaySeconds: 30
142118
periodSeconds: 30
143-
{{- end }}
144119
resources: {{- toYaml .Values.resources | nindent 12 }}
145120
volumeMounts:
146121
- {name: data, mountPath: /opt/data}
147122
- {name: config, mountPath: /config-src, readOnly: true}
123+
{{- if .Values.panda.enabled }}
124+
# panda-server + dockerd sidecar. Privileged (dockerd) and the ONLY
125+
# container holding the bot credential. Same image, "panda-stack" arg.
126+
- name: panda-server
127+
image: {{ include "panda-chat.image" . }}
128+
imagePullPolicy: {{ .Values.image.pullPolicy }}
129+
securityContext:
130+
privileged: true
131+
command: ["/opt/panda/entrypoint.sh"]
132+
args: ["panda-stack"]
133+
env:
134+
- {name: DOCKERD_STORAGE_DRIVER, value: {{ .Values.panda.storageDriver | quote }}}
135+
envFrom:
136+
- secretRef:
137+
name: {{ include "panda-chat.pandaSecretName" . }}
138+
# exec probes: panda-server binds 127.0.0.1, which kubelet httpGet
139+
# (pod IP) can't reach. ~5 min budget: dockerd + sandbox pull + server.
140+
startupProbe:
141+
exec:
142+
command: ["sh", "-c", "curl -sf http://127.0.0.1:2480/health"]
143+
periodSeconds: 5
144+
failureThreshold: 60
145+
livenessProbe:
146+
exec:
147+
command: ["sh", "-c", "curl -sf http://127.0.0.1:2480/health"]
148+
periodSeconds: 30
149+
resources: {{- toYaml .Values.panda.resources | nindent 12 }}
150+
volumeMounts:
151+
- {name: data, mountPath: /opt/data}
152+
{{- end }}
148153
volumes:
149154
- name: config
150155
configMap:

‎charts/panda-chat/templates/secret.yaml‎

Lines changed: 22 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,14 @@
11
{{/*
2-
Holds: API_SERVER_KEY (Hermes bearer, generated once and preserved),
3-
<llm.apiKeyEnv> (model key), PANDA_BOT_USERNAME / PANDA_BOT_TOKEN (the
4-
Authentik service-account identity panda-server mints client_credentials
5-
tokens with) and HERMES_LANGFUSE_* (tracing keys).
2+
Two Secrets with a deliberate trust boundary between them:
3+
4+
- <hermes>-secret: API_SERVER_KEY (Hermes bearer, generated once and
5+
preserved), <llm.apiKeyEnv> (model key) and HERMES_LANGFUSE_* (tracing
6+
keys). envFrom'd ONLY into the unprivileged hermes container.
7+
- <hermes>-panda-secret: PANDA_BOT_USERNAME / PANDA_BOT_TOKEN (the
8+
Authentik service-account identity panda-server mints client_credentials
9+
tokens with). envFrom'd ONLY into the panda-server sidecar. Hermes
10+
executes LLM-driven shell commands, so it must never share an
11+
environment (or container) with the bot credential.
612
*/}}
713
{{- $secretName := include "panda-chat.secretName" . -}}
814
{{- $existing := lookup "v1" "Secret" .Release.Namespace $secretName -}}
@@ -24,10 +30,6 @@ stringData:
2430
{{- with .Values.credentials.llmApiKey }}
2531
{{ $.Values.llm.apiKeyEnv }}: {{ . | quote }}
2632
{{- end }}
27-
{{- if .Values.panda.enabled }}
28-
PANDA_BOT_USERNAME: {{ required "credentials.panda.botUsername is required when panda.enabled" .Values.credentials.panda.botUsername | quote }}
29-
PANDA_BOT_TOKEN: {{ required "credentials.panda.botToken is required when panda.enabled" .Values.credentials.panda.botToken | quote }}
30-
{{- end }}
3133
{{- if .Values.langfuse.enabled }}
3234
{{- with .Values.credentials.langfuse.publicKey }}
3335
HERMES_LANGFUSE_PUBLIC_KEY: {{ . | quote }}
@@ -36,3 +38,15 @@ stringData:
3638
HERMES_LANGFUSE_SECRET_KEY: {{ . | quote }}
3739
{{- end }}
3840
{{- end }}
41+
{{- if .Values.panda.enabled }}
42+
---
43+
apiVersion: v1
44+
kind: Secret
45+
metadata:
46+
name: {{ include "panda-chat.pandaSecretName" . }}
47+
labels: {{- include "panda-chat.labels" . | nindent 4 }}
48+
type: Opaque
49+
stringData:
50+
PANDA_BOT_USERNAME: {{ required "credentials.panda.botUsername is required when panda.enabled" .Values.credentials.panda.botUsername | quote }}
51+
PANDA_BOT_TOKEN: {{ required "credentials.panda.botToken is required when panda.enabled" .Values.credentials.panda.botToken | quote }}
52+
{{- end }}

‎charts/panda-chat/values.yaml‎

Lines changed: 17 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -43,9 +43,11 @@ systemPrompt: |
4343
skill, and join the network with the `join-devnet` skill. Be concise and
4444
always scope data queries to this devnet.
4545
46-
# Panda integration. When enabled the agent pod runs panda-server + dockerd and is privileged.
46+
# Panda integration. When enabled the pod gains a separate privileged
47+
# "panda-server" sidecar container (panda-server + dockerd) holding the bot
48+
# credential; the hermes container stays unprivileged and credential-free.
4749
panda:
48-
# -- Enable the panda sidecar processes + privileged pod
50+
# -- Enable the panda-server sidecar container (privileged; the hermes container is not)
4951
enabled: true
5052
# -- Hosted panda-proxy URL (analytics data plane)
5153
proxyUrl: "https://panda-proxy.analytics.production.platform.ethpandaops.io"
@@ -59,6 +61,14 @@ panda:
5961
sandboxImage: "ethpandaops/panda:sandbox-v0.31.0"
6062
# -- dockerd storage driver (overlay2; set to vfs if overlayfs is unavailable in-pod)
6163
storageDriver: overlay2
64+
# -- Resources for the panda-server sidecar (panda-server + dockerd + sandboxes)
65+
resources:
66+
requests:
67+
cpu: 200m
68+
memory: 512Mi
69+
limits:
70+
cpu: 2000m
71+
memory: 4Gi
6272

6373
# LLM-call tracing via Hermes' bundled observability/langfuse plugin.
6474
# Keys come from credentials.langfuse; fail-open if absent.
@@ -105,14 +115,14 @@ credentials:
105115
# -- Langfuse secret key (sk-lf-...)
106116
secretKey: ""
107117

108-
# -- Resources for the agent pod (Hermes + panda-server + dockerd + sandboxes)
118+
# -- Resources for the hermes container (panda-server sidecar sized separately under `panda.resources`)
109119
resources:
110120
requests:
111-
cpu: 300m
112-
memory: 1Gi
121+
cpu: 200m
122+
memory: 768Mi
113123
limits:
114-
cpu: 3000m
115-
memory: 6Gi
124+
cpu: 1000m
125+
memory: 2Gi
116126

117127
persistence:
118128
# -- Enable a PVC for /opt/data (Hermes state + panda config/creds/storage)

0 commit comments

Comments
 (0)