feat(pod): allow Karpenter to consume open ODCR and also reserved ids

[elgalu]

Karpenter and EC2 Capacity Reservations (ODCRs): what we shipped, and the better follow-up

The problem

We buy On-Demand Capacity Reservations (ODCRs) for CyberWeek or GPU serving endpoints but Karpenter-launched nodes never consumed them: the reservation sat idle-but-billed while the pod ran on regular on-demand capacity.

Root cause: Karpenter's native ODCR support (the ReservedCapacity feature gate, enabled fleet-wide as a side effect of the Karpenter v1.8.0 bump in October 2025) launches every on-demand and spot node with CapacityReservationPreference: none, which opts the instance out of every open ODCR. Upstream does this deliberately so its own reservation accounting stays correct, but nothing at Zalando used what the gate enables (no NodeClass declared a reservation, no NodePool offered the reserved capacity type), so the bump silently ended open-ODCR consumption for the whole fleet.

What we merged (the gate-off route, live today)

Three PRs make the gate a per-cluster choice and turn it off where we buy open ODCRs:

• kubernetes-on-aws #11614: new cluster config item karpenter_feature_reserved_capacity (default true, so no cluster changes silently), templated into Karpenter's FEATURE_GATES env.
• teapot-cluster-config-manager #1759: sets the item to false on deepthought-euc1-test.
• teapot-cluster-config-manager #1760: same for deepthought-euc1 (prod; takes effect when the kubernetes-on-aws change promotes through the stable channel).

With the gate off, Karpenter reverts to its pre-October behavior: it stops writing CapacityReservationPreference: none, and an open ODCR is consumed automatically by any matching instance (same type, same AZ), exactly as for a hand-started EC2 instance. Verified live on 2026-07-10: after replacing the pre-rollout node, the zmclip g6.4xlarge landed in cr-00bab72b5a26ddc4a (available: 0).

What this route gives up, deliberately: Karpenter stays blind to reservations. It will not prefer the reservation's AZ (the pod spec must pin instance type + AZ), will not scale into it preferentially, and cannot use targeted reservations at all, which rules out Capacity Blocks for ML. There is also no way to select one specific reservation when two identical ones exist: AWS picks.

The better follow-up (native reservations, 2 PRs prepared)

Two prepared PRs keep the gate on and give Karpenter the reservations to model, which is upstream's intended design:

• kubernetes-on-aws (branch karpenter-capacity-reservation-discovery): the EC2NodeClass gains tag-scoped capacityReservationSelectorTerms (karpenter.sh/discovery: <cluster-id>/CapacityReservation, the same discovery convention as security groups) and the NodePool offers the reserved capacity type. Both render only on gate-on clusters, and are inert until a reservation actually carries the tag, so consuming a reservation becomes self-service: tag it at purchase, no per-cluster configuration.
• admission-controller (branch karpenter-require-on-demand-admits-reserved): the require-on-demand runtime policy emits a required node affinity capacity-type In [on-demand, reserved] instead of the equality selector that excluded reserved nodes, and additionally excludes Capacity Blocks (capacity-reservation-type NotIn [capacity-block]), because those are time-bounded and reclaimed at block end, an interruption the policy exists to prevent.

Why this is better than gate-off:

• Karpenter models each reservation as its own offering: it prefers reserved capacity (already paid), tracks the available count, and falls back to plain on-demand when the reservation is exhausted. Scaling beyond what you reserved just works.
• Explicit selection works: a pod can pin one specific reservation via the karpenter.k8s.aws/capacity-reservation-id node label, which the gate-off route cannot express.
• Targeted reservations and Capacity Blocks for ML become usable (the launch names the reservation id).
• No AZ pin is strictly required for consumption: Karpenter schedules into the reservation it knows about.

Does the native route also support open ODCRs? Yes, with one condition: the reservation must carry the discovery tag. A tagged open ODCR is discovered and launched into explicitly, which works regardless of its open match criteria. What the native route does not do is consume an untagged open ODCR: with the gate on, plain launches keep preference: none. So the tag-at-purchase step is the price of the native route, and the only scenario needing gate-off is "open ODCRs consumed with zero configuration".

How the two compose

The config item makes this per cluster: deepthought keeps gate-off today (zero-config open ODCRs, already verified), and any cluster that needs explicit targeting, Capacity Blocks, or exhaustion-aware fallback keeps the default gate-on and gets the native route once the two prepared PRs merge. A later flip of deepthought back to gate-on needs only the tag on its existing reservations.

One caveat shared by both routes: reservations are matched per cluster on a shared pool. Any pod whose instance type and AZ match can consume a reservation another team paid for; the discovery tag (native route) or the type+AZ pin (gate-off route) scopes intent, not access.