Support opt-in idempotency for updateTable in the Iceberg REST catalog by huaxingao · Pull Request #5037 · apache/polaris

huaxingao · 2026-07-11T00:24:33Z

Extends the entity-property (single-transaction) idempotency introduced for createTable
(#4912) to the single-table updateTable path.

What changed

Commit-time stamping (LocalIcebergCatalog.doCommit, update branch): the
Idempotency-Key is written into the table entity's internal properties in the same
transaction as the metadata change, so the key and the update commit atomically. The prior
key window is carried forward across updates (it lives only in internal properties, which
setInternalProperties would otherwise replace), and expired keys are purged inline.
Replay (IcebergCatalogHandler.updateTable): when the target entity already carries a
live copy of the request's key, the handler returns current catalog state instead of
re-applying the update.
Concurrent-race handling: the update is wrapped so that a CommitFailedException is
retried as a replay when a same-key race winner has already committed the key atomically
with its metadata change; otherwise the exception is surfaced as before.
This is behind the existing polaris.idempotency.enabled flag (default false) and reuses
the existing key TTL (polaris.idempotency.ttl, default PT5M).

Scope / intentional exclusions

Single-table updateTable only, mirroring the focused scope of createTable.
delete is intentionally not covered. The entity-property model stamps the key onto
the entity produced by the operation; a delete removes the entity, so there is nowhere to
record the key within the same transaction.
Several polish items surfaced in review (TTL-derived window cap, a constant rename, and
decode() graceful degradation) are deliberately deferred to a separate clean-up PR to
keep this change focused.

Testing

New EntityIdempotencyUpdateTableTest covers:

a keyed retry replays the original success without re-applying a different payload,
updates without a key apply on every attempt (non-idempotent), and
a different key on an already-updated table is treated as a new update, not a replay
(this also exercises the key-window carry-forward path).
The concurrent-race CommitFailedException branch is not unit-tested, as it can't be
triggered deterministically; the logic mirrors the reviewed createTable race handling.

Checklist

🛡️ Don't disclose security issues! (contact security@apache.org)
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
📚 Updated documentation in site/content/in-dev/unreleased (if needed)

huaxingao · 2026-07-13T18:12:03Z

cc @dimas-b @flyrain Could you please take a look when you have a moment? Thanks a lot!

dimas-b · 2026-07-13T21:08:40Z

+        if (winner != null
+            && EntityIdempotency.hasLiveKey(
+                winner.getInternalPropertiesAsMap(), idempotencyKey, clock().instant())) {
+          return catalogHandlerUtils().loadTable(baseCatalog, tableIdentifier);


nit: this fragment looks identical to lines 1236-1241... Would it be worth introducing a helper method?

Extracted the duplicated fragment into a replayIfKeyLive(tableIdentifier, key) helper.

dimas-b · 2026-07-13T21:44:06Z

+    // Same key, different payload: because the key is still live on the entity, the request replays
+    // current catalog state and does NOT apply the new value, so "p" stays "first".
+    LoadTableResponse replay =
+        updateTableProperties(services, tableId, Map.of("p", "second"), IDEMPOTENCY_KEY);


nit: this test is not correct from the IRC spec POV: reusing the same idempotency key is allowed only between requests with the same payload 🤷

Would it be possible to check the entity version ID instead?

It is fine as a "white box" test for the current Polaris impl., though.

Reworked the test to retry with the same key and same payload and assert via the table version (first update advances metadataFileLocation, retry leaves it unchanged).

dimas-b · 2026-07-13T21:46:06Z

+ * catalog state instead of re-applying the update (which could fail the request's requirements
+ * against the already-advanced table).
+ */
+public class EntityIdempotencyUpdateTableTest {


Why not add test cases to EntityIdempotencyTest?

EntityIdempotencyTest only unit-tests the EntityIdempotency helper, which this PR doesn't change. The updateTable logic is in the handler and commit path, so it needs an end-to-end test, this file is just the update counterpart of the existing EntityIdempotencyCreateTableTest.

dimas-b · 2026-07-13T21:47:03Z

 - Added an OpenTelemetry event listener for emitting Polaris audit events as OpenTelemetry log records.
 - Added optional `sessionPolicy` field to `SigV4AuthenticationParameters` for catalog federation. When set, the IAM session policy JSON is attached to the STS AssumeRole request, allowing administrators to restrict vended credentials to only the required AWS services and actions (Principle of Least Privilege).
 - Added opt-in idempotency for `createTable` in the Iceberg REST catalog. When enabled via `polaris.idempotency.enabled=true` (default `false`), a client-supplied `Idempotency-Key` header is embedded into the new table entity and committed in the same transaction; a retry carrying the same key within the TTL window (`polaris.idempotency.ttl`, default `PT5M`) replays the original success instead of failing with `AlreadyExists`.
+- Extended opt-in idempotency to `updateTable` in the Iceberg REST catalog. With idempotency enabled, the `Idempotency-Key` supplied on a commit is stamped onto the table entity in the same transaction as the metadata change; a retry carrying the same key within the TTL window replays the original success from current catalog state instead of failing with `CommitFailedException` when the request's requirements no longer match the already-advanced table.


Since opt-in idempotency for createTable has not been released yet, it might be worth grouping both under the same changelog item.

Merged both into a single changelog entry

flyrain

LGTM overall. Thanks @huaxingao

flyrain · 2026-07-14T00:48:11Z

+        Map<String, String> internalProperties = storedProperties;
+        // setInternalProperties replaces the whole map, and the idempotency window lives only in
+        // internal properties (not table metadata), so carry it forward to keep prior keys live
+        // across updates even when a given update carries no key of its own.
+        String priorWindow =
+            entity.getInternalPropertiesAsMap().get(EntityIdempotency.IDEMPOTENCY_KEYS_PROPERTY);
+        if (priorWindow != null) {
+          internalProperties.put(EntityIdempotency.IDEMPOTENCY_KEYS_PROPERTY, priorWindow);
+        }
+        if (idempotencyRequestContext.isActive()) {
+          // Stamp this update's key atomically with the metadata change, purging expired keys.
+          internalProperties =
+              EntityIdempotency.recordKey(
+                  internalProperties,
+                  idempotencyRequestContext.pendingKey(),
+                  idempotencyRequestContext.pendingExpiry(),
+                  Instant.now());
+        }


Nit: we add more code into this already long method. Not sure if we can refactor a bit to avoid a ~200 LOC method.

Pulled the window/key-stamping logic out of doCommit into a small idempotencyInternalProperties helper.

Copilot

Pull request overview

This pull request extends the existing entity-internal-properties idempotency mechanism (previously used for Iceberg REST createTable) to also support opt-in idempotency for single-table updateTable, so a retry with the same Idempotency-Key can replay success from current catalog state instead of re-applying an update.

Changes:

Persist the active Idempotency-Key into the table entity’s internal properties atomically with updateTable commits, while carrying forward and purging the existing key window across updates.
Add updateTable replay handling in the Iceberg REST handler: if the key is already live on the entity, return current table state (and also retry-as-replay on a same-key CommitFailedException race).
Add end-to-end tests for the updateTable idempotency behavior and update the changelog entry to include updateTable.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
runtime/service/src/test/java/org/apache/polaris/service/catalog/iceberg/EntityIdempotencyUpdateTableTest.java	Adds end-to-end coverage for keyed replay, unkeyed non-idempotent behavior, and different-key behavior for `updateTable`.
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/LocalIcebergCatalog.java	Ensures idempotency key windows are preserved across updates and records the current key atomically with metadata commits.
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/IcebergCatalogHandler.java	Implements `updateTable` replay when a live key is present and handles same-key concurrent races via a replay check on `CommitFailedException`.
CHANGELOG.md	Updates the unreleased feature note to reflect opt-in idempotency support for both `createTable` and `updateTable`.

Support opt-in idempotency for updateTable in the Iceberg REST catalog

8c0bc67

github-project-automation Bot added this to Basic Kanban Board Jul 11, 2026

github-project-automation Bot moved this to PRs In Progress in Basic Kanban Board Jul 11, 2026

dimas-b previously approved these changes Jul 13, 2026

View reviewed changes

github-project-automation Bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Jul 13, 2026

flyrain previously approved these changes Jul 14, 2026

View reviewed changes

address comments

d4f2001

Copilot AI review requested due to automatic review settings July 14, 2026 02:18

huaxingao dismissed stale reviews from flyrain and dimas-b via d4f2001 July 14, 2026 02:18

Copilot started reviewing on behalf of huaxingao July 14, 2026 02:18 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

flyrain approved these changes Jul 14, 2026

View reviewed changes

Uh oh!

Conversation

huaxingao commented Jul 11, 2026

What changed

Scope / intentional exclusions

Testing

Checklist

Uh oh!

huaxingao commented Jul 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants