[2/2] Enforce encrypted Iceberg metadata integrity

[hkwi]

Summary

• Store a versioned canonical SHA-256 digest for protected encrypted Iceberg metadata in trusted Polaris internal entity state.
• Verify the key-ID pin and digest when metadata is loaded and before asynchronous cleanup traverses it.
• Define register new and missing-table notification as authorized trust-on-first-use admissions.
• Define register overwrite as TABLE_FULL_METADATA-authorized re-admission after key-ID validation.
• Reject notification updates of existing protected encrypted tables until an authenticated expected digest can be bound to the request.
• Map integrity failures to non-retryable HTTP 422 responses and stop asynchronous cleanup without deleting storage when verification fails.

This implements the metadata-integrity alternative in Iceberg's catalog security requirements for deployments where plaintext metadata.json is held in storage that must not be treated as authoritative. The digest construction follows Iceberg HiveCatalog: canonical TableMetadataParser JSON, UTF-8 SHA-256, and Base64 storage outside the metadata file.

Stack

This is 2/2 of a stacked change and depends on the key-ID pinning PR: #5185.

Both PRs target main because their head branches are in an external fork. The branch for this PR contains the first PR's commit followed by one metadata-integrity commit. Until the first PR lands, the incremental second-layer diff is:

hkwi/polaris@upstream/enforce-immutable-encryption-key-id...upstream/enforce-encrypted-metadata-integrity

After the first PR is merged, this head branch will need to be updated against the resulting main history so that this PR contains only the metadata-integrity layer. This follow-up Git operation is required because the fork-only parent branch cannot be selected as the base of a PR in apache/polaris.

Guarantee and compatibility boundary

• New protected encrypted tables atomically store the pin marker, exact key ID, metadata pointer, digest, and digest version.
• Protected encrypted state with a missing or unsupported digest fails closed.
• Existing unpinned tables retain legacy behavior without automatic digest backfill.
• Direct object-storage rereads by clients that ignore inline REST metadata remain outside the guarantee.

Verification

• ./gradlew format compileAll
• Runtime-service checkstyle passed.
• All 22,103 runtime-service unit tests passed (0 failures, 0 errors, 43 skipped).
• Runtime-service integration tests fail before test execution in the common Quarkus TestHTTPResourceManager initialization path; all 28 suites report the same ParameterResolutionException caused by the NPE at TestHTTPResourceManager.java:190.

Checklist

• 🛡️ Don't disclose security issues! (contact security@apache.org)
• 🔗 Clearly explained why the changes are needed, or linked related issues: related to Iceberg encryption support #2829 and complementary to Support encrypted table operations and server-side purge #5060
• 🧪 Added/updated tests with good coverage, or manually tested (and explained how)
• 💡 Added comments for complex logic
• 🧾 Updated CHANGELOG.md (if needed)
• 📚 Updated documentation in site/content/in-dev/unreleased (if needed): not needed for internal catalog invariants

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.