fix(transactions): remove aborted queued lock operations - #10455
Open
ReubenBond wants to merge 5 commits into
Open
fix(transactions): remove aborted queued lock operations#10455ReubenBond wants to merge 5 commits into
ReubenBond wants to merge 5 commits into
Conversation
6 tasks
ReubenBond
force-pushed
the
reubenbond-fix-transaction-recovery-flake
branch
from
August 21, 2026 23:19
11d1e36 to
f0018c8
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a transactional lock-queue correctness issue in Orleans.Transactions where an aborted transaction could remain in a queued lock group and later have its stale pending operations executed when that group was promoted. The change ties queued/pending lock operations to a transaction id so they can be aborted/removed when the transaction is rolled back, preventing cascading aborts and recovery stalls described in #5211.
Changes:
- Replace queued “task callbacks” with transaction-associated pending operations (execute vs abort) and abort/remove them when a queued transaction is rolled back or expires.
- Update rollback logic to search/remove transactions across queued lock groups (not just the active group), aborting any queued pending operations for those transactions.
- Add a regression test ensuring multiple queued pending operations for an aborted transaction are completed as cascading aborts and do not execute later.
Show a summary per file
| File | Description |
|---|---|
src/Orleans.Transactions/State/ReaderWriterLock.cs |
Associates queued lock callbacks with transaction ids and ensures aborted/expired queued transactions have their pending operations aborted rather than executed later. |
test/Transactions/Orleans.Transactions.Tests/TransactionRecoveryLatencyTests.cs |
Adds a regression test covering multiple pending operations for one aborted queued transaction and verifies subsequent queued transactions proceed normally. |
Review details
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 2/2 changed files
- Comments generated: 0
- Review effort level: Lite
ReubenBond
force-pushed
the
reubenbond-fix-transaction-recovery-flake
branch
from
August 30, 2026 08:14
f0018c8 to
125306c
Compare
ReubenBond
enabled auto-merge
August 30, 2026 14:10
ReubenBond
force-pushed
the
reubenbond-fix-transaction-recovery-flake
branch
from
August 30, 2026 14:16
125306c to
63d3f3e
Compare
ReubenBond
disabled auto-merge
August 31, 2026 15:01
ReubenBond
requested review from
Copilot
and removed request for
a team and
sebastianburckhardt
September 1, 2026 08:04
ReubenBond
force-pushed
the
reubenbond-fix-transaction-recovery-flake
branch
from
September 1, 2026 08:30
63d3f3e to
e0bd1be
Compare
Track pending lock operations by transaction so an abort removes and completes all queued work before the lock group is promoted. This prevents stale operations from surviving recovery and poisoning later transactions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
ReubenBond
force-pushed
the
reubenbond-fix-transaction-recovery-flake
branch
from
September 2, 2026 01:10
e0bd1be to
7f3a961
Compare
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Queued operation execution currently runs task() unconditionally, so a defensive membership check is needed to prevent any stale pending operation from executing if group membership and pending-operation lists ever diverge.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
 |
src/Orleans.Transactions/State/ReaderWriterLock.cs — PendingOperation.Execute ultimately runs completion(), which calls task() unconditionally. If… |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A transaction aborted while waiting for a transactional-state lock remained in its queued lock group because rollback only searched the active group. When that queued group was later promoted, the stale transaction still appeared valid and its pending operations executed, producing cascading aborts and prepare timeouts that could prevent recovery from settling.
A rejected queued write upgrade had the same lifetime gap: the upgrade threw before the participant callback executed, so the transaction agent did not yet know about that participant and could not reliably release its earlier queued operations.
This change associates pending lock operations with their transaction and removes them whenever that transaction is rolled back, expires, loses a conflict, fails a lock upgrade, or its queued group is reset. Each removed operation completes with the existing cascading-abort semantics before the group can be promoted, while unaffected transactions continue through the queue.
Fixes #5211.