feat(durable-messaging): add durable inbox and outbox - #10693
feat(durable-messaging): add durable inbox and outbox#10693ReubenBond wants to merge 42 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new durable, grain-scoped inbox/outbox messaging primitive (Microsoft.Orleans.DurableMessaging) built on Orleans Journaling + Durable Jobs, alongside supporting journaling participant/observer capabilities and Durable Jobs execution semantics needed for reliability, routing, and recovery.
Changes:
- Introduces
Microsoft.Orleans.DurableMessaging(envelopes, routing handlers, inbox/outbox contracts, DI/hosting integration, diagnostics, and instrumentation). - Extends Journaling with composable grain participants and write/recovery observers, plus rollback fencing and codec fixes for snapshot reference-scope replay.
- Extends Durable Jobs with feature handler registry, explicit
RetryAtdisposition/reset-rescheduling support, and improved execution deduplication.
Show a summary per file
| File | Description |
|---|---|
| test/Orleans.Journaling.Tests/OrleansBinaryCommandCodecTests.cs | Adds regression tests ensuring pre-change snapshot payloads replay with independent reference scopes. |
| test/Orleans.Journaling.Tests/JournaledGrainParticipantTests.cs | Adds integration coverage for composed journaling participants initialization and failure propagation. |
| test/Orleans.Journaling.Tests/DurableListDirectWriteTests.cs | Updates test double to implement new revert/rollback surface. |
| test/Orleans.Journaling.Tests/DurableCollectionDirectWriteTests.cs | Updates test double to implement new revert/rollback surface. |
| test/Orleans.DurableMessaging.Tests/Support/SnapshotProbe.cs | Adds snapshot-based test synchronization utility. |
| test/Orleans.DurableMessaging.Tests/Support/HandlerProbe.cs | Adds barrier utility to block/release handlers deterministically in tests. |
| test/Orleans.DurableMessaging.Tests/Support/DurableMessagingTestGrains.cs | Adds durable-messaging test grain, message/effect models, and handler behavior for scenarios. |
| test/Orleans.DurableMessaging.Tests/Support/DurableMessagingClusterFixture.cs | Adds reusable in-process cluster fixture with Durable Jobs + Journaling + Durable Messaging wiring. |
| test/Orleans.DurableMessaging.Tests/Support/ControlledJournalStorageProvider.cs | Adds controllable journal storage provider for injecting write blocking/failures in tests. |
| test/Orleans.DurableMessaging.Tests/Orleans.DurableMessaging.Tests.csproj | Introduces new Durable Messaging test project. |
| test/Orleans.DurableMessaging.Tests/Hosting/PublicDurableMessagingRegistrationTests.cs | Validates DI registrations, options behavior, and absence of friend-access requirements. |
| test/Orleans.DurableMessaging.Tests/Functional/MultiSiloDurableMessagingFailoverTests.cs | Adds multi-silo failover recovery test for stable inbox job ownership and exactly-once effects. |
| test/Orleans.DurableMessaging.Tests/Functional/InboxCapacityBehaviorTests.cs | Adds backpressure + recovery behavior tests under capacity constraints. |
| test/Orleans.DurableMessaging.Tests/Functional/DedupeExpiryBehaviorTests.cs | Adds dedupe expiry/compaction behavior test allowing later reprocessing. |
| test/Orleans.DurableMessaging.Tests/Contracts/HandlerRoutingContractTests.cs | Adds routing contract tests for exact, prefix, correlation, and typed handler behaviors. |
| test/Orleans.DurableMessaging.Tests/Contracts/DurableEnvelopeContractTests.cs | Adds envelope/builder/serializer contract tests including reply-to and context. |
| test/Orleans.DurableMessaging.Tests/Contracts/DeliveryAndOptionsContractTests.cs | Adds contracts for delivery result/status and options validation defaults/boundaries. |
| test/Orleans.DurableJobs.Tests/DurableJobs/JobShardTests.cs | Adds shard-level test distinguishing reschedule reset vs failure retry attempt semantics. |
| test/Orleans.DurableJobs.Tests/DurableJobs/JobShardManagerTestsRunner.cs | Adds reassignment test ensuring reschedule reset semantics persist through shard reassignment. |
| test/Orleans.DurableJobs.Tests/DurableJobs/DurableJobFeatureHandlerTests.cs | Adds coverage for registry behavior and durable job run-result compatibility semantics. |
| test/Orleans.Core.Tests/Orleans.Core.Tests.csproj | Links HierarchicalKey tests into core test project. |
| test/Orleans.Core.Tests/DurableJobs/ShardExecutorTests.cs | Adds executor tests for RetryAt, unknown disposition handling, and legacy shard behavior. |
| test/Orleans.Core.Tests/DurableJobs/DurableJobsExtensionsTests.cs | Adds DI scope/registry behavior tests and explicit replacement/decoration rejection. |
| test/Orleans.Core.Tests/DurableJobs/DurableJobReceiverExtensionTests.cs | Updates execution identity to (JobId, RunId), adds retention/dedup + feature handler precedence tests. |
| test/Benchmarks/Journaling/DurableListJournalBenchmarks.cs | Updates benchmark test double to implement new revert/rollback surface. |
| src/Orleans.Journaling/JournaledStateManager.cs | Adds rollback support flag, observer notifications, recovery fencing, and revert-pending-changes support. |
| src/Orleans.Journaling/IJournaledStateObserver.cs | Introduces observer interface for write/recovery boundaries. |
| src/Orleans.Journaling/IJournaledStateManager.cs | Adds SupportsRollback, observer registration, and RevertPendingChangesAsync surface. |
| src/Orleans.Journaling/IJournaledGrainParticipant.cs | Introduces participant initialization hook for composed journaling features. |
| src/Orleans.Journaling/Formats/OrleansBinary/OrleansBinaryDurableSetCommandCodec.cs | Fixes snapshot replay to reset reference scope per element. |
| src/Orleans.Journaling/Formats/OrleansBinary/OrleansBinaryDurableQueueCommandCodec.cs | Fixes snapshot replay to reset reference scope per element. |
| src/Orleans.Journaling/Formats/OrleansBinary/OrleansBinaryDurableListCommandCodec.cs | Fixes snapshot replay to reset reference scope per element. |
| src/Orleans.Journaling/Formats/OrleansBinary/OrleansBinaryDurableDictionaryCommandCodec.cs | Fixes snapshot replay to reset reference scope per key/value entry. |
| src/Orleans.Journaling/Formats/OrleansBinary/OrleansBinaryCommandCodecHelpers.cs | Adds helper to read values using independent serializer sessions (reference-scope reset). |
| src/Orleans.Journaling/DurableGrain.cs | Initializes composed participants before recovery and documents WriteStateAsync. |
| src/Orleans.DurableMessaging/RoutePrefixHandler.cs | Adds prefix-based routing handler base class and documentation. |
| src/Orleans.DurableMessaging/RouteKeyHandler.cs | Adds exact-route routing handler base class and documentation. |
| src/Orleans.DurableMessaging/README.md | Adds package readme with configuration and semantics overview. |
| src/Orleans.DurableMessaging/Orleans.DurableMessaging.csproj | Adds new packable Durable Messaging project. |
| src/Orleans.DurableMessaging/InboxHandlerContext.cs | Adds handler context implementation for sending outbox messages and building envelopes. |
| src/Orleans.DurableMessaging/IInboxHandlerContext.cs | Adds public handler context contract. |
| src/Orleans.DurableMessaging/IInboxHandler.cs | Adds handler interfaces including typed handler adapter and documentation. |
| src/Orleans.DurableMessaging/IDurableOutbox.cs | Adds outbox public contract. |
| src/Orleans.DurableMessaging/IDurableMessagingDiagnostics.cs | Adds diagnostics contract + internal implementation for dead letters. |
| src/Orleans.DurableMessaging/IDurableInboxExtension.cs | Adds grain extension contract for durable inbox delivery. |
| src/Orleans.DurableMessaging/IDurableInbox.cs | Adds inbox public contract including handler registration and lookup. |
| src/Orleans.DurableMessaging/Hosting/DurableMessagingExtensions.cs | Adds ISiloBuilder/IServiceCollection registration, option validation, and rollback requirement checks. |
| src/Orleans.DurableMessaging/DurableMessagingPumpResults.cs | Adds pump execution result tracking and one-shot timer handle helper. |
| src/Orleans.DurableMessaging/DurableMessagingInstruments.cs | Adds meters/counters/histograms for inbox/outbox behavior and depth tracking. |
| src/Orleans.DurableMessaging/DurableMessagingGrainParticipant.cs | Ensures messaging services materialize via journaling participant initialization. |
| src/Orleans.DurableMessaging/DurableMessageState.cs | Adds durable state models for inbox/outbox attempts and dead letters. |
| src/Orleans.DurableMessaging/DurableInbox.cs | Adds inbox implementation with handler registration/selection and storage-backed message access. |
| src/Orleans.DurableMessaging/DurableEnvelopeData.cs | Adds deferred (slice-based) envelope body/context storage and accessors. |
| src/Orleans.DurableMessaging/DurableEnvelope.cs | Adds durable envelope type (routing, correlation, reply-to, metadata, and payload). |
| src/Orleans.DurableMessaging/DeliveryStatus.cs | Adds delivery status enum contract. |
| src/Orleans.DurableMessaging/DeliveryResult.cs | Adds delivery result struct contract and factories. |
| src/Orleans.DurableMessaging/CorrelationHandler.cs | Adds correlation-hierarchy-based routing handler base class. |
| src/Orleans.DurableMessaging/Configuration/DurableInboxOptions.cs | Adds durable messaging options and validation contract. |
| src/Orleans.DurableJobs/ShardExecutor.cs | Adds RetryAt handling with reset-rescheduling path and explicit legacy-shard failure. |
| src/Orleans.DurableJobs/JournaledJobShard.cs | Implements reset-rescheduling in journaled shard via IResettableJobShard. |
| src/Orleans.DurableJobs/JobShard.cs | Adds reset-rescheduling support to base shard and introduces IResettableJobShard. |
| src/Orleans.DurableJobs/IDurableJobReceiverExtension.cs | Updates receiver extension for feature-handler lookup, new execution identity, and retention semantics. |
| src/Orleans.DurableJobs/IDurableJobHandlerRegistry.cs | Adds activation-scoped feature handler registry + lookup implementation. |
| src/Orleans.DurableJobs/Hosting/DurableJobsOptions.cs | Adds completed-attempt retention option and validation. |
| src/Orleans.DurableJobs/Hosting/DurableJobsExtensions.cs | Registers registry in DI and explicitly rejects replacement/decoration. |
| src/Orleans.DurableJobs/DurableJobRunResult.cs | Adds RetryAt disposition and related fields/compatibility semantics. |
| src/api/Orleans.Journaling/Orleans.Journaling.cs | Updates generated API surface for journaling observer/participant/rollback additions. |
| src/api/Orleans.DurableJobs/Orleans.DurableJobs.cs | Updates generated API surface for feature handlers/registry and RetryAt semantics. |
| src/api/Orleans.Core.Abstractions/Orleans.Core.Abstractions.cs | Updates generated API surface to include HierarchicalKey type and codec. |
| Orleans.slnx | Adds Durable Messaging projects (src + tests) to the solution. |
| docs/site/src/data/unpublished-api-packages.json | Marks Durable Messaging package as unpublished. |
| docs/site/src/data/external-link-allowlist.json | Allow-lists NuGet link for unpublished Durable Messaging package. |
| docs/site/src/content/docs/toc.yml | Adds Durable Messaging doc page to Journaling section. |
| docs/site/src/content/docs/resources/nuget-packages.md | Adds Durable Messaging to NuGet package catalog and updates guidance blurb. |
| docs/site/src/content/docs/grains/durable-messaging.md | Adds end-user documentation for durable messaging semantics and requirements. |
Review details
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 83/83 changed files
- Comments generated: 4
- Review effort level: Lite
There was a problem hiding this comment.
Review details
Suppressed comments (4)
Previously missed (1) — in code that hasn't changed since the last review.
src/Orleans.DurableMessaging/DurableInbox.cs:43
- Exact-route handler storage uses the default string comparer, which is culture-sensitive. Route keys are protocol identifiers and should use ordinal comparisons (matching the rest of this PR’s routing semantics). Use
StringComparer.Ordinalfor_exactRouteHandlersto avoid surprising behavior under non-invariant cultures (e.g., Turkish-I issues).
_inbox = inbox;
_processed = processed;
_handlers = new List<IInboxHandler>();
_exactRouteHandlers = new Dictionary<string, IInboxHandler>();
_capacity = capacity;
src/Orleans.Journaling/IJournaledStateObserver.cs:16
- The remarks contradict the interface contract: they say the manager does not notify observers before a write, but this interface defines OnWriteStarted and JournaledStateManager invokes it before capturing state. This is confusing for implementers; update the remarks to reflect the actual callback sequence.
src/Orleans.DurableMessaging/RouteKeyHandler.cs:21 - The remarks suggest implementing
IInboxHandlerdirectly for prefix-based routing, but this PR also introducesRoutePrefixHandlerfor that exact scenario. Updating the docs helps steer users to the supported helper type and keeps the guidance consistent.
src/Orleans.DurableMessaging/InboxHandlerContext.cs:125 - The XML docs reference
IStateMachineManager.WriteStateAsync(), which doesn't exist in this package and appears to be a leftover name. This should point toIJournaledStateManager.WriteStateAsync()to avoid misleading API consumers.
- Files reviewed: 83/83 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
Review details
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
src/Orleans.DurableJobs/IDurableJobReceiverExtension.cs:185
- LongPollGetJobStatusAsync creates a CancellationTokenSource but never cancels it. That means the Task.Delay will always run to completion even when the job finishes quickly, causing avoidable timer allocations/work per call.
using var cts = new CancellationTokenSource();
var longPollDuration = TimeSpan.FromTicks(Math.Min(_shared.MessagingOptions.ResponseTimeout.Divide(2).Ticks, _shared.Options.JobStatusPollInterval.Ticks));
await Task.WhenAny(Task.Delay(longPollDuration, cts.Token), state.Task);
if (!state.Task.IsCompleted)
{
return DurableJobRunResult.PollAfter(_shared.Options.JobStatusPollInterval);
}
src/Orleans.DurableMessaging/Hosting/DurableMessagingExtensions.cs:58
- The DurableInboxOptions validation predicate swallows all exceptions and turns them into a generic validation failure. That can hide unexpected exceptions (eg, NullReferenceException) and makes misconfiguration harder to diagnose. Consider only converting known validation exceptions into a failed predicate and letting other exceptions bubble.
optionsBuilder.Validate(
options =>
{
try
{
options.Validate();
return true;
}
catch
{
return false;
}
},
"DurableInboxOptions validation failed.");
- Files reviewed: 83/83 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
test/Orleans.DurableMessaging.Tests/Support/SnapshotProbe.cs:33
- WaitAsync adds a waiter to the per-grain list and then returns
waiter.Completion.Task.WaitAsync(...), but if that wait times out/throws, the waiter remains in_waitersindefinitely. This can leak memory and can also keep evaluating stale predicates on subsequent Publish calls. Consider awaiting with a cleanupfinallythat removes the waiter (and optionally removes the empty list from_waiters).
- Files reviewed: 85/85 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
Review details
Suppressed comments (4)
Previously missed (3) — in code that hasn't changed since the last review.
src/Orleans.Journaling/JournaledStateManager.cs:309
- WorkLoop captures a stable observer snapshot into the local
observersarray, butOnWriteStarted()is invoked by iterating_observersdirectly. If an observer is registered duringOnWritePreparingAsync(or from within another observer callback), it can change_observersmid-iteration, causing inconsistent callback sequencing and potentialInvalidOperationExceptiondue to collection mutation during enumeration. Use the capturedobserverssnapshot forOnWriteStartedto keep the write boundary consistent and mutation-safe.
src/Orleans.Journaling/JournaledStateManager.cs:493 OnWriteCompleted()is invoked by iterating_observersdirectly, even though a stableobserverssnapshot was captured earlier for this write. If observers are registered during the write pipeline, enumerating the live HashSet can throw (collection modified) or cause some observers to seeOnWriteCompletedwithout the correspondingOnWritePreparingAsync/OnWriteStarted. Use the per-write snapshot for completion callbacks.
This issue also appears on line 516 of the same file.
src/Orleans.Journaling/JournaledStateManager.cs:826
- Recovery completion notifies observers by enumerating
_observersdirectly inside a lock. If any observer callback registers another observer (re-entrantly) this can mutate the HashSet during enumeration and throw, potentially breaking recovery. Take a snapshot of_observersbefore iterating so callbacks are mutation-safe and sequencing is stable.
src/Orleans.Journaling/JournaledStateManager.cs:516
- In the no-op write path (
!hasCommittedBuffer),OnWriteCompleted()is called by iterating_observersdirectly. This has the same collection-mutation risk as the committed write path and can also notify observers which were registered afterOnWritePreparingAsyncran for this write. Prefer using the stableobserverssnapshot captured at the start of the work item.
- Files reviewed: 85/85 changed files
- Comments generated: 0 new
- Review effort level: Lite
|
Addressed the latest observer-snapshot review findings in \3d4829b8d. Each write now uses one stable observer snapshot for preparation, start, and completion (including no-op writes), and recovery snapshots observers before callbacks so re-entrant registration cannot mutate an active enumeration. Focused regression coverage passes on net8.0 and net10.0. |
There was a problem hiding this comment.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
test/Orleans.DurableMessaging.Tests/Support/SnapshotProbe.cs:33
- WaitAsync adds a waiter to the per-grain list but never removes it if the 30s timeout elapses (WaitAsync throws). That can leak waiters across a test run and slow down Publish() due to ever-growing lists.
- Files reviewed: 85/85 changed files
- Comments generated: 1
- Review effort level: Lite
|
Addressed the remaining suppressed review findings in two focused commits:\n\n- \�222b4b77\ removes timed-out SnapshotProbe waiters so stale predicates are not retained or evaluated; regression coverage passes on net8.0 and net10.0.\n- \�d214b07b\ narrows durable inbox options validation to the documented \ArgumentOutOfRangeException, allowing unexpected faults to surface; the options-contract test passes on net8.0 and net10.0. |
There was a problem hiding this comment.
Review details
Suppressed comments (1)
src/Orleans.Journaling/IJournaledStateManager.cs:58
IJournaledStateManageris a public interface andRevertPendingChangesAsyncis added as a required member (no default implementation). This is a breaking change for any external implementations ofIJournaledStateManager. Consider providing a default interface implementation (similar toRegisterObserver) which throwsNotSupportedException, and useSupportsRollbackto indicate capability.
- Files reviewed: 86/86 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
src/Orleans.DurableMessaging/DurableOutbox.cs:168
- The XML docs reference
IStateMachineManager.WriteStateAsync(), but durable messaging commits are driven by journaling. This should referenceIJournaledStateManager.WriteStateAsync()to match the rest of the package docs and avoid misleading API consumers.
/// The message is persisted atomically with grain state when IStateMachineManager.WriteStateAsync()
/// is called. The background pump will deliver the message to the target grain ONLY AFTER
/// the message has been durably persisted.
- Files reviewed: 86/86 changed files
- Comments generated: 0 new
- Review effort level: Lite
Move the draft hierarchical correlation API, serializer, generated API surface, and tests into Microsoft.Orleans.DurableMessaging. Preserve the original Orleans.HierarchicalKey wire alias and field identifiers for persisted envelope compatibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 209c9811-96cc-46b3-8705-e2ad263b1b98
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 209c9811-96cc-46b3-8705-e2ad263b1b98
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
It introduces new runtime subsystems plus journaling and durable-jobs behavior changes whose correctness and compatibility impacts require careful human validation beyond this automated review.
Review tier: Lite
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
src/Orleans.DurableMessaging/DurableOutbox.cs — The XML doc for OnWriteStarted says it runs after writes are durably persisted, but… |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The generated public API baseline for Orleans.Journaling omits default interface implementations (e.g., observer ValueTask methods and RegisterObserver), which can break downstream compilation against reference assemblies and must be corrected.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 2
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
src/api/Orleans.Journaling/Orleans.Journaling.cs — In the API-surface file, IJournaledStateManager.RegisterObserver is declared without the default… |
|
src/api/Orleans.Journaling/Orleans.Journaling.cs — In the API-surface file, IJournaledStateObserver.OnDeletePreparingAsync is missing its default… |
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
src/Orleans.DurableMessaging/DurableOutbox.cs — The XML doc for OnWriteStarted says it runs after writes are durably persisted, but… View resolved comment |
Suppressed comments (2)
src/api/Orleans.Journaling/Orleans.Journaling.cs:255
- In the API-surface file, IJournaledStateObserver.OnWriteFinalizingAsync is missing its default interface implementation (the real interface returns a default ValueTask). Without the body here, downstream implementations compiled against the reference assembly will be required to implement it unexpectedly.
src/api/Orleans.Journaling/Orleans.Journaling.cs:256 - In the API-surface file, IJournaledStateObserver.OnWritePreparingAsync is missing its default interface implementation (the real interface returns a default ValueTask). This makes the method appear abstract in the reference API and can break external implementers at compile time.
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Observer registration currently uses value-based equality in a HashSet, which can incorrectly reject distinct observer instances that override equality semantics.
Review tier: Lite
Findings: None
Issues resolved since last review (2)
| Severity | Finding |
|---|---|
src/api/Orleans.Journaling/Orleans.Journaling.cs — In the API-surface file, IJournaledStateObserver.OnDeletePreparingAsync is missing its default… View resolved comment |
|
src/api/Orleans.Journaling/Orleans.Journaling.cs — In the API-surface file, IJournaledStateManager.RegisterObserver is declared without the default… View resolved comment |
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
src/Orleans.Journaling/JournaledStateManager.cs:22
- _observers is a HashSet using the default equality comparer, so observer instances which override Equals/GetHashCode (eg, records) can be treated as duplicates and rejected even when they are different instances. Observer registration should typically be identity-based to avoid surprising collisions.
src/Orleans.DurableMessaging/DurableInboxExtension.cs:285 - DeliverAsync checks handler existence by calling TryFindHandler here, and the inbox pump calls TryFindHandler again when actually processing the message. This doubles CanHandle evaluation per delivery and can become noticeable under high throughput (especially if handlers do non-trivial metadata checks).


Proposal
Problem
Orleans applications need a durable, grain-scoped messaging primitive for state transitions which must atomically emit follow-up work and survive activation or silo loss. Ordinary grain calls do not provide an inbox/outbox commit boundary, retained deduplication, backpressure, or dead-letter state.
API
This introduces
Microsoft.Orleans.DurableMessagingwith durable envelopes, inbox/outbox contracts, typed handlers, exact-route, route-prefix, and hierarchical-correlation routing, diagnostics, hosting integration, and a grain participant. Envelopes carry sender, target, route,Orleans.DurableMessaging.HierarchicalKeycorrelation, generalReplyTometadata, serialized body, and request context. The initial surface focuses on direct durable messaging; Durable Jobs provides scheduling, and application protocols define response and error envelopes.Guarantees and semantics
Acceptedis returned only after the inbox envelope and stable drain-job ownership are durable.(SenderId, MessageId), giving effectively-once effects while the configured deduplication record is retained. Expired records permit reprocessing.Recovery
Inbox and outbox use stable, independent Durable Jobs with monotonic ownership generations. A job polls its current attempt while activation recovery or a matching commit is incomplete. After recovery, a generation with no committed owner and no pending work is a confirmed orphan and completes so Durable Jobs removes it. If work exists without matching ownership, recovery schedules and commits a new generation before the old generation terminates; stale generations complete without disturbing newer ownership. This bounds schedule-before-commit crash remnants while preserving at-least-once delivery. Non-interleaving grain timer turns prevent infrastructure writes from committing provisional handler state. Activation remains fenced until the recovery-completion observer boundary, and a dedicated orphan-reclamation counter supports diagnosis. Tests cover inbox/outbox precommit crashes, provisional visibility, stale generations, recovery interleaving, commit races, reactivation, and multi-silo owner loss.
Layering
The package composes public Journaling and Durable Jobs contracts. It uses
IJournaledStateManager.RevertPendingChangesAsyncfor rollback andIJournaledStateManager.RegisterObserverfor durable boundary notifications. Unsupported observer registration produces a durable-messaging-specific activation diagnostic. The narrowIJournaledStateObservercontract identifies pre-write, commit, and recovery-completion boundaries, preventing later outbox enqueues from becoming dispatchable because an earlier write completed. Observer failures are isolated from completed durable boundaries.The correlation hierarchy is owned by
Microsoft.Orleans.DurableMessagingasOrleans.DurableMessaging.HierarchicalKey. Draft consumers should update the namespace fromOrleans.HierarchicalKey; the original serialized alias and field identifiers are retained for persisted envelope compatibility.Depends on #10679 at
efa5809544cd6794df340fb94e7ff316b109b04fand #10681 atc7062487ac693dcdc8af75ae8178617f57fca6ad. This branch merges those exact rebased heads on main8d7594ab4f1229620fa1de9054194a452e7318ae.Microsoft Reviewers: Open in CodeFlow