feat(event-sourcing): add journaled state log consistency provider#10108
Draft
ReubenBond wants to merge 6 commits into
Draft
feat(event-sourcing): add journaled state log consistency provider#10108ReubenBond wants to merge 6 commits into
ReubenBond wants to merge 6 commits into
Conversation
fe36814 to
3bcc650
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Introduces a new log-consistency provider, Orleans.EventSourcing.JournaledState, that stores the event log and write-vector for log-consistent grains inside the grain's own Orleans.Journaling state machine. To support the explicit-reread recovery flow used by the provider, IJournaledStateManager gains a new ReadStateAsync method backed by a ReadStateWorkItem in the journaling work loop, and a related migration-snapshot fix avoids double-appending during format migration. New silo builder extensions, public-API surface updates, and end-to-end tests (including a grain variant with auxiliary durable state) round out the change.
Changes:
- Add
IJournaledStateManager.ReadStateAsyncand the matching work-loop handler; suppress an append during the migration-snapshot path; honor cancellation onInitialize/Delete/ReadviaWaitAsync. - Add the
JournaledStatelog-consistency provider (LogConsistencyProvider,LogViewAdaptor<TLogView, TLogEntry>) andAddJournaledStateBasedLogConsistencyProvider[AsDefault]silo-builder extensions, registering durable list/value for the event log and write-vector. - Add tests for the new provider (persistence across activation,
Clearpreserving auxiliary journaled state), a journaling read-state test, a JSON-to-binary migration "no-append" test, plus interface plumbing in test grains and fixtures; update generated API surfaces; swapstoredEtag/currentEtagarguments inAzureBlobJournalStorage.CreateInconsistentWalStateException.
Show a summary per file
| File | Description |
|---|---|
| src/Orleans.EventSourcing/JournaledState/LogConsistencyProvider.cs | New ILogViewAdaptorFactory that constructs the journaled-state log view adaptor. |
| src/Orleans.EventSourcing/JournaledState/LogViewAdaptor.cs | Core adaptor that persists event log and write-vector via IJournaledStateManager, with read/write/recovery loops. |
| src/Orleans.EventSourcing/Hosting/JournaledStateSiloBuilderExtensions.cs | Silo builder extensions to register the provider; sets JournalFormatKey to orleans-binary. |
| src/Orleans.EventSourcing/Orleans.EventSourcing.csproj | Adds project reference to Orleans.Journaling. |
| src/Orleans.Journaling/IJournaledStateManager.cs | Adds ReadStateAsync interface method. |
| src/Orleans.Journaling/JournaledStateManager.cs | Implements ReadStateAsync/ReadStateWorkItem, applies WaitAsync for cancellation, and avoids appending during migration snapshots. |
| src/Azure/Orleans.Journaling.AzureStorage/AzureBlobJournalStorage.cs | Swaps the storedEtag/currentEtag arguments when creating InconsistentStateException. |
| src/api/Orleans.EventSourcing/Orleans.EventSourcing.cs | Public-API surface updates for new provider and extensions. |
| src/api/Orleans.Journaling/Orleans.Journaling.cs | Public-API surface update for ReadStateAsync. |
| test/Orleans.EventSourcing.Tests/EventSourcingTests/EventSourcingClusterFixture.cs | Registers the new provider and a volatile journal storage in the test cluster. |
| test/Orleans.EventSourcing.Tests/EventSourcingTests/LogTestGrainClearTests.cs | Adds tests for Clear, persistence, and preservation of auxiliary durable state. |
| test/Grains/TestGrainInterfaces/ILogTestGrain.cs | Adds ILogTestGrainWithAuxiliaryState interface. |
| test/Grains/TestGrains/LogTestGrainVariations.cs | Adds journaled-state grain variants, including one with auxiliary durable state. |
| test/Orleans.Journaling.Tests/JournalBatchTests.cs | Adds test verifying ReadStateAsync discards unflushed changes. |
| test/Orleans.Journaling.Tests/DurableListDirectWriteTests.cs | Implements new ReadStateAsync on test fake. |
| test/Orleans.Journaling.Tests/DurableCollectionDirectWriteTests.cs | Implements new ReadStateAsync on test fake. |
| test/Orleans.Journaling.Json.Tests/CodecRecoveryTests.cs | Adds test that JSON→binary migration replaces without appending, plus a CountingStorage helper. |
Copilot's findings
- Files reviewed: 17/17 changed files
- Comments generated: 2
| ArgumentException.ThrowIfNullOrWhiteSpace(name); | ||
|
|
||
| services.AddLogConsistencyProtocolServicesFactory(); | ||
| services.Configure<JournaledStateManagerOptions>(options => options.JournalFormatKey = OrleansBinaryJournalFormatKey); |
Comment on lines
+148
to
+185
| while (true) | ||
| { | ||
| try | ||
| { | ||
| await stateManager.WriteStateAsync(CancellationToken.None); | ||
|
|
||
| Services.Log(LogLevel.Debug, "write ({0} updates) success v{1}", updates.Length, eventLog.Count); | ||
|
|
||
| UpdateConfirmedViewFromJournal(); | ||
| LastPrimaryIssue.Resolve(Host, Services); | ||
|
|
||
| exit_operation("WriteAsync"); | ||
| return updates.Length; | ||
| } | ||
| catch (Exception exception) | ||
| { | ||
| LastPrimaryIssue.Record(new UpdateJournaledStateFailed { Exception = exception }, Host, Services); | ||
| } | ||
|
|
||
| Services.Log(LogLevel.Debug, "write failed {0}", LastPrimaryIssue); | ||
|
|
||
| await LastPrimaryIssue.DelayBeforeRetry(); | ||
| await RecoverAfterWriteFailureAsync(); | ||
|
|
||
| if (writeBit == GetWriteVectorBit()) | ||
| { | ||
| Services.Log(LogLevel.Debug, "last write ({0} updates) was actually a success v{1}", updates.Length, eventLog.Count); | ||
|
|
||
| UpdateConfirmedViewFromJournal(); | ||
| LastPrimaryIssue.Resolve(Host, Services); | ||
|
|
||
| exit_operation("WriteAsync"); | ||
| return updates.Length; | ||
| } | ||
|
|
||
| exit_operation("WriteAsync"); | ||
| return 0; | ||
| } |
10d751f to
4e8eb63
Compare
Add an Orleans.EventSourcing log-consistency provider backed by the host grain journaled state. The provider persists event log entries and write-vector metadata through Orleans.Journaling, performs explicit durable rereads for recovery, and clears only provider-owned state. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Correct Azure Blob WAL conflict ETag diagnostics and avoid appending new-format journal bytes before migration snapshots replace old-format storage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the stored ETag unknown for Azure WAL conflicts and report the cached If-Match ETag as the current ETag passed by the caller. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
30c427c to
0069edf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Orleans.EventSourcing currently has storage-backed log consistency providers, but no provider that stores the relevant grain's event log in its own Orleans.Journaling state machine.
Solution
Add a journaled-state-backed log consistency provider which registers durable state for the event log and write-vector metadata. The provider wires journaling registration through silo builder extensions, uses Orleans binary journaling for arbitrary event payloads, performs explicit durable rereads for recovery, and clears only provider-owned journaled state so unrelated grain state is preserved.
Rationale
Using the grain's journaled state keeps event-sourcing events colocated with the activation's journaling infrastructure while preserving the existing log consistency protocol semantics. The explicit read API avoids relying on initialization for ambiguous write recovery, and the clear behavior avoids deleting unrelated durable state owned by the same grain.
Notes
The new provider depends on the experimental Orleans.Journaling APIs, so warning suppressions are kept local to the new integration points.
Microsoft Reviewers: Open in CodeFlow