Migrate diagnostics persistence to Groundwork

.specify/feature.json

@@ -1,3 +1,3 @@
 {
-  "feature_directory": "specs/091-domain-owned-apis"
+  "feature_directory": "specs/091-groundwork-diagnostics-persistence"
 }

AGENTS.md

@@ -161,5 +161,5 @@ New work should move toward this rule:
 <!-- SPECKIT START -->
 For additional context about technologies, project structure, shell commands, contracts, and
 validation scenarios for the active work unit, read
-`specs/092-domain-owned-apis/plan.md`.
+`specs/091-groundwork-diagnostics-persistence/plan.md`.
 <!-- SPECKIT END -->

EXTENSION_POINTS.md

@@ -157,6 +157,7 @@ not automatic constitution violations.
 
 | Domain | Catalog |
 |---|---|
+| Diagnostics persistence lifecycle (provider-neutral drain, bridge, observer, and replacement registration) | [`src/Elsa/Diagnostics/Persistence/EXTENSION_POINTS.md`](src/Elsa/Diagnostics/Persistence/EXTENSION_POINTS.md) |
 | Structured logs (log store, live feed, sink, source provider) | [`src/Elsa/Diagnostics/StructuredLogs/EXTENSION_POINTS.md`](src/Elsa/Diagnostics/StructuredLogs/EXTENSION_POINTS.md) |
 | OpenTelemetry (telemetry store, live feed, ingestor, redactor, source registry, provider, collector config) | [`src/Elsa/Diagnostics/OpenTelemetry/EXTENSION_POINTS.md`](src/Elsa/Diagnostics/OpenTelemetry/EXTENSION_POINTS.md) |
 

Elsa.Server.slnx

@@ -68,12 +68,16 @@
   </Folder>
   <Folder Name="/src/Elsa/Diagnostics/">
     <Project Path="src/Elsa/Diagnostics/ConsoleLogStreaming/Elsa.Diagnostics.ConsoleLogStreaming.csproj" />
+    <Project Path="src/Elsa/Diagnostics/Persistence/Elsa.Diagnostics.Persistence.csproj" />
   </Folder>
   <Folder Name="/src/Elsa/Diagnostics/OpenTelemetry/">
     <Project Path="src/Elsa/Diagnostics/OpenTelemetry/Core/Elsa.Diagnostics.OpenTelemetry.Core.csproj" />
     <Project Path="src/Elsa/Diagnostics/OpenTelemetry/Elsa.Diagnostics.OpenTelemetry.csproj" />
   </Folder>
   <Folder Name="/src/Elsa/Diagnostics/OpenTelemetry/Persistence/" />
+  <Folder Name="/src/Elsa/Diagnostics/OpenTelemetry/Persistence/Groundwork/">
+    <Project Path="src/Elsa/Diagnostics/OpenTelemetry/Persistence/Groundwork/Elsa.Diagnostics.OpenTelemetry.Persistence.Groundwork.csproj" />
+  </Folder>
   <Folder Name="/src/Elsa/Diagnostics/OpenTelemetry/Persistence/EFCore/">
     <Project Path="src/Elsa/Diagnostics/OpenTelemetry/Persistence/EFCore/Elsa.Diagnostics.OpenTelemetry.Persistence.EFCore.csproj" />
     <Project Path="src/Elsa/Diagnostics/OpenTelemetry/Persistence/EFCore/Sqlite/Elsa.Diagnostics.OpenTelemetry.Persistence.EFCore.Sqlite.csproj" />
@@ -316,6 +320,9 @@
     <Project Path="tests/Elsa/Caching/Tests/Elsa.Caching.Tests.csproj" />
   </Folder>
   <Folder Name="/tests/Elsa/Diagnostics/" />
+  <Folder Name="/tests/Elsa/Diagnostics/Persistence/">
+    <Project Path="tests/Elsa/Diagnostics/Persistence/Tests/Elsa.Diagnostics.Persistence.Tests.csproj" />
+  </Folder>
   <Folder Name="/tests/Elsa/Diagnostics/ConsoleLogStreaming/">
     <Project Path="tests/Elsa/Diagnostics/ConsoleLogStreaming/Tests/Elsa.Diagnostics.ConsoleLogStreaming.Tests.csproj" />
   </Folder>
@@ -325,6 +332,9 @@
   <Folder Name="/tests/Elsa/Diagnostics/OpenTelemetry/Persistence/">
     <Project Path="tests/Elsa/Diagnostics/OpenTelemetry/Persistence/Tests/Elsa.Diagnostics.OpenTelemetry.Persistence.Tests.csproj" />
   </Folder>
+  <Folder Name="/tests/Elsa/Diagnostics/OpenTelemetry/Persistence/Groundwork/">
+    <Project Path="tests/Elsa/Diagnostics/OpenTelemetry/Persistence/Groundwork/Tests/Elsa.Diagnostics.OpenTelemetry.Persistence.Groundwork.Tests.csproj" />
+  </Folder>
   <Folder Name="/tests/Elsa/Diagnostics/StructuredLogs/">
     <Project Path="tests/Elsa/Diagnostics/StructuredLogs/Tests/Elsa.Diagnostics.StructuredLogs.Tests.csproj" />
   </Folder>

NuGet.config

@@ -17,6 +17,7 @@
       <package pattern="coverlet.*" />
       <package pattern="Cronos" />
       <package pattern="DistributedLock.*" />
+      <package pattern="DnsClient" />
       <package pattern="Docker.DotNet.*" />
       <package pattern="FastEndpoints" />
       <package pattern="FastEndpoints.*" />
@@ -44,7 +45,9 @@
       <package pattern="Parlot" />
       <package pattern="Polly" />
       <package pattern="Polly.*" />
+      <package pattern="SharpCompress" />
       <package pattern="SharpZipLib" />
+      <package pattern="Snappier" />
       <package pattern="SourceGear.*" />
       <package pattern="SQLitePCLRaw.*" />
       <package pattern="SSH.NET" />
@@ -58,6 +61,7 @@
       <package pattern="xunit" />
       <package pattern="xunit.*" />
       <package pattern="Xunit.*" />
+      <package pattern="ZstdSharp.Port" />
     </packageSource>
     <packageSource key="CShells Preview">
       <package pattern="CShells" />

docs/maps/extension-point-map.md

@@ -6,9 +6,9 @@ Records Markdown catalog facts from `EXTENSION_POINTS.md` files. It does not val
 
 ## Summary
 
-- Catalog files discovered: 58
-- Source catalogs discovered: 57
-- Source catalogs indexed from root: 57
+- Catalog files discovered: 59
+- Source catalogs discovered: 58
+- Source catalogs indexed from root: 58
 - Discovered source catalogs not linked from root index: 0
 - Root-indexed catalogs missing on disk: 0
 
@@ -40,6 +40,7 @@ Records Markdown catalog facts from `EXTENSION_POINTS.md` files. It does not val
 | [src/Elsa/Api/FastEndpoints/EXTENSION_POINTS.md](../../src/Elsa/Api/FastEndpoints/EXTENSION_POINTS.md) | Elsa.Api.FastEndpoints | Elsa.Api | Feature-inheritance point<br>Secure-by-construction endpoint base classes<br>Implementable contributor interfaces<br>Per-shell security feature | `FastEndpointsFeatureBase`<br>`IFastEndpointsConfigurator` (CShells contract)<br>`IFastEndpointFilter`<br>`ApiSecurityFeature` / `ApiSecurityOptions` | Feature inheritance point (abstract base implementing CShells `IFastEndpointsShellFeature`).<br>Contributor (fan-in; every registered configurator is applied when the shell maps its FastEndpoints).<br>Contributor (fan-in; excludes endpoints from a shell's mapped set).<br>Shell feature with bindable options (`AllowAnonymous`, default `false`; `ShellName` is populated from the shell settings). | 0 | yes |
 | [src/Elsa/Caching/Memory/EXTENSION_POINTS.md](../../src/Elsa/Caching/Memory/EXTENSION_POINTS.md) | Elsa.Caching.Memory | Elsa.Caching | Overridable contracts | `ICacheManager` *(Core — `Elsa.Caching.Core`)*<br>`IChangeTokenSignaler` *(Core — `Elsa.Caching.Core`)*<br>`IChangeTokenSignalInvoker` *(Core — `Elsa.Caching.Core`)* | 0 | yes |  |
 | [src/Elsa/Diagnostics/OpenTelemetry/EXTENSION_POINTS.md](../../src/Elsa/Diagnostics/OpenTelemetry/EXTENSION_POINTS.md) | Elsa.Diagnostics.OpenTelemetry | Elsa.Diagnostics | Overridable contracts<br>Ingestion & transport (not extension points)<br>Deferred<br>Notes | `IOpenTelemetryStore` *(Core — `Elsa.Diagnostics.OpenTelemetry.Core`)*<br>`IOpenTelemetryLiveFeed` *(Core — `Elsa.Diagnostics.OpenTelemetry.Core`)*<br>`IOpenTelemetryIngestor` *(Core — `Elsa.Diagnostics.OpenTelemetry.Core`)*<br>`IOpenTelemetryRedactor` *(Core — `Elsa.Diagnostics.OpenTelemetry.Core`)*<br>`IOpenTelemetrySourceRegistry` *(Core — `Elsa.Diagnostics.OpenTelemetry.Core`)*<br>`IOpenTelemetryProvider` *(Core — `Elsa.Diagnostics.OpenTelemetry.Core`)*<br>`ICollectorConfigurationProvider` *(Core — `Elsa.Diagnostics.OpenTelemetry.Core`)* | 0 | yes |  |
+| [src/Elsa/Diagnostics/Persistence/EXTENSION_POINTS.md](../../src/Elsa/Diagnostics/Persistence/EXTENSION_POINTS.md) | Elsa.Diagnostics.Persistence | Elsa.Diagnostics | Overridable contracts<br>Implementable contributor interfaces<br>Events<br>Production bridges and registration | `IDiagnosticsDrainTarget<TItem, TResult>` *(Feature contract — `Elsa.Diagnostics.Persistence`)*<br>`IDiagnosticsPersistenceObserver` *(Feature contract — `Elsa.Diagnostics.Persistence`)* | Adapter / Bridge; single Replacement.<br>Adapter / Bridge; single Replacement. | 0 | yes |
 | [src/Elsa/Diagnostics/StructuredLogs/EXTENSION_POINTS.md](../../src/Elsa/Diagnostics/StructuredLogs/EXTENSION_POINTS.md) | Elsa.Diagnostics.StructuredLogs | Elsa.Diagnostics | Overridable contracts<br>Persistence<br>Notes | `IStructuredLogStore` *(Core — `Elsa.Diagnostics.StructuredLogs.Core`)*<br>`IStructuredLogLiveFeed` *(Core — `Elsa.Diagnostics.StructuredLogs.Core`)*<br>`IStructuredLogLivePublisher` *(Core — `Elsa.Diagnostics.StructuredLogs.Core`)*<br>`IStructuredLogSink` *(Core — `Elsa.Diagnostics.StructuredLogs.Core`)*<br>`IStructuredLogSourceProvider` *(Core — `Elsa.Diagnostics.StructuredLogs.Core`)*<br>Groundwork diagnostic records<br>Temporary EF Core compatibility override | 0 | yes |  |
 | [src/Elsa/Events/EXTENSION_POINTS.md](../../src/Elsa/Events/EXTENSION_POINTS.md) | Elsa.Events | Elsa.Events | Implementable contributor interfaces<br>Publishing strategies | `IEventHandler<T>` *(Core — `Elsa.Events.Core`)* where `T : IEvent`<br>`IEvent` *(Core — `Elsa.Events.Core`)*<br>`IEventMiddleware` *(Core — `Elsa.Events.Core`)* | Contributor (event subscriber — handles a specific event type).<br>Marker interface — implement to define an event type. Typically a `sealed record`.<br>Contributor (event pipeline middleware). Composes the event dispatch pipeline. | 1 | yes |
 | [src/Elsa/Expressions/Api/EXTENSION_POINTS.md](../../src/Elsa/Expressions/Api/EXTENSION_POINTS.md) | Elsa.Expressions.Api | Elsa.Expressions | Descriptor Sources | 0 | yes |  |  |

specs/091-groundwork-diagnostics-persistence/checklists/requirements.md

@@ -0,0 +1,34 @@
+# Specification Quality Checklist: Durable Diagnostics Persistence
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-07-13
+**Feature**: [spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details beyond the ratified persistence boundary and named program dependency
+- [x] Focused on operator, host-owner, and maintainer outcomes
+- [x] Written so behavioral requirements can be reviewed independently of code structure
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No `[NEEDS CLARIFICATION]` markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria describe externally verifiable outcomes
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] Technology names appear only where they define the ratified provider boundary or a removal criterion
+
+## Notes
+
+- Ready for implementation planning. The existing diagnostics workload report remains the canonical detailed inventory; this specification links behavior into one executable work unit rather than duplicating that inventory.

specs/091-groundwork-diagnostics-persistence/contracts/diagnostics-persistence.md

@@ -0,0 +1,49 @@
+# Contract: Diagnostics Persistence Mapping
+
+This contract maps Elsa-owned operations to Groundwork capabilities. It is a behavioral contract, not a public promise of Groundwork types in Elsa core assemblies.
+
+## Structured Logs
+
+| Elsa operation | Required behavior | Groundwork realization |
+|---|---|---|
+| Append | Durable, ordered, idempotent batch commit; authoritative cursor | Append diagnostic records under explicit binding and operation identity |
+| Read recent | Exact filters, bounded newest selection, oldest-to-newest response | Indexed bounded record query with deterministic tie-breaks |
+| Capture after cursor | Binding validation, stable commit order, exact-once page, bounded advancement | Provider position query plus opaque Elsa cursor codec |
+| Trim | Keep exact newest count, including zero; high-water never rewinds | Idempotent bounded record retention operation |
+| Inspect/readiness | Visible schema/capability/operational failures | Groundwork validation plus adapter binding checks |
+
+Cursor-unavailable is reserved for malformed, trimmed, or foreign cursors. Cancellation, schema, provider, and serialization failures must not be translated to cursor-unavailable or empty results.
+
+## OpenTelemetry
+
+| Elsa operation | Required behavior | Groundwork realization |
+|---|---|---|
+| Write normalized batch | Nonblocking capture followed by durable, idempotent acknowledgement | Elsa drain plus one or more transactional/bounded Groundwork operations |
+| Query resources | Scoped filters and deterministic bounded catalog result | Groundwork document query over resource catalog |
+| Query traces | Inclusive ranges, exact filters, stable ordering and limits | Indexed diagnostic-record query over trace summaries |
+| Get trace | Exact summary, spans, resources, and related logs | Bounded coordinated queries by scoped trace/resource identities |
+| Query metrics | Instrument catalog plus filtered points and latest-per-key semantics | Document catalog query plus indexed metric-point record query |
+| Query logs | Exact declared filters, ordering, and limit | Indexed telemetry-log record query |
+| Get diagnostics | Exact counts/high-water/drop information | Scoped Groundwork inspection plus Elsa lifecycle counters |
+| Catalog retention | Deterministic least-recently-seen capacity | Bounded document mutation ordered by last-seen/tie-breaker |
+
+## Composition and deployment
+
+- The host selects one Groundwork provider and supplies provider-neutral store/session access.
+- Each concrete diagnostics Groundwork adapter owns and contributes its schema declaration; the shared Elsa drain/lifecycle project remains Groundwork-free.
+- Schema validation and application can run before application startup through the shared Groundwork CLI/tooling.
+- Readiness fails on missing schema, drift, capability mismatch, or unavailable storage.
+- Registration replaces only `IStructuredLogStore` and `IOpenTelemetryStore`; live feeds, redaction, parsing, authorization, and domain policy stay Elsa-owned.
+- Provider choice must not create provider-specific Elsa store classes or EF migration sets.
+
+## Cross-provider conformance
+
+SQLite, SQL Server, PostgreSQL, and MongoDB must execute the same fixture and assertions for ordering, filters, inclusive boundaries, latest selection, exact counts, retention, scope isolation, restart, concurrency, cancellation, acknowledgement loss, operation-identity conflict, readiness, and bounded execution evidence.
+
+## Removal gate
+
+Diagnostics EF projects may be deleted only when all shared conformance tests, provider readiness tests, execution-plan checks, lifecycle tests, and the ratified performance matrix pass. The final architecture test must prove:
+
+1. no diagnostics project references EF Core or an EF diagnostics project;
+2. no diagnostics EF context/entity/configuration/migration/registration remains; and
+3. diagnostics core projects do not reference Groundwork.

specs/091-groundwork-diagnostics-persistence/data-model.md

@@ -0,0 +1,80 @@
+# Data Model: Durable Diagnostics Persistence
+
+All identities below are scoped by an explicit `StorageScope`. Provider table/collection names are resolved by Groundwork naming and routing policy; Elsa does not embed provider-specific identifiers in its public contracts.
+
+## Storage binding
+
+- `StorageScope`: opaque host-selected isolation boundary
+- `SourceId`: diagnostics source binding
+- `StreamKind`: structured logs, traces, spans, metric points, or telemetry logs
+- `StreamId`: stable logical stream identity derived from scope, source, and kind
+- `SchemaVersion`: declared adapter schema version
+
+Validation rejects missing scope/source, unsupported schema, and a cursor or operation identity bound to a different tuple.
+
+## Immutable record streams
+
+### Common record envelope
+
+- Provider-issued durable sequence/position
+- Operation identity and item identity for idempotency
+- Commit timestamp
+- Occurrence timestamp
+- Stable tie-break fields required by the domain query
+- Filterable metadata declared by the adapter
+- Canonical payload
+
+### Structured Log record
+
+Adds logical display sequence, level, category, event, source, and correlation filter fields. The opaque Elsa replay cursor binds version, scope, source, stream, provider position, and record anchor; it exposes none of those values as a supported parsing contract.
+
+### Trace summary and span records
+
+Trace summary records support bounded latest/history queries. Span records support exact trace-detail lookup by trace identity, with stable span ordering.
+
+### Metric point records
+
+Metric points bind an instrument identity and support inclusive time ranges plus stable latest-per-logical-key selection where required by the existing contract.
+
+### Telemetry log records
+
+Telemetry logs support inclusive time, severity, trace, resource, and text-related filters exactly as declared by `OpenTelemetryLogFilter`.
+
+## Mutable catalogs
+
+### Resource catalog entry
+
+- Scoped resource identity
+- Resource attributes/payload
+- `LastSeenUtc`
+- Stable last-seen tie-breaker
+- Revision/concurrency token
+
+### Instrument catalog entry
+
+- Scoped instrument identity
+- Resource identity
+- Name, unit, description, and instrument metadata
+- `LastSeenUtc`
+- Stable last-seen tie-breaker
+- Revision/concurrency token
+
+Catalog upsert is idempotent. Capacity enforcement keeps exactly the newest configured entries by `(LastSeenUtc, tie-breaker)` within one scope. Capacity zero removes every current entry without changing immutable history high-water metadata.
+
+## Capture batch and operation ledger
+
+A capture batch contains one bounded normalized set for one storage binding. Its operation identity is stable across retries and restart. Reusing the identity for different canonical input fails. A committed operation returns the original authoritative outcome after acknowledgement loss.
+
+Retention operations use the same identity/fingerprint rule and record exact affected counts. Lifetime high-water and cursor sequence never rewind when records are trimmed.
+
+## Lifecycle state
+
+The Elsa drain moves through `Created -> Running -> Closing -> Draining -> Stopped/TimedOut`. Producers are nonblocking while running. Every accepted acknowledgement completes exactly once as committed or failed. Loss counters distinguish queue overflow, retry exhaustion, shutdown timeout, writes after closure, durable retention deletion, and subscriber delivery loss.
+
+## Relationships
+
+- One storage scope owns many source/stream bindings.
+- One resource catalog entry can be referenced by traces, spans, metric instruments, metric points, and telemetry logs.
+- One instrument catalog entry owns many metric point records.
+- One trace summary owns many span records and may relate to telemetry log records.
+- One replay position belongs to exactly one structured-log binding.