Add diagnostics when workflow state rehydration skips activity contexts or completion callbacks

[sfmskywalker]

Summary

WorkflowStateExtractor silently drops some persisted workflow-state entries while rehydrating a workflow execution context. Some skips are intentional for workflow version migration, but the same code paths can also hide stale or internally inconsistent state.

Add structured diagnostics that distinguish expected migration behavior from unexpected unresolved references, without changing the existing migration-compatible skip behavior.

Current behavior

Unresolved activity execution contexts

In ApplyActivityExecutionContextsAsync, CreateActivityExecutionContextAsync resolves the persisted ScheduledActivityNodeId against the target workflow graph:

var activity = workflowExecutionContext.FindActivityByNodeId(activityExecutionContextState.ScheduledActivityNodeId);

// Activity can be null in case the workflow instance was migrated to a newer version that no longer contains this activity.
if (activity == null)
    return null;

The caller filters out the null result without logging it. Removing an activity during an intentional workflow migration is valid, but the same path is taken for a stale/corrupt node ID or another unexpected definition mismatch.

Unresolved completion callbacks

In ApplyCompletionCallbacks, a callback is silently skipped when:

• its persisted OwnerInstanceId does not resolve to a rehydrated activity execution context; or
• its persisted ChildNodeId does not resolve in the target workflow graph.

A missing owner may be a downstream consequence of an activity execution context having been skipped. The callback cannot be restored generically without its owner, but silently discarding it makes the condition difficult to diagnose and can leave a composite activity without its child-completion notification.

Proposed behavior

Log structured warnings when rehydration skips:

1. An activity execution context because ScheduledActivityNodeId cannot be resolved.
2. A completion callback because OwnerInstanceId cannot be resolved.
3. A completion callback because ChildNodeId cannot be resolved.

Where available, include:

• workflow instance ID;
• persisted workflow definition ID/version ID/version;
• target workflow definition ID/version ID/version;
• activity execution context ID;
• scheduled activity node ID;
• completion callback owner instance ID;
• completion callback child node ID;
• whether the persisted and target definition versions differ.

When the persisted and target definition version IDs differ, identify the skip as compatible with workflow migration. When they are the same, identify it as unexpected so operators can distinguish expected migration cleanup from possible state inconsistency.

Warnings should be concise and structured to support log queries and alerting. Consider avoiding duplicate warnings for the same skipped item within one rehydration operation.

Acceptance criteria

• An unresolved activity execution context produces a structured warning.
• The warning distinguishes a definition-version migration from an unexpected same-version miss.
• A completion callback with a missing owner produces a structured warning.
• A completion callback with a missing child node produces a structured warning.
• Intentional workflow migration continues to skip removed activity contexts rather than throwing.
• Tests cover expected migration skips and unexpected unresolved references.
• Tests verify the relevant workflow, definition, context, owner, and child identifiers are included in diagnostics.

Out of scope

• Automatically dropping orphaned callbacks during extraction as a general repair strategy. A callback may be required for its owner composite to complete, so that behavior needs a separate reproducible, semantics-preserving design.
• Changing the existing ExtractCompletionCallbacks fail-fast invariant.
• Treating these observations as a confirmed cause of Quartz, MassTransit, Kubernetes, or memory-pressure incidents without a corresponding reproduction or stack trace.

Version context

The silent skip paths are present in Elsa 3.7.1 and current main.

Related issues

• [BUG]fail: Microsoft.AspNetCore.Server.Kestrel[13] Connection id "0HN5CTU3SMF6S", Request id "0HN5CTU3SMF6S:00000003": An unhandled exception was thrown by the application. #5843 — older open report of Lost an owner context during extraction.
• "Lost an owner context" error while trying to complete a Join activity using WaitAny, on a workflow using a before/after/always commit strategy. #7335 — reproducible WaitAny/flowchart ordering case, since fixed.
• Refactor inbound connection handling to schedule outbound activity before canceling remaining activities #7340 — schedules outbound work before cancelling remaining inbound work; included in 3.7.1.

[sfmskywalker]

Closed via #7899