NonExistentActivationException thrown on shutdown for client response messages

[ReubenBond]

I was investigating some flakiness in the TransactionWillRecoverAfterRandomSiloUnGracefulShutdown tests and encountered this.

When MessageCenter shuts down, it sets Gateway to null and that results in this exception when sending client responses, because TryDeliverToProxy returns false (Gateway is null) and TargetSilo is the local silo, so the silo tries to handle the message locally. Catalog eventually throws this exception from GetOrCreateActivation.

Note that the GrainId in the above image refers to a client.

This results in a degraded experience for clients during graceful shutdown (eg, rolling update) scenarios.

We are marking this issue as stale due to the lack of activity in the past six months. If there is no further activity within two weeks, this issue will be closed. You can always create a new issue based on the guidelines provided in our pinned announcement.

[ReubenBond]

This was fixed incidentally by later messaging refactors and is no longer reproducible on main.

The original failure had two parts:

• At the issue-era commit (b181de63a1e4c77c62cd1c5d96e1ef98a31dc055), MessageCenter.StopAcceptingClientMessages stopped the gateway and assigned Gateway = null.
• A client-addressed response which then looped back locally reached Dispatcher.ReceiveMessage, which called Catalog.GetOrCreateActivation before checking Message.Direction. Since the target grain id represented a client, activation lookup could throw NonExistentActivationException.

Two main-line changes removed that path:

• 5356f648e246293ff2b028db4b77c106cd7b0e79 (Add Orleans.Serialization library as a high-fidelity, version-tolerant serializer #7070, 2021-06-17) made Gateway get-only and stopped nulling it during shutdown.
• dee08479f5f84f307a458b8eaf38600c98f9c31f (Refactor activation message flow #7153, 2021-08-24) classified undeliverable responses before Catalog.GetOrCreateActivation, routing them to InsideRuntimeClient.ReceiveResponse instead.

The current path preserves normal proxy routing first (TryDeliverToProxy). If the client is unavailable, a locally addressed response loops through ReceiveMessage, fails proxy delivery, and is handled as a response without attempting grain activation.

I verified this with a deterministic probe which stopped client-message acceptance and sent a response addressed to a disconnected client through the local silo. It completed without exception on both net8.0 and net10.0.

This is separate from #10165 / #10539, which tracks requests forwarded through a live gateway when their destination silo subsequently dies.