When a controller registers an extra mapped input using qtransform.MapperFuncFromTyped, the destroy event of that input can get lost. The mapper reads the input resource to compute the mapping, and if the resource is already destroyed by the time the mapper runs, the read returns a not found error and the mapper returns no pointers:
https://github.com/cosi-project/runtime/blob/v1.16.1/pkg/controller/generic/qtransform/options.go#L75-L88
Because of this, the outputs depending on that input are never re-reconciled, and they keep the data derived from the destroyed resource forever.
Example with ClusterMachineStatusController, which maps MachineRequestStatus to ClusterMachine using the machine ID from its spec:
- A
MachineRequestStatus with a provider ID is created, and the matching ClusterMachineStatus gets its provision status populated from it.
- The
MachineRequestStatus is destroyed.
- The mapper runs for the destroy event, tries to read the resource, gets a not found error and returns no pointers.
- The
ClusterMachineStatus is never re-reconciled, so its provision status keeps pointing to a machine request that does not exist anymore.
It should clear the provision status instead.
This happens sporadically - the mapper processes events from a queue, so if it runs before the resource is actually destroyed, the read still succeeds and the mapping works. It only breaks when the destroy wins the race. There is a unit test reproducing it reliably (by putting the queue under pressure before the destroy) in #2640, closed in favor of this issue.
The problem affects the call sites where the mapping is computed from the spec of the input. Mappers that use only the metadata (e.g., MapperSameID, MapExtractLabelValue) are not affected, as the metadata is still available on the destroy event. All usages of MapperFuncFromTyped need to be checked one by one, atm these are:
internal/backend/runtime/omni/controllers/omni/cluster_machine_status.go
internal/backend/runtime/omni/controllers/omni/join_token_status.go
internal/backend/runtime/omni/controllers/omni/machine.go
internal/backend/runtime/omni/controllers/omni/machine_provision.go
internal/backend/runtime/omni/controllers/omni/pending_machine_status.go
internal/backend/runtime/omni/controllers/omni/service_account_status.go
The mapper cannot recover the mapping after the resource is gone, since the key lives in the spec. So the fix is probably not in MapperFuncFromTyped itself but on the call sites - we could register these inputs with WithExtraMappedDestroyReadyInput instead, so that the input is not destroyed before the controller observes its teardown and the mapper can still read it.
When a controller registers an extra mapped input using
qtransform.MapperFuncFromTyped, the destroy event of that input can get lost. The mapper reads the input resource to compute the mapping, and if the resource is already destroyed by the time the mapper runs, the read returns a not found error and the mapper returns no pointers:https://github.com/cosi-project/runtime/blob/v1.16.1/pkg/controller/generic/qtransform/options.go#L75-L88
Because of this, the outputs depending on that input are never re-reconciled, and they keep the data derived from the destroyed resource forever.
Example with
ClusterMachineStatusController, which mapsMachineRequestStatustoClusterMachineusing the machine ID from its spec:MachineRequestStatuswith a provider ID is created, and the matchingClusterMachineStatusgets its provision status populated from it.MachineRequestStatusis destroyed.ClusterMachineStatusis never re-reconciled, so its provision status keeps pointing to a machine request that does not exist anymore.It should clear the provision status instead.
This happens sporadically - the mapper processes events from a queue, so if it runs before the resource is actually destroyed, the read still succeeds and the mapping works. It only breaks when the destroy wins the race. There is a unit test reproducing it reliably (by putting the queue under pressure before the destroy) in #2640, closed in favor of this issue.
The problem affects the call sites where the mapping is computed from the spec of the input. Mappers that use only the metadata (e.g.,
MapperSameID,MapExtractLabelValue) are not affected, as the metadata is still available on the destroy event. All usages ofMapperFuncFromTypedneed to be checked one by one, atm these are:internal/backend/runtime/omni/controllers/omni/cluster_machine_status.gointernal/backend/runtime/omni/controllers/omni/join_token_status.gointernal/backend/runtime/omni/controllers/omni/machine.gointernal/backend/runtime/omni/controllers/omni/machine_provision.gointernal/backend/runtime/omni/controllers/omni/pending_machine_status.gointernal/backend/runtime/omni/controllers/omni/service_account_status.goThe mapper cannot recover the mapping after the resource is gone, since the key lives in the spec. So the fix is probably not in
MapperFuncFromTypeditself but on the call sites - we could register these inputs withWithExtraMappedDestroyReadyInputinstead, so that the input is not destroyed before the controller observes its teardown and the mapper can still read it.