Skip to content

qtransform.MapperFuncFromTyped drops the destroy event of the mapped input #3257

Description

@utkuozdemir

When a controller registers an extra mapped input using qtransform.MapperFuncFromTyped, the destroy event of that input can get lost. The mapper reads the input resource to compute the mapping, and if the resource is already destroyed by the time the mapper runs, the read returns a not found error and the mapper returns no pointers:

https://github.com/cosi-project/runtime/blob/v1.16.1/pkg/controller/generic/qtransform/options.go#L75-L88

Because of this, the outputs depending on that input are never re-reconciled, and they keep the data derived from the destroyed resource forever.

Example with ClusterMachineStatusController, which maps MachineRequestStatus to ClusterMachine using the machine ID from its spec:

  1. A MachineRequestStatus with a provider ID is created, and the matching ClusterMachineStatus gets its provision status populated from it.
  2. The MachineRequestStatus is destroyed.
  3. The mapper runs for the destroy event, tries to read the resource, gets a not found error and returns no pointers.
  4. The ClusterMachineStatus is never re-reconciled, so its provision status keeps pointing to a machine request that does not exist anymore.

It should clear the provision status instead.

This happens sporadically - the mapper processes events from a queue, so if it runs before the resource is actually destroyed, the read still succeeds and the mapping works. It only breaks when the destroy wins the race. There is a unit test reproducing it reliably (by putting the queue under pressure before the destroy) in #2640, closed in favor of this issue.

The problem affects the call sites where the mapping is computed from the spec of the input. Mappers that use only the metadata (e.g., MapperSameID, MapExtractLabelValue) are not affected, as the metadata is still available on the destroy event. All usages of MapperFuncFromTyped need to be checked one by one, atm these are:

  • internal/backend/runtime/omni/controllers/omni/cluster_machine_status.go
  • internal/backend/runtime/omni/controllers/omni/join_token_status.go
  • internal/backend/runtime/omni/controllers/omni/machine.go
  • internal/backend/runtime/omni/controllers/omni/machine_provision.go
  • internal/backend/runtime/omni/controllers/omni/pending_machine_status.go
  • internal/backend/runtime/omni/controllers/omni/service_account_status.go

The mapper cannot recover the mapping after the resource is gone, since the key lives in the spec. So the fix is probably not in MapperFuncFromTyped itself but on the call sites - we could register these inputs with WithExtraMappedDestroyReadyInput instead, so that the input is not destroyed before the controller observes its teardown and the mapper can still read it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions