FAILED FSx lifecycle state returns generic error, preventing user diagnosis

[infinite-turtles]

/kind bug

What happened?

When an FSx for Lustre filesystem enters the FAILED lifecycle state during provisioning, the PVC event shows a generic error with no failure reason:

reason=ProvisioningFailed type=Warning count=17
msg="failed to provision volume with StorageClass \"fsx-eu-central-1c\": rpc error: code = Internal
desc = Filesystem is not ready: unexpected state for filesystem fs-123456789: \"FAILED\""

1. FAILED is a documented terminal FSx lifecycle state ("unrecoverable failure"), but falls into the default branch of WaitForFileSystemAvailable and is reported as "unexpected state".

2. The FSx API provides FailureDetails.Message explaining why the filesystem failed. This is never surfaced.

3. The error repeats indefinitely (count=17 and growing) — CreateFileSystem uses the volume name as ClientRequestToken, so retries return the same failed filesystem via AWS idempotency.

Current code in pkg/cloud/cloud.go:

func (c *cloud) WaitForFileSystemAvailable(ctx context.Context, fileSystemId string) error {
    // ...
    switch string(fs.Lifecycle) {
    case "AVAILABLE":
        return true, nil
    case "CREATING":
        return false, nil
    default:
        // FAILED lands here — no FailureDetails, no typed error
        return true, fmt.Errorf("unexpected state for filesystem %s: %q", fileSystemId, string(fs.Lifecycle))
    }
}

What you expected to happen?

A dedicated case "FAILED" that returns a typed sentinel error with FailureDetails.Message, so the PVC event shows the root cause:

desc = Filesystem is not ready: filesystem reached terminal FAILED state: filesystem
fs-123456789: Amazon FSx is unable to create a new file system because the
specified subnet is out of available IP addresses

Sketch:

// pkg/cloud/cloud.go
var ErrFsLifecycleFailed = errors.New("filesystem reached terminal FAILED state")

// in WaitForFileSystemAvailable:
case "FAILED":
    msg := "unknown reason"
    if fs.FailureDetails != nil && fs.FailureDetails.Message != nil {
        msg = *fs.FailureDetails.Message
    }
    return true, fmt.Errorf("%w: filesystem %s: %s", ErrFsLifecycleFailed, fileSystemId, msg)

The typed error also lets the controller distinguish permanent from transient failures, a prerequisite for any potential cleanup of the orphaned FAILED filesystem (which would break the retry loop). The EBS CSI driver follows this pattern with typed sentinel errors from its cloud layer.

How to reproduce it (as minimally and precisely as possible)?

1. Create a StorageClass and PVC that triggers FSx for Lustre filesystem creation
2. Cause the filesystem to fail (e.g., subnet with no available IPs, invalid security group)
3. Observe PVC events — error says "unexpected state ... \"FAILED\"" with no detail

Anything else we need to know?

The failed filesystem remains in AWS as an orphaned resource since the PV is never created and DeleteVolume is never called. Cleanup is a related but separate concern that the typed error would enable.

Environment

• Kubernetes version: v1.34.4-eks-f69f56f
• Driver version: v1.9.0 (applies to latest)

[dlanov]

I'd like to work on this. I'll focus on handling the terminal FAILED lifecycle state, surfacing FailureDetails.Message, and adding focused regression coverage.