fall back to kubeconfig loader behavior (exec plugin capable)

[reaper8055]

Description

This PR fixes #985

Notes for Reviewers

Added useKubeconfigAuth := c.RestConfig.ExecProvider != nil && c.RestConfig.BearerToken == "" to createHelmActionConfig

1. c.RestConfig.ExecProvider != nil means auth came from an exec plugin (EKS style aws eks get-token, etc).
2. c.RestConfig.BearerToken == "" means there is no static token available in the rest.Config right now.

(1.) && (2.) means:

Cluster needs exec-plugin token retrieval and static bearer-token wiring is insufficient.
Therefore, use kubeconfig-based auth resolution path (so client-go can execute plugin), instead of forcing /dev/null and expecting bearer token to already exist.

Signed commits

• Yes, I signed my commits.

[gemini-code-assist]

Code Review

This pull request updates the Helm chart application logic to support exec-auth kubeconfigs by conditionally bypassing the override of the local kubeconfig. A critical issue was identified where essential configuration parameters, such as the API server address and insecure skip TLS verify flag, are omitted when using this new authentication path, potentially causing Helm to target the wrong cluster.

[gemini-code-assist]

When useKubeconfigAuth is true, the APIServer and Insecure settings (as well as basic auth credentials) are not applied to the kubeConfig flags. This causes Helm to fall back to the default values found in the local kubeconfig file, which may point to a different cluster than the one intended by the Client's RestConfig. These settings should be applied unconditionally to ensure consistency with the client's configuration even when using the kubeconfig loader for authentication.

Suggested change

	if !useKubeconfigAuth {
	// Set KubeConfig to DevNull to prevent read from local kubeconfig
	// to prevent conflicts between "data" and "files" properties (CAFile, CAData and KeyFile, KeyData)
	// ConfigFlags only allows setting CAFile, KeyFile but not CAData, KeyData.
	// When the library reads the original kubeconfig containing cert data / key data AND we specify cert file / key file, these configurations conflict
	devNull := os.DevNull
	kubeConfig.KubeConfig = &devNull
	kubeConfig.APIServer = &c.RestConfig.Host
	kubeConfig.BearerToken = &c.RestConfig.BearerToken
	kubeConfig.Insecure = &c.RestConfig.Insecure
	// Set username and password for basic auth if available
	if c.RestConfig.Username != "" {
	kubeConfig.Username = &c.RestConfig.Username
	}
	if c.RestConfig.Password != "" {
	kubeConfig.Password = &c.RestConfig.Password
	}
	kubeConfig.APIServer = &c.RestConfig.Host
	kubeConfig.Insecure = &c.RestConfig.Insecure
	// Set username and password for basic auth if available
	if c.RestConfig.Username != "" {
	kubeConfig.Username = &c.RestConfig.Username
	}
	if c.RestConfig.Password != "" {
	kubeConfig.Password = &c.RestConfig.Password
	}
	if !useKubeconfigAuth {
	// Set KubeConfig to DevNull to prevent read from local kubeconfig
	// to prevent conflicts between "data" and "files" properties (CAFile, CAData and KeyFile, KeyData)
	// ConfigFlags only allows setting CAFile, KeyFile but not CAData, KeyData.
	// When the library reads the original kubeconfig containing cert data / key data AND we specify cert file / key file, these configurations conflict
	devNull := os.DevNull
	kubeConfig.KubeConfig = &devNull
	kubeConfig.BearerToken = &c.RestConfig.BearerToken

[leecalcote]

When useKubeconfigAuth is true, kubeConfig.KubeConfig is left unset. Client-go falls back to $KUBECONFIG on the user's system, which could point to a different cluster. Helm potentially silently installs Meshery into the wrong cluster.

We'll probably want to serializing the in-memory kubeconfig (including the ExecProvider) to a temp file, appending it to tempFiles, and pointing kubeConfig.KubeConfig at that file so cluster/context stay bound to what the user provided.

As we work through this, let's be aware of the flows initiated via both Meshery CLI and Server.

[leecalcote]

Test coverage 👀

[leecalcote]

It'd be good to get our follow up issue for GKE, OKE, AKS, opened and referenced now.

[reaper8055]

When useKubeconfigAuth is true, kubeConfig.KubeConfig is left unset. Client-go falls back to $KUBECONFIG on the user's system, which could point to a different cluster. Helm potentially silently installs Meshery into the wrong cluster.

Yes this can happen if the user runs mesheryctl system start -p kubernetes without running aws eks update-kubeconfig first, but since the user would have run aws eks update-kubeconfig --name [YOUR_CLUSTER_NAME] --region [YOUR_REGION] as per this https://docs.meshery.io/installation/kubernetes/eks/#in-cluster-installation the current context for k8s cluster in the $KUBECONFIG would be pointing to the right cluster.

Maybe, we can have a prompt saying:

meshery is going to install in <cluster-name> obtained from your current context in $KUBECONFIG, are you sure to proceed (y/n)? 

We'll probably want to serializing the in-memory kubeconfig (including the ExecProvider) to a temp file, appending it to tempFiles, and pointing kubeConfig.KubeConfig at that file so cluster/context stay bound to what the user provided.

I'm assuming, we would want to somehow persist this file across start/stop commands? And since $KUBECONFIG is updated by aws eks update-kubeconfig, one way to do this is to read the current kubecontext and copy it to a temp file using:

f, err := os.CreateTemp("/tmp", "meshery-eks-kubeconfig")

OR

Initiate the EKS flow using aws sdk which generates the in memory config and use client-go to write this config to a temp file instead of relying on user to run aws update kube-config. In this case, we will have to prompt the user to input the cluster-name and region for their EKS cluster similar to what mesheryctl connection create command does.

As we work through this, let's be aware of the flows initiated via both Meshery CLI and Server.

ACK!

The saving of current config depends on the remote provider, which happens in two paths:

• Kubeconfig -> k8s connection payload (+ credential secret)

  • addK8SConfig() calls provider.SaveK8sContext()
  • Remote path is RemoteProvider.SaveK8sContext()
  • It sends CredentialSecret: {"auth":..., "cluster":...} through SaveConnection()

• Post-login self-registration (second meshery connection)

[reaper8055]

Test coverage 👀

Hello @leecalcote do you want me to increase the test coverage for this package? The current test coverage looks low:

ok  github.com/meshery/meshkit/utils/kubernetes	1.188s          coverage: 7.9% of statements
    github.com/meshery/meshkit/utils/kubernetes/describe        coverage: 0.0% of statements
    github.com/meshery/meshkit/utils/kubernetes/expose	        coverage: 0.0% of statements
ok  github.com/meshery/meshkit/utils/kubernetes/kompose	1.564s	coverage: 75.4% of statements

This can be done but will require a fair bit of refactoring of logic especially to be able to mock behavior effectively.

[lekaf974]

@reaper8055 just wanted to confirm this is ready for review ?

[reaper8055]

@reaper8055 just wanted to confirm this is ready for review ?

Yes @lekaf974 this is ready for review

[lekaf974]

small comments LGTM otherwise

[PragalvaXFREZ]

LGTM

[leecalcote]

We're a couple of months into this one. I have my 🤞 for it.

[reaper8055]

We're a couple of months into this one. I have my 🤞 for it.

Resoling the comments and updating the PR as I write this. Will get this done!

[aabidsofi19]

@reaper8055 regarding error handling are we bubbling up errors like missing the exec binary , failure to exec etc to users