Does HPA autoscales based on custom metrics when keda is installed ? HPA autoscaling works fine using custom metrics when keda is not installed

[c4siva]

We are exploring Keda and installed for the first time on our EKS clusters using helm chart and default values. We already have HPA and datadog cluster agent running on our EKS clusters. We did ran into issues with v1beta1.external.metrics.k8s.io external metric apiservice and got those sorted out. We have the keda-admission-webhooks, keda-operator, keda-operator-metrics-apiserver deployment and pods running in kube-system namepsace.

Please find the logs from keda metrics apiserver
I0514 01:03:41.069653 1 welcome.go:34] "msg"="Starting metrics server" "logger"="keda_metrics_adapter"
I0514 01:03:41.069793 1 welcome.go:35] "msg"="KEDA Version: 2.17.0" "logger"="keda_metrics_adapter"
I0514 01:03:41.069803 1 welcome.go:36] "msg"="Git Commit: dafd9a8" "logger"="keda_metrics_adapter"
I0514 01:03:41.069816 1 welcome.go:37] "msg"="Go Version: go1.23.8" "logger"="keda_metrics_adapter"
I0514 01:03:41.069824 1 welcome.go:38] "msg"="Go OS/Arch: linux/amd64" "logger"="keda_metrics_adapter"
I0514 01:03:41.070003 1 welcome.go:39] "msg"="Running on Kubernetes 1.30+" "logger"="keda_metrics_adapter" "version"={"major":"1","minor":"30+","gitVersion
I0514 01:03:41.070433 1 maxprocs.go:31] "msg"="maxprocs: Updating GOMAXPROCS=1: determined from CPU quota" "logger"="keda_metrics_adapter"
I0514 01:03:41.071071 1 main.go:111] "msg"="Connecting Metrics Service gRPC client to the server" "address"="keda-operator.kube-system.svc.cluster.local:96
I0514 01:03:41.077929 1 provider.go:56] "msg"="starting" "logger"="keda_metrics_adapter.provider"
I0514 01:03:41.077956 1 main.go:272] "msg"="starting adapter..." "logger"="keda_metrics_adapter"
I0514 01:03:41.081284 1 client.go:103] "msg"="Waiting for establishing a gRPC connection to KEDA Metrics Server" "logger"="keda_metrics_adapter.provider"
I0514 01:03:41.081355 1 main.go:184] "msg"="starting /metrics server endpoint" "logger"="keda_metrics_adapter"
I0514 01:03:42.048820 1 secure_serving.go:213] Serving securely on :6443
I0514 01:03:42.050182 1 requestheader_controller.go:172] Starting RequestHeaderAuthRequestController
I0514 01:03:42.051148 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController
I0514 01:03:42.051339 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/certs/tls.crt::/certs/tls.key"
I0514 01:03:42.051820 1 tlsconfig.go:243] "Starting DynamicServingCertificateController"
I0514 01:03:42.052153 1 dynamic_cafile_content.go:160] "Starting controller" name="client-ca-bundle::/certs/ca.crt"
I0514 01:03:42.053709 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requesthea
I0514 01:03:42.053862 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-cl
I0514 01:03:42.106630 1 provider.go:64] "msg"="Connection to KEDA Metrics Service gRPC server has been successfully established" "logger"="keda_metrics_ada
I0514 01:03:42.151839 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController
I0514 01:03:42.155167 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-f

We have the HPA resources configured to scale using CPU, Memory and other external / custom metrics and all these were working fine before installing Keda. With Keda, HPA's configured to scale based on CPU and Memory metrics are working fine. However, HPA's configured to scale based on external / custom metrics are failing to read metric data from external metric apiservice with below error "scaledObject name is not specified":

FailedGetExternalMetric horizontal-pod-autoscaler unable to get external metric payment-qa0
7/payment_orchestrator.requests/&LabelSelector{MatchLabels:map[string]string{approle: payment-orchestrator,deployenv: qa07,
env: payment-qa07,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get payment_orchestrator.requests.external.metrics.k8s.io)
Warning FailedGetExternalMetric 3m24s (x4868 over 20h) horizontal-pod-autoscaler unable to get external metric payment-qa07/payment_orchestrator.requests/&LabelSelector{MatchLabels:map[string]string{approle: payment-orchestrator,deployenv: qa07,
env: payment-qa07,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: scaledObject name is not specified

HPA :
metadata:
annotations:
meta.helm.sh/release-name: payment-orchestrator-payment-qa07
meta.helm.sh/release-namespace: payment-qa07
creationTimestamp: "2024-09-13T20:14:32Z"
labels:
app: payment-orchestrator
app.kubernetes.io/instance: payment-orchestrator-payment-qa07
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: payment-orchestrator
approle: payment-orchestrator
service: payment-orchestrator
version: 2025.05.09b013
name: payment-orchestrator
namespace: payment-qa07
spec:
maxReplicas: 80
metrics:

• external:
  metric:
  name: payment_orchestrator.requests
  selector:
  matchLabels:
  approle: payment-orchestrator
  deployenv: qa07
  env: payment-qa07
  target:
  type: Value
  value: "30"
  type: External
• resource:
  name: cpu
  target:
  averageUtilization: 50
  type: Utilization
  type: Resource
  minReplicas: 6
  scaleTargetRef:
  apiVersion: apps/v1
  kind: Deployment
  name: payment-orchestrator

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/payment-qa07/payment_orchestrator.requests"
Error from server: scaledObject name is not specified

expected :
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/payment-uat/payment_orchestrator.requests"
{"kind":"ExternalMetricValueList","apiVersion":"external.metrics.k8s.io/v1beta1","metadata":{},"items":[{"metricName":"payment_orchestrator.requests","metricLabels":{"approle":"payment-orchestrator","deployenv":"uat","env":"payment-uat"},"timestamp":"2025-05-14T01:19:48Z","value":"411120n"}]}

Does HPA scales based on custom / external metrics when Keda is in installed and manages the v1beta1.external.metrics.k8s.io apiservice