Just annoying message (loading OpenAPI spec for "v1alpha1...." failed)

[EnochTheWise]

Hi ;o)

kubernetes: v1.22.4
cert-manager: v1.9.1
cert-manager-webhook-pdns: v2.2.0

kube-apiserver log messages:

E0810 07:18:21.893411       1 controller.go:116] loading OpenAPI spec for "v1alpha1.fintechiq" failed with: OpenAPI spec does not exist
I0810 07:18:21.893429       1 controller.go:129] OpenAPI AggregationController: action for item v1alpha1.fintechiq: Rate Limited Requeue.

[zachomedia]

Hello @EnochTheWise, can you elaborate on what the issue is with the cert-manager-webhook-pdns? Can you provide logs from the cert-manager-webhook-pdns pod?

As an aside, I would say your issue in the kube-apiserver logs is with the fintechiq custom resource and not anything related to this project.

[EnochTheWise]

Hi 8)

Hello @EnochTheWise, can you elaborate on what the issue is with the cert-manager-webhook-pdns? Can you provide logs from the cert-manager-webhook-pdns pod?

E0815 02:59:57.617695       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.1/tools/cache/reflector.go:167: Failed to watch *v1beta2.FlowSchema: failed to list *v1beta2.FlowSchema: the server could not find the requested resource
W0815 02:59:58.536460       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.1/tools/cache/reflector.go:167: failed to list *v1beta2.PriorityLevelConfiguration: the server could not find the requested resource

Hello @zachomedia , i have same issue as @EnochTheWise.
Logs from pod is

W0914 18:02:05.994387       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.1/tools/cache/reflector.go:167: failed to list *v1beta2.FlowSchema: the server could not find the requested resource
E0914 18:02:05.994439       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.1/tools/cache/reflector.go:167: Failed to watch *v1beta2.FlowSchema: failed to list *v1beta2.FlowSchema: the server could not find the requested resource
W0914 18:02:21.547800       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.1/tools/cache/reflector.go:167: failed to list *v1beta2.PriorityLevelConfiguration: the server could not find the requested resource
E0914 18:02:21.547838       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.1/tools/cache/reflector.go:167: Failed to watch *v1beta2.PriorityLevelConfiguration: failed to list *v1beta2.PriorityLevelConfiguration: the server could not find the requested resource

meanwhile cluster not have this api-resources:

root@mycluster-master-01 ~ # kubectl api-resources --verbs=list | grep -i flow
flowschemas                     flowcontrol.apiserver.k8s.io/v1beta1   false    FlowSchema
prioritylevelconfigurations     flowcontrol.apiserver.k8s.io/v1beta1   false    PriorityLevelConfiguration

so i think ACME resolver try to work with not existed resources
Is it possible to switch v1beta2.flow... to v1beta1.flow...?

---

googled that this error not affects anything.

[zachomedia]

@EnochTheWise / @ABess-Flant Can you share what version of Kubernetes you're running? I will see if I can reproduce the issue. There is nothing in my code that is loading those resources, so it might be something with the Kubernetes libraries.

k8s version 1.21.14.
But i suppose that some libraries ( maybe go lib ) queries all know versions and print those, which can't find.
I successfully set this webhook up, so this error not affect anything.

[EnochTheWise]

A some addition.
ApiService is exist:

$ k get apiservices v1alpha1.v1alpha1.fintechiq
NAME               SERVICE                                  AVAILABLE   AGE
v1alpha1.fintechiq   cert-manager/cert-manager-webhook-pdns   True        64d

[necipakca]

We have same issue :

E1218 13:26:46.615437       1 controller.go:102] loading OpenAPI spec for "v1alpha1.acme.zacharyseguin.ca" failed with: failed to download v1alpha1.acme.zacharyseguin.ca: resource not found                                                     
I1218 13:26:46.615465       1 controller.go:109] OpenAPI AggregationController: action for item v1alpha1.acme.zacharyseguin.ca: Rate Limited Requeue.

This is not breaking any functionality but annoying to see those logs. Also I don't know what is real impact on kube-apiserver.
k8s : v1.29.5
cert-manager-webhook-pdns : v2.5.1

[iceman91176]

We have seen this issues for years with various versions of cert-manager, cert-manager-webhook-pdns and kubernetes.
The kube-apiserver logs are spammed with

E1114 09:52:43.840547       1 controller.go:146] "Unhandled Error" err="Error updating APIService \"v1alpha1.acme-powerdns.witcom.services\" with err: failed to download v1alpha1.acme-powerdns.witcom.services: resource not found" logger="UnhandledError"
E1114 09:52:44.828832       1 controller.go:102] "Unhandled Error" err="loading OpenAPI spec for \"v1alpha1.acme-powerdns.witcom.services\" failed with: failed to download v1alpha1.acme-powerdns.witcom.services: resource not found" logger="UnhandledError"
I1114 09:54:36.887407       1 controller.go:109] OpenAPI AggregationController: action for item v1alpha1.acme-powerdns.witcom.services: Rate Limited Requeue.

The api-service has been registered

NAME                                     SERVICE                                                AVAILABLE   AGE
v1alpha1.acme-powerdns.witcom.services   cert-manager/acme-powerdns-cert-manager-webhook-pdns   True        14m

Querying it is possible

kubectl get --raw /apis/acme-powerdns.witcom.services/v1alpha1 | jq
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "acme-powerdns.witcom.services/v1alpha1",
  "resources": [
    {
      "name": "pdns",
      "singularName": "ChallengePayload",
      "namespaced": false,
      "group": "webhook.acme.cert-manager.io",
      "version": "v1alpha1",
      "kind": "ChallengePayload",
      "verbs": [
        "create"
      ]
    }
  ]
}

On a fresh install of cert-manager-webhook-pdns (v2.5.1) k8s 1.33.x we are seeing the following

During the first startup of cert-manager-webhook-pdns the registered api-service is not available yet so some legitimate errors are logged (503s, etc.)

I1114 09:52:32.910731       1 alloc.go:328] "allocated clusterIPs" service="cert-manager/acme-powerdns-cert-manager-webhook-pdns" clusterIPs={"IPv4":"10.96.187.157"}
I1114 09:52:32.919929       1 cidrallocator.go:277] updated ClusterIP allocator for Service CIDR 10.96.0.0/16
I1114 09:52:32.924666       1 handler.go:288] Adding GroupVersion acme-powerdns.witcom.services v1alpha1 to ResourceManager
W1114 09:52:32.929136       1 handler_proxy.go:99] no RequestInfo found in the context
E1114 09:52:32.929394       1 controller.go:146] "Unhandled Error" err=<
        Error updating APIService "v1alpha1.acme-powerdns.witcom.services" with err: failed to download v1alpha1.acme-powerdns.witcom.services: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
        , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
 > logger="UnhandledError"
I1114 09:52:32.937660       1 controller.go:667] quota admission added evaluator for: certificates.cert-manager.io
I1114 09:52:32.937718       1 controller.go:667] quota admission added evaluator for: certificates.cert-manager.io
I1114 09:52:32.947241       1 controller.go:667] quota admission added evaluator for: issuers.cert-manager.io
W1114 09:52:33.465022       1 handler_proxy.go:99] no RequestInfo found in the context
E1114 09:52:33.465073       1 controller.go:146] "Unhandled Error" err=<
        Error updating APIService "v1alpha1.acme-powerdns.witcom.services" with err: failed to download v1alpha1.acme-powerdns.witcom.services: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
        , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
 > logger="UnhandledError"
W1114 09:52:33.925052       1 handler_proxy.go:99] no RequestInfo found in the context
W1114 09:52:33.925117       1 handler_proxy.go:99] no RequestInfo found in the context
E1114 09:52:33.925190       1 controller.go:113] "Unhandled Error" err="loading OpenAPI spec for \"v1alpha1.acme-powerdns.witcom.services\" failed with: Error, could not get list of group versions for APIService" logger="UnhandledError"
E1114 09:52:33.925309       1 controller.go:102] "Unhandled Error" err=<
        loading OpenAPI spec for "v1alpha1.acme-powerdns.witcom.services" failed with: failed to download v1alpha1.acme-powerdns.witcom.services: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
        , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
 > logger="UnhandledError"
I1114 09:52:33.926387       1 controller.go:126] OpenAPI AggregationController: action for item v1alpha1.acme-powerdns.witcom.services: Rate Limited Requeue.
I1114 09:52:33.926433       1 controller.go:109] OpenAPI AggregationController: action for item v1alpha1.acme-powerdns.witcom.services: Rate Limited Requeue.
E1114 09:52:35.546726       1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"context canceled\"}: context canceled" logger="UnhandledError"
E1114 09:52:35.547834       1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1114 09:52:35.549021       1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1114 09:52:35.550308       1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1114 09:52:35.551814       1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="5.486318ms" method="GET" path="/apis/fluxcd.controlplane.io/v1/namespaces/cert-manager/resourcesets/cert-manager" result=null
E1114 09:52:38.820553       1 remote_available_controller.go:448] "Unhandled Error" err="v1alpha1.acme-powerdns.witcom.services failed with: failing or missing response from https://10.96.187.157:443/apis/acme-powerdns.witcom.services/v1alpha1: Get \"https://10.96.187.157:443/apis/acme-powerdns.witcom.services/v1alpha1\": dial tcp 10.96.187.157:443: connect: connection refused" logger="UnhandledError"
W1114 09:52:38.820562       1 handler_proxy.go:99] no RequestInfo found in the context
E1114 09:52:38.820662       1 controller.go:146] "Unhandled Error" err=<
        Error updating APIService "v1alpha1.acme-powerdns.witcom.services" with err: failed to download v1alpha1.acme-powerdns.witcom.services: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
        , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
 > logger="UnhandledError"
W1114 09:52:39.821719       1 handler_proxy.go:99] no RequestInfo found in the context
W1114 09:52:39.821753       1 handler_proxy.go:99] no RequestInfo found in the context
E1114 09:52:39.821825       1 controller.go:102] "Unhandled Error" err=<
        loading OpenAPI spec for "v1alpha1.acme-powerdns.witcom.services" failed with: failed to download v1alpha1.acme-powerdns.witcom.services: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
        , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
 > logger="UnhandledError"
E1114 09:52:39.821828       1 controller.go:113] "Unhandled Error" err="loading OpenAPI spec for \"v1alpha1.acme-powerdns.witcom.services\" failed with: Error, could not get list of group versions for APIService" logger="UnhandledError"
I1114 09:52:39.823031       1 controller.go:126] OpenAPI AggregationController: action for item v1alpha1.acme-powerdns.witcom.services: Rate Limited Requeue.
I1114 09:52:39.823068       1 controller.go:109] OpenAPI AggregationController: action for item v1alpha1.acme-powerdns.witcom.services: Rate Limited Requeue.
W1114 09:52:43.826664       1 handler_proxy.go:99] no RequestInfo found in the context
E1114 09:52:43.826690       1 remote_available_controller.go:448] "Unhandled Error" err="v1alpha1.acme-powerdns.witcom.services failed with: failing or missing response from https://10.96.187.157:443/apis/acme-powerdns.witcom.services/v1alpha1: Get \"https://10.96.187.157:443/apis/acme-powerdns.witcom.services/v1alpha1\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" logger="UnhandledError"
E1114 09:52:43.826724       1 controller.go:146] "Unhandled Error" err=<
        Error updating APIService "v1alpha1.acme-powerdns.witcom.services" with err: failed to download v1alpha1.acme-powerdns.witcom.services: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
        , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
 > logger="UnhandledError"
I1114 09:52:43.834345       1 handler.go:288] Adding GroupVersion acme-powerdns.witcom.services v1alpha1 to ResourceManager
E1114 09:52:43.840547       1 controller.go:146] "Unhandled Error" err="Error updating APIService \"v1alpha1.acme-powerdns.witcom.services\" with err: failed to download v1alpha1.acme-powerdns.witcom.services: resource not found" logger="UnhandledError"

But they soon disapear and only the loading OpenAPI spec errors remain.

The reason seems to be that cert-manager-webhook-pdns does not publish its OpenApi-Specs, and the kubernetes AggregstionController tries to get them over and over again.

A similar issue has been described here for the kubernetes metrics-server. Initially there was the idea to handle such api-services differently to avoid emitting those error. But it never came to a solution on the kube-api-server-side.

But it was fixed in metrics-server - they started to serve their open-api schema

Here is the commit - maybe someone who is more ivolved with Go than I am can pull out soem relevant information ;-)

kubernetes-sigs/metrics-server@06bcca5