cert-manager: Cannot deploy CRDs to Kubernetes 1.11

Describe the bug:

When deploying CRDs to Kubernetes 1.11, they fail to apply with validation errors: https://prow.build-infra.jetstack.net/view/gcs/jetstack-logs/pr-logs/pull/jetstack_cert-manager/2199/pull-cert-manager-e2e-v1-11/1182598410243411968

The relevant error: must only have "properties", "required" or "description" at the root if the status subresource is enabled

This is because we specify type: object at the root of our OpenAPI schema: https://github.com/jetstack/cert-manager/blob/4ec682db20d2f030c5bbaac67c4f2f5301a7b081/deploy/manifests/00-crds.yaml#L1366

You can see in Kubernetes 1.11, this field is in fact not allowed: https://github.com/kubernetes/kubernetes/blob/9016740a6ffe91bb29824f80c34087b993903bd6/staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/validation/validation.go#L303-L306

We added it is a requirement to make our OpenAPI schema structural - discussing on #sig-api-machinery, it may be possible to relax the requirement to have this field in order for a schema to be structural, however it will require a bugfix/patch which will take some time to propogate through to cloud providers/managed service providers.

Expected behaviour:

We should either explicitly declare 1.11 as unsupported, or otherwise remove this field so that users can deploy to kubernetes 1.11.

For what it’s worth, 1.11 is now EOL, and EKS for example is upgrading all 1.11 users to 1.12 in November: https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html

ref #2064

Steps to reproduce the bug:

Attempt to apply the deploy/manifests/00-crds.yaml file against a Kubernetes 1.11 cluster.

Environment details::

Kubernetes version (e.g. v1.10.2): v1.11
Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): N/A
cert-manager version (e.g. v0.4.0): v0.11.0, v0.10.x
Install method (e.g. helm or static manifests): N/A

/kind bug /area deploy

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 6
Comments: 18 (8 by maintainers)

Commits related to this issue

fix(crd): remove breaking change when applying the on a v3.11 cluster I get: $ oc apply -f deploy/crds/gcp.managed.openshift.io_projectreferences_crd.yaml is invalid: spec.validation.openAPIV3Schema:... — committed to georgettica/gcp-project-operator by georgettica 4 years ago
fix(crd): remove breaking change when applying the on a v3.11 cluster I get: $ oc apply -f deploy/crds/gcp.managed.openshift.io_projectreferences_crd.yaml is invalid: spec.validation.openAPIV3Schema:... — committed to georgettica/gcp-project-operator by georgettica 4 years ago

Most upvoted comments

Works…

If you want a no brainer, just grab:

#!/bin/bash

# https://github.com/jetstack/cert-manager/issues/2200#L42
# ./install-crd.sh | oc create --validate=false -f -

curl -sL https://github.com/jetstack/cert-manager/releases/download/v0.12.0/cert-manager-openshift.yaml | \
  sed -e 's/preserveUnknownFields: false/preserveUnknownFields: true/g' | \
  sed -e 's/type: object//g'

jkroepke on Dec 24, 2019

I also this: must only have "properties", "required" or "description" at the root if the status subresource is enabled

And i comment this: #subresources: #status: {} That is OK.

yehaifeng on Oct 18, 2019

Hi, just saying that dropping support for Kubernetes 1.11 means that you also drop support for OpenShift v3.11 which is as for now used by many enterprises. Consider to revert this breaking change for the switchover period when enterprises will start using OpenShift 4.X which is Kubernetes 1.14 based.

grzechukol on Nov 8, 2019

I have the same error.

depohmel on Oct 18, 2019

This argument won’t actually help in specifically Kubernetes 1.11’s case - because it refuses to accept our CRD manifest as it sets the type: object field on our CRDs schema (as highlighted in the first post).

It’s a real PITA, and after discussion with #sig-api-machinery I think it may be one that we can’t work around without removing this field altogether, which will mean our CRD schemas are no longer ‘structural’ (which is a requirement for CRDv1).

Whilst we could remove this field and it will work for 1.11, I’m not sure if it’s really worthwhile (and it means we’re reverting progress forward). That said, I’m not too comfortable with the way we’ve discovered this, and the ‘surprise’ drop of support…

munnerz on Oct 11, 2019

Sadly that still is not possible - the full list of fields we have to ‘transform’ to create the 3.11 compatible CRDs can be found here: https://github.com/jetstack/cert-manager/blob/f3df90305c03c82c0ccf113d87cd6025de7eca3e/hack/filter-crd/main.go#L174-L186

Notably, we have to remove the new API version (v1alpha3) altogether, as well as a few other additional OpenAPI schema options (such as x-kubernetes-preserve-unknown-fields) to avoid users having to use --validate=false on all their kubectl calls (which is often not possible with GitOps tools).

I think we are stuck with supporting two variants of the CRDs for the foreseeable future - the earliest we could remove it is once Kubernetes v1.14 is no longer a supported platform 😞

munnerz on Apr 16, 2020

I found that commenting out the following section broke things later in the deployment.

subresources: 
  status: {}

Instead, I commented out the (somewhat redundant): type: object line inside each opernAPIV3Schema in the bundled cert-manager-openshift.yaml file before running oc apply -f cert-manager-openshift.yaml. This worked like a champ.

Carl4 on Jan 20, 2020

It might only affect some German users but I’d like to throw this one in: I just tried to install cert manager to Open Telekom Cloud (OTC) which is also stuck on K8S K8s v1.11.7-r1-CCE2.0.23.B001. I already asked when they consider to upgrade but until then I will also need to apply a workaround to make this run.

When I run kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.11.1/cert-manager.yaml no challenges, orders, certificaterequests, certificates, clusterissuers, or issuers are created.

(Edit with the change from @jkroepke I was able to install cert manager 0.11.1 as well as 0.12 (using the k8s version like so curl -sL https://github.com/jetstack/cert-manager/releases/download/v0.12.0/cert-manager.yaml | sed -e 's/preserveUnknownFields: false/preserveUnknownFields: true/g' | sed -e 's/type: object//g' > cert-manager-0.12.0-patched.yaml). Will report whether I can configure it to grab ACME certs on 0.12 and if that does not work whether it does with 0.11.1)

Edit2: I was able to install and use version 0.12 on the above mentioned OTC K8s cluster version v1.11.7-something with the above mentioned changes. I just received and deployed ACME certs successfully by pretty much following the standard tutorial, except that I used a cluster-issuer instead of an issuer.

tmerten on Dec 29, 2019

@munnerz

Since crds are inside the helm chart, its possible to add some conditions inside the crds? Like a hybrid chart for kubernetes and openshift?

jkroepke on Dec 25, 2019

Hm, it seems that 3.11 will be deprecated in June 2022 (for ‘full support’). This seems like a really long time for us to hinder the UX of users on 4.x and beyond… If we also want to continue to support 3.x, we’ll probably need to ship multiple different OpenShift variants because of this issue 🤦‍♂ https://access.redhat.com/support/policy/updates/openshift

munnerz on Dec 10, 2019