velero: Known Issue: CRDs cannot be restored due to fields being identical values

Edit from @nrb -> Please find a summary of the causes HERE. THIS IS A KNOWN ISSUE WITH A FIX TARGETTED FOR VERSION 1.4

What steps did you take and what happened: I’ve made a backup of my namespaces with elastic-operator, elasticsearch, kibana and filebeat.

Then I’ve deleted them with following script:

$ kubectl delete ns elastic-system elk-stack filebeat

Then I’ve restored it. The restored Elasticsearch+Filebeat+Kibana seems to work, but velero showed errors.

What did you expect to happen: Restore without errors.

The output of the following commands will help us better understand what’s going on:

$ velero restore logs elk-backup-20200401140210 | grep error
time="2020-04-01T12:02:11Z" level=info msg="error restoring elasticsearches.elasticsearch.k8s.elastic.co: CustomResourceDefinition.apiextensions.k8s.io \"elasticsearches.elasticsearch.k8s.elastic.co\" is invalid: [spec.versions[0].additionalPrinterColumns[0].JSONPath: Required value, spec.versions[0].additionalPrinterColumns[1].JSONPath: Required value, spec.versions[0].additionalPrinterColumns[2].JSONPath: Required value, spec.versions[0].additionalPrinterColumns[3].JSONPath: Required value, spec.versions[0].additionalPrinterColumns[4].JSONPath: Required value, spec.versions[1].additionalPrinterColumns[0].JSONPath: Required value, spec.versions[1].additionalPrinterColumns[1].JSONPath: Required value, spec.versions[1].additionalPrinterColumns[2].JSONPath: Required value, spec.versions[1].additionalPrinterColumns[3].JSONPath: Required value, spec.versions[1].additionalPrinterColumns[4].JSONPath: Required value, spec.versions[2].additionalPrinterColumns[0].JSONPath: Required value, spec.versions[2].additionalPrinterColumns[1].JSONPath: Required value, spec.versions[2].additionalPrinterColumns[2].JSONPath: Required value, spec.versions[2].additionalPrinterColumns[3].JSONPath: Required value, spec.versions[2].additionalPrinterColumns[4].JSONPath: Required value, spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:\"v1\", Served:true, Storage:true, Schema:(*apiextensions.CustomResourceValidation)(0xc015f52f60), Subresources:(*apiextensions.CustomResourceSubresources)(0xc02abd9700), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Elasticsearch version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"phase\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1beta1\", Served:true, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc015f52f68), Subresources:(*apiextensions.CustomResourceSubresources)(0xc02abd9f60), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Elasticsearch version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"phase\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1alpha1\", Served:false, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc015f52f70), Subresources:(*apiextensions.CustomResourceSubresources)(0xc021e5e7a0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Elasticsearch version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"phase\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}}: per-version schemas may not all be set to identical values (top-level validation should be used instead), spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:\"v1\", Served:true, Storage:true, Schema:(*apiextensions.CustomResourceValidation)(0xc015f52f60), Subresources:(*apiextensions.CustomResourceSubresources)(0xc02abd9700), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Elasticsearch version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"phase\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1beta1\", Served:true, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc015f52f68), Subresources:(*apiextensions.CustomResourceSubresources)(0xc02abd9f60), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Elasticsearch version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"phase\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1alpha1\", Served:false, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc015f52f70), Subresources:(*apiextensions.CustomResourceSubresources)(0xc021e5e7a0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Elasticsearch version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"phase\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}}: per-version subresources may not all be set to identical values (top-level subresources should be used instead), spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:\"v1\", Served:true, Storage:true, Schema:(*apiextensions.CustomResourceValidation)(0xc015f52f60), Subresources:(*apiextensions.CustomResourceSubresources)(0xc02abd9700), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Elasticsearch version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"phase\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1beta1\", Served:true, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc015f52f68), Subresources:(*apiextensions.CustomResourceSubresources)(0xc02abd9f60), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Elasticsearch version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"phase\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1alpha1\", Served:false, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc015f52f70), Subresources:(*apiextensions.CustomResourceSubresources)(0xc021e5e7a0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Elasticsearch version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"phase\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}}: per-version additionalPrinterColumns may not all be set to identical values (top-level additionalPrinterColumns should be used instead)]" logSource="pkg/restore/restore.go:1199" restore=velero/elk-backup-20200401140210
time="2020-04-01T12:02:12Z" level=info msg="error restoring kibanas.kibana.k8s.elastic.co: CustomResourceDefinition.apiextensions.k8s.io \"kibanas.kibana.k8s.elastic.co\" is invalid: [spec.versions[0].additionalPrinterColumns[0].JSONPath: Required value, spec.versions[0].additionalPrinterColumns[1].JSONPath: Required value, spec.versions[0].additionalPrinterColumns[2].JSONPath: Required value, spec.versions[0].additionalPrinterColumns[3].JSONPath: Required value, spec.versions[1].additionalPrinterColumns[0].JSONPath: Required value, spec.versions[1].additionalPrinterColumns[1].JSONPath: Required value, spec.versions[1].additionalPrinterColumns[2].JSONPath: Required value, spec.versions[1].additionalPrinterColumns[3].JSONPath: Required value, spec.versions[2].additionalPrinterColumns[0].JSONPath: Required value, spec.versions[2].additionalPrinterColumns[1].JSONPath: Required value, spec.versions[2].additionalPrinterColumns[2].JSONPath: Required value, spec.versions[2].additionalPrinterColumns[3].JSONPath: Required value, spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:\"v1\", Served:true, Storage:true, Schema:(*apiextensions.CustomResourceValidation)(0xc033543c70), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00a6d7c00), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Kibana version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1beta1\", Served:true, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc033543c78), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00ad500f0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Kibana version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1alpha1\", Served:false, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc033543c80), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00ad50500), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Kibana version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}}: per-version schemas may not all be set to identical values (top-level validation should be used instead), spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:\"v1\", Served:true, Storage:true, Schema:(*apiextensions.CustomResourceValidation)(0xc033543c70), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00a6d7c00), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Kibana version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1beta1\", Served:true, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc033543c78), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00ad500f0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Kibana version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1alpha1\", Served:false, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc033543c80), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00ad50500), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Kibana version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}}: per-version subresources may not all be set to identical values (top-level subresources should be used instead), spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:\"v1\", Served:true, Storage:true, Schema:(*apiextensions.CustomResourceValidation)(0xc033543c70), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00a6d7c00), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Kibana version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1beta1\", Served:true, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc033543c78), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00ad500f0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Kibana version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}, apiextensions.CustomResourceDefinitionVersion{Name:\"v1alpha1\", Served:false, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(0xc033543c80), Subresources:(*apiextensions.CustomResourceSubresources)(0xc00ad50500), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:\"health\", Type:\"string\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"nodes\", Type:\"integer\", Format:\"\", Description:\"Available nodes\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"version\", Type:\"string\", Format:\"\", Description:\"Kibana version\", Priority:0, JSONPath:\"\"}, apiextensions.CustomResourceColumnDefinition{Name:\"age\", Type:\"date\", Format:\"\", Description:\"\", Priority:0, JSONPath:\"\"}}}}: per-version additionalPrinterColumns may not all be set to identical values (top-level additionalPrinterColumns should be used instead)]" logSource="pkg/restore/restore.go:1199" restore=velero/elk-backup-20200401140210

It shows erros when restoring CRDs but afterwards they’re restored:

$ kubectl get crd -A | grep elastic
apmservers.apm.k8s.elastic.co                  2020-01-20T16:36:56Z
elasticsearches.elasticsearch.k8s.elastic.co   2020-01-20T16:36:56Z
kibanas.kibana.k8s.elastic.co                  2020-01-20T16:36:56Z

kubectl logs deployment/velero -n velero https://termbin.com/uvbx
velero restore describe elk-backup-20200401140210 https://termbin.com/k87h
velero restore logs elk-backup-20200401140210 https://termbin.com/u23p

Environment:

Velero version (use velero version):

Client:
	Version: v1.3.1
	Git commit: -
Server:
	Version: v1.3.1

Velero features (use velero client config get features):

features: <NOT SET>

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-13T18:08:14Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:18:29Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes installer & version: Manually installed with kubeadm. Version 1.16.
Cloud provider or hardware configuration: Physical hardware + KVM VMs.
OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 17 (13 by maintainers)

Most upvoted comments

A summary so far:

Right now, Velero only gets the server’s preferred version of a resource.

In Kubernetes v1.16+, that’s v1 of CustomResourceDefinitions.

With v1.3.0, Velero applies a heuristic to see if it had a CRD that was v1 but was really a v1beta1 CRD retrieved through the v1 endpoint. If so, then the backup plugin switched the version string from v1 to v1beta1, and that’s it.

As documented in this issue, that’s the wrong approach, because the overall structure returned from the v1 endpoint is invalid for v1 CRDs. Our sample size of CRDs was too small to catch this, which will be remedied in #2447.

I believe the heuristics on backup are working - they’re catching the v1beta1 CRDs and applying the string swap, which is what then causes this problem.

We have 3 options for fixing the current behavior, captured from the discussion at the community meeting on April 21, 2020:

Keep the heuristic, and if matched, use a v1beta1 CRD client to fetch the correct version of the object. This stays in line with Velero’s current approach of dropping k8s cluster objects directly into the tarball.
Explore adjusting the Velero backup format to store CRDs as apiextension.CustomResourceDefinition, which is the unversioned, internal format that the Kubernetes API server uses. This would very likely trigger a major version bump for the backup file format version, and thus a major version bump for Velero. That’s because the current expectation is that the files inside the Velero backup tarball are assumed to be directly usable via kubectl without any conversion. Using an internal format breaks that assumption. It would require conversion at back up and restore time.
Building on #2373, remove the current CRD backup/restore plugins. Instead, try to restore v1 of a CRD first. If it fails, try to restore v1beta1, since it’s already captured in the backup tarball.

Of these options, I think 1 & 3 are viable for the v1.4 timeframe. 3 builds on existing work. 1 duplicates some of 3’s logic.

Long term, I think 2 may be the most correct thing to do, but I haven’t done the proper research to know what it fully entails.

All of these versions special case the CustomResourceDefinition API group, which I think is unavoidable.

The upstream CRD code states that v1beta1 is planned for removal in v1.19. However, Velero will need to be able to take v1beta1 backups and restore them going forward longer than this window.

@vmware-tanzu/velero-owners Please let me know what your thoughts are! I’d like to dig into this in earnest this week.

@hasheddan Do you have anything to add? I see you’ve worked on the upstream CRD code - have I gotten anything wrong in this summary?

nrb on Apr 21, 2020

@hasheddan Thanks for digging into this! Velero shouldn’t be adding anything extra to the schemas, but I could be mistaken.

The CRD manipulations we do are in these plugins, if you’d like to double check my work which would be appreciated 😃 These were introduced with v1.3.0

Backup:

https://github.com/vmware-tanzu/velero/blob/master/pkg/backup/remap_crd_version_action.go

Restore:

https://github.com/vmware-tanzu/velero/blob/master/pkg/restore/crd_v1_preserve_unknown_fields_action.go

Thanks for that small CRD, too…I was wondering if it was something about the definitions themselves, but given your data, clearly something else is going on.

nrb on Apr 20, 2020

I’ve got a branch working with the elasticsearch CRDs now, but I need to update the unit tests to account for the changes. PR should be up tomorrow.

nrb on Apr 28, 2020

@hasheddan Storing the internal representation does seem like a good idea, especially since https://github.com/kubernetes/apiextensions-apiserver/blob/c47e10e6d5a3d95d427c5bee109d934602f0861e/pkg/apis/apiextensions/v1/zz_generated.conversion.go#L306 exists.

I think I’ll have to revisit the CRD backup/restore logic entirely. Thank you for your pointers! Do you mind if I tag you on reviews for these changes?

nrb on Apr 20, 2020

@hasheddan No worries! I was capturing notes as I found things. I’ll take a look at that approach.

We also have #2373 which will get both versions, too.

nrb on Apr 20, 2020

Ok, I think I see what’s happening now.

Using kubectl get --raw /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/elasticsearches.elasticsearch.k8s.elastic.co, I see that I get the correct object; that is, there’s only 1 top-level additionalPrinterColumns entry, not one per version.

I got a hint from this commit, which said that v1 CRDs don’t support the top-level field anymore, and also remembered that our plugin just sets the version string, while keeping the object that was fetched from the preferred version endpoint (which on Kubernetes v1.16+ would have been a v1 CRD)

Instead, I think the correct path will be to get a proper v1beta1 client and retrieve the CRD from the correct endpoint on backup.

nrb on Apr 20, 2020

Hello, I’ve used ECK operator to install it: https://www.elastic.co/guide/en/cloud-on-k8s/current/index.html

And then I’ve used this YAML to create Elasticsearch:

---

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: ${ELK_STACK_NAMESPACE}
spec:
  version: 7.5.2
  nodeSets:

  - name: master-dc1
    count: 2
    config:
      node.master: true
      node.data: false
      node.ingest: false
      node.ml: true
      xpack.slm.enabled: true
      path.repo: ["/mount/backups"]
      node.attr.zone: dc1
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      metadata:
        annotations:
          backup.velero.io/backup-volumes: elasticsearch-data,elasticsearch-backup
        labels:
          k8s-app: ${ELK_STACK_NAMESPACE}
          component: elasticsearch
          tier: master
      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: site
                  operator: In
                  values:
                  - dc1
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
        - name: elasticsearch
          resources:
            requests:
              memory: 1Gi
              cpu: "0.2"
            limits:
              memory: 5Gi
              cpu: "2"
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms1g -Xmx1g"
          volumeMounts:
            - mountPath: "/mount/backups"
              name: elasticsearch-backup
        volumes:
        - name: elasticsearch-data
          emptyDir: {}
        - name: elasticsearch-backup
          persistentVolumeClaim:
            claimName: elasticsearch-backup


  - name: master-dc2
    count: 2
    config:
      node.master: true
      node.data: false
      node.ingest: false
      node.ml: true
      xpack.slm.enabled: true
      path.repo: ["/mount/backups"]
      node.attr.zone: dc2
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      metadata:
        annotations:
          backup.velero.io/backup-volumes: elasticsearch-data,elasticsearch-backup
        labels:
          k8s-app: ${ELK_STACK_NAMESPACE}
          component: elasticsearch
          tier: master
      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: site
                  operator: In
                  values:
                  - dc2
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
        - name: elasticsearch
          resources:
            requests:
              memory: 1Gi
              cpu: "0.2"
            limits:
              memory: 5Gi
              cpu: "2"
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms1g -Xmx1g"
          volumeMounts:
            - mountPath: "/mount/backups"
              name: elasticsearch-backup
        volumes:
        - name: elasticsearch-data
          emptyDir: {}
        - name: elasticsearch-backup
          persistentVolumeClaim:
            claimName: elasticsearch-backup


  - name: data-dc1
    count: 2
    config:
      node.master: false
      node.data: true
      node.ingest: true
      node.ml: true
      xpack.slm.enabled: true
      path.repo: ["/mount/backups"]
      node.attr.zone: dc1
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      metadata:
        annotations:
          backup.velero.io/backup-volumes: elasticsearch-data,elasticsearch-backup
        labels:
          k8s-app: ${ELK_STACK_NAMESPACE}
          component: elasticsearhc
          tier: data
      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: site
                  operator: In
                  values:
                  - dc1
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
        - name: elasticsearch
          # specify resource limits and requests
          resources:
            requests:
              memory: ${ELK_DATA_XMX_SIZE}Gi
              cpu: "0.5"
            limits:
              memory: 16Gi
              cpu: "3"
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms${ELK_DATA_XMX_SIZE}g -Xmx${ELK_DATA_XMX_SIZE}g"
          volumeMounts:
            - mountPath: "/mount/backups"
              name: elasticsearch-backup
        volumes:
        - name: elasticsearch-data
          emptyDir: {}
        - name: elasticsearch-backup
          persistentVolumeClaim:
            claimName: elasticsearch-backup


  - name: data-dc2
    count: 2
    config:
      node.master: false
      node.data: true
      node.ingest: true
      node.ml: true
      xpack.slm.enabled: true
      path.repo: ["/mount/backups"]
      node.attr.zone: dc2
      cluster.routing.allocation.awareness.attributes: zone
    podTemplate:
      metadata:
        annotations:
          backup.velero.io/backup-volumes: elasticsearch-data,elasticsearch-backup
        labels:
          k8s-app: ${ELK_STACK_NAMESPACE}
          component: elasticsearhc
          tier: data
      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: site
                  operator: In
                  values:
                  - dc2
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
        - name: elasticsearch
          # specify resource limits and requests
          resources:
            requests:
              memory: ${ELK_DATA_XMX_SIZE}Gi
              cpu: "0.5"
            limits:
              memory: 16Gi
              cpu: "3"
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms${ELK_DATA_XMX_SIZE}g -Xmx${ELK_DATA_XMX_SIZE}g"
          volumeMounts:
            - mountPath: "/mount/backups"
              name: elasticsearch-backup
        volumes:
        - name: elasticsearch-data
          emptyDir: {}
        - name: elasticsearch-backup
          persistentVolumeClaim:
            claimName: elasticsearch-backup


  http:
    tls:
      certificate:
        # provide our own certificate
        secretName: ${ELK_STACK_NAMESPACE}.${EXTERNAL_DOMAIN}-tls

PS. I use envsubst to resolve the shell variables.

TomaszKlosinski on Apr 14, 2020