kubernetes: server-side dry-run apply on custom resource fails no-op conversion with error about missing metadata
What happened?
On a new 1.27 kubernetes cluster, Using server-side dry-run to apply a custom resource with an older version results in the following error:
$ kubectl apply --server-side --dry-run=server -fclusters/my-clusters/image-repo.yaml --field-manager=kustomize-controller
Error from server: failed to prune fields: failed add back owned items: failed to convert pruned object at version image.toolkit.fluxcd.io/v1beta2: conversion for image.toolkit.fluxcd.io/v1beta2, Kind=ImagePolicy returned invalid metadata in object at index 1: missing metadata in converted object
Error from server: failed to prune fields: failed add back owned items: failed to convert pruned object at version image.toolkit.fluxcd.io/v1beta2: conversion for image.toolkit.fluxcd.io/v1beta2, Kind=ImagePolicy returned invalid metadata in object at index 1: missing metadata in converted object
This error seems to be flaky, occasionally the dry run passes for one or both of them.
What did you expect to happen?
For dry-run to happen successfully
How can we reproduce it (as minimally and precisely as possible)?
- Create a new Kubernetes cluster
- Install a CRD that has an older version
- Server-side apply a custom resource that still has the older version specifying a field manager
kubectl apply --server-side --dry-run=server -f image.yaml --field-manager=kustomize-controller
Anything else we need to know?
You ca see an example of the crd at https://github.com/fluxcd/image-reflector-controller/blob/main/config/crd/bases/image.toolkit.fluxcd.io_imagerepositories.yaml
I don’t get the error when attempting dry-run apply with client
Kubernetes version
$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:33:11Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.0", GitCommit:"1b4df30b3cdfeaba6024e81e559a6cd09a089d65", GitTreeState:"clean", BuildDate:"2023-04-11T20:52:43Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/arm64"}
Cloud provider
kind
OS version
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here
# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
Install tools
Container runtime (CRI) and version (if applicable)
kind
Related plugins (CNI, CSI, …) and versions (if applicable)
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 19 (16 by maintainers)
Commits related to this issue
- Revert: Revert: Clean up CRD conversion code structure This is a new attempt on #113151 and #114959, which got reverted in #117301 because they seem to be causing #117260. Signed-off-by: Dr. Stefan ... — committed to sttts/kubernetes by sttts 10 months ago
- apiextensions/integration: add test for https://github.com/kubernetes/kubernetes/issues/117260 Signed-off-by: Dr. Stefan Schimanski <stefan.schimanski@gmail.com> — committed to sttts/kubernetes by sttts 9 months ago
@liggitt thank you so much for the quick turnaround. When 1.27.1 gets released we’ll let Flux users know they can safely upgrade.
bisected to https://github.com/kubernetes/kubernetes/pull/114959 as the point at which the symptom started with that reproducer
script to reproduce on a local cluster with etcd locally available:
chatted offline and got these reproducer steps:
etcd.json:
apply.yaml:
The dry-run apply inconsistently fails.