kubernetes: CRDs merge-patch failing with "unable to find api field in struct Unstructured for the json field "metadata""
Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened: Under heavy load (~50 similar request going into API simultaneously) I’m starting to sometimes get:
1 status.go:64] apiserver received an error that is not an metav1.Status: unable to find api field in struct Unstructured for the json field "metadata"
1 wrap.go:42] PATCH /apis/openstack.org/v1/kuryrports/5a114c6b-0f24-4202-b1ae-3669286416a1: (5.793432ms) 500
when trying to do an update on my CRD:
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: kuryrports.openstack.org
spec:
group: openstack.org
version: v1
scope: Cluster
names:
plural: kuryrports
singular: kuryrport
kind: KuryrPort
shortNames:
- kp
The update is in a form of application/merge-patch+json and its content look like this:
{
"metadata": {
"annotations": {
"podName": ""
},
"resourceVersion": 123
}
}
On failure the requests are repeated in 1 second intervals, if I wait long enough (few repeats of each) all of them will eventually succeed.
What you expected to happen:
I expect request to either succeed or get 409 Conflict if something else updated the resource before me.
How to reproduce it (as minimally and precisely as possible):
I don’t have a synthetic example, but I think What happened section describes the situation.
Anything else we need to know?:
API client is implemented in Python using requests library.
Environment:
- Kubernetes version (use
kubectl version):
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T11:40:06Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T11:40:06Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
- Cloud provider or hardware configuration:
QEMU VM on libvirt.
- OS (e.g. from /etc/os-release):
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
- Kernel (e.g.
uname -a):
Linux localhost 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
- Install tools:
kuryr-kubernetes DevStack plugin (hyperkube in a container).
- Others:
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 15 (10 by maintainers)
Commits related to this issue
- collapse patch conflict retry onto GuaranteedUpdate builds on #62868 1. When the incoming patch specified a resourceVersion that failed as a precondition, the patch handler would retry uselessly 5 t... — committed to liggitt/kubernetes by liggitt 6 years ago
- Merge pull request #63146 from liggitt/remove-patch-retry Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://... — committed to kubernetes/kubernetes by deleted user 6 years ago
@nikhita @Stono - we tracked the helm issue down to an assumption that no longer holds in 1.9 around returned error types when attempting a conversion on a CRD object
see https://kubernetes.slack.com/archives/C0EG7JC6T/p1517858912000634 for discussion with @mattfarina