kubernetes: CRDs merge-patch failing with "unable to find api field in struct Unstructured for the json field "metadata""

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened: Under heavy load (~50 similar request going into API simultaneously) I’m starting to sometimes get:

1 status.go:64] apiserver received an error that is not an metav1.Status: unable to find api field in struct Unstructured for the json field "metadata"
1 wrap.go:42] PATCH /apis/openstack.org/v1/kuryrports/5a114c6b-0f24-4202-b1ae-3669286416a1: (5.793432ms) 500

when trying to do an update on my CRD:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: kuryrports.openstack.org
spec:
  group: openstack.org
  version: v1
  scope: Cluster
  names:
    plural: kuryrports
    singular: kuryrport
    kind: KuryrPort
    shortNames:
    - kp

The update is in a form of application/merge-patch+json and its content look like this:

{
  "metadata": {
    "annotations": {
      "podName": ""
    },
    "resourceVersion": 123
  }
}

On failure the requests are repeated in 1 second intervals, if I wait long enough (few repeats of each) all of them will eventually succeed.

What you expected to happen:

I expect request to either succeed or get 409 Conflict if something else updated the resource before me.

How to reproduce it (as minimally and precisely as possible):

I don’t have a synthetic example, but I think What happened section describes the situation.

Anything else we need to know?:

API client is implemented in Python using requests library.

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T11:40:06Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T11:40:06Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:

QEMU VM on libvirt.

  • OS (e.g. from /etc/os-release):
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
  • Kernel (e.g. uname -a):
Linux localhost 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:

kuryr-kubernetes DevStack plugin (hyperkube in a container).

  • Others:

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 15 (10 by maintainers)

Commits related to this issue

Most upvoted comments

@nikhita @Stono - we tracked the helm issue down to an assumption that no longer holds in 1.9 around returned error types when attempting a conversion on a CRD object

see https://kubernetes.slack.com/archives/C0EG7JC6T/p1517858912000634 for discussion with @mattfarina