kubernetes: Namespace deletion fails because object does not implement protobuf marshalling interface

What happened: Namespace stuck in deletion because API resource does not implement the protobuf marshalling interface.

What you expected to happen: kube-controller-manager to be able to terminate the namespace.

How to reproduce it (as minimally and precisely as possible):

  1. Setup sample-apiserver@0.16.4 running on Kubernetes v1.16.4 (minikube provider) using the minikube walkthrough guide - https://github.com/kubernetes/sample-apiserver/blob/v0.16.4/docs/minikube-walkthrough.md.

  2. Create namespace

$ k create ns foo
  1. Delete the namespace and ensure the deletion fails
$ k delete ns foo

$ k get ns foo -o yaml
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2019-12-15T19:41:31Z"
  deletionTimestamp: "2019-12-15T19:41:33Z"
  name: foo
  resourceVersion: "798"
  selfLink: /api/v1/namespaces/foo
  uid: e5b47622-6789-4f41-a2bd-eb04aadc6d90
spec:
  finalizers:
  - kubernetes
status:
  conditions:
  - lastTransitionTime: "2019-12-15T19:41:38Z"
    message: All resources successfully discovered
    reason: ResourcesDiscovered
    status: "False"
    type: NamespaceDeletionDiscoveryFailure
  - lastTransitionTime: "2019-12-15T19:41:38Z"
    message: All legacy kube types successfully parsed
    reason: ParsedGroupVersions
    status: "False"
    type: NamespaceDeletionGroupVersionParsingFailure
  - lastTransitionTime: "2019-12-15T19:41:38Z"
    message: 'Failed to delete all resource types, 1 remaining: object *v1alpha1.FlunderList
      does not implement the protobuf marshalling interface and cannot be encoded
      to a protobuf message'
    reason: ContentDeletionFailed
    status: "True"
    type: NamespaceDeletionContentFailure
  phase: Terminating

$ k logs wardle-server-lgkcm -n wardle wardle-server

E1227 11:06:58.528242       1 writers.go:105] apiserver was unable to write a JSON response: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:06:58.909046       1 writers.go:105] apiserver was unable to write a JSON response: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:06:59.321062       1 writers.go:105] apiserver was unable to write a JSON response: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:07:00.314837       1 writers.go:105] apiserver was unable to write a JSON response: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:07:01.743636       1 writers.go:105] apiserver was unable to write a JSON response: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:07:04.592075       1 writers.go:105] apiserver was unable to write a JSON response: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:07:09.755761       1 writers.go:105] apiserver was unable to write a JSON response: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:07:20.189966       1 writers.go:105] apiserver was unable to write a JSON response: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:07:40.774322       1 writers.go:105] apiserver was unable to write a JSON response: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:08:21.927852       1 writers.go:105] apiserver was unable to write a JSON response: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:09:43.951662       1 writers.go:105] apiserver was unable to write a JSON response: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:11:57.372764       1 writers.go:105] apiserver was unable to write a JSON response: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message

$ k logs -n kube-system kube-controller-manager-minikube

E1227 11:07:01.906469       1 namespace_controller.go:148] deletion of namespace foo failed: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:07:04.611506       1 namespace_controller.go:148] deletion of namespace foo failed: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:07:09.862402       1 namespace_controller.go:148] deletion of namespace foo failed: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:07:20.210405       1 namespace_controller.go:148] deletion of namespace foo failed: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:07:40.807497       1 namespace_controller.go:148] deletion of namespace foo failed: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:08:21.965230       1 namespace_controller.go:148] deletion of namespace foo failed: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:09:44.008476       1 namespace_controller.go:148] deletion of namespace foo failed: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:11:57.431584       1 namespace_controller.go:148] deletion of namespace foo failed: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1227 11:16:57.455757       1 namespace_controller.go:148] deletion of namespace foo failed: object *v1alpha1.FlunderList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message

Anything else we need to know?: k8s.io/sample-apiserver types do not implement the protobuf marshalling interface but the kube-controller-manager does not fallback to json. Is it a hard requirement for Kubernetes v1.16 the api resources to implement the protobuf marshaling interface or I miss some configuration of kube-contoroller-manager?

Environment:

  • Kubernetes version (use kubectl version):
$ k version --short
Client Version: v1.16.1
Server Version: v1.16.3
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools: minikube
  • Network plugin and version (if this is a network-related bug):
  • Others:

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Comments: 25 (19 by maintainers)

Commits related to this issue

Most upvoted comments

To help others who build a custom apiserver with apiserver-builder-alpha, generate protobuf converters for your api with:

apiserver-boot build generated --generator protobuf

To be honest, the command doesn’t work for me because it doesn’t pass the --go-header-file argument to go-to-protobuf, but from the output you’ll work out the go-to-protobuf command to use.

See also: apiserver-builder-alpha/docs/protobuf_serialization.md

+3
EronWright on Aug 4, 2020

before it fixed in k8s, i bypass the problem use the code below. hope it will help. @ialidzhikov


type protocolShieldSerializers struct {
	*serializer.CodecFactory
	accepts []runtime.SerializerInfo
}

func newProtocolShieldSerializers(codecs *serializer.CodecFactory) *protocolShieldSerializers {
	if codecs == nil {
		return nil
	}
	pss := &protocolShieldSerializers{
		CodecFactory: codecs,
		accepts:      []runtime.SerializerInfo{},
	}
	for _, info := range codecs.SupportedMediaTypes() {
		if info.MediaType == runtime.ContentTypeProtobuf {
			continue
		}
		pss.accepts = append(pss.accepts, info)
	}
	return pss
}

func (pss *protocolShieldSerializers) SupportedMediaTypes() []runtime.SerializerInfo {
	if pss == nil {
		return nil
	}
	return pss.accepts
}

func (pss *protocolShieldSerializers) EncoderForVersion(encoder runtime.Encoder, gv runtime.GroupVersioner) runtime.Encoder {
	if pss == nil {
		return nil
	}
	return pss.CodecFactory.CodecForVersions(encoder, nil, gv, nil)
}
func (pss *protocolShieldSerializers) DecoderToVersion(decoder runtime.Decoder, gv runtime.GroupVersioner) runtime.Decoder {
	if pss == nil {
		return nil
	}
	return pss.CodecFactory.CodecForVersions(nil, decoder, nil, gv)
}

replace the default Serializers with the above one.

apiGroupInfo := genericapiserver.NewDefaultAPIGroupInfo(storage.GroupName, Scheme, metav1.ParameterCodec, Codecs)
apiGroupInfo.NegotiatedSerializer = newProtocolShieldSerializers(&Codecs)
+3
leopoldxx on May 19, 2020

Please open a new issue for your problems. Thank you!

+2
ialidzhikov on Jan 21, 2020
Read more comments on GitHub
← kubernetes: Nameserver not set when dnsPolicy=None
kubernetes: Namespace stuck in Terminating when deleted if ApiService doesn't implement Aggregated Discovery →