kubernetes: Unable to decode an event from the watch stream: object to decode was longer than maximum allowed size

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened: Create a job with lots of envs. The job can’t run normally. Controller keeps logging Unable to decode an event from the watch stream: object to decode was longer than maximum allowed size.

[root@kubernetes-master yaml]# kubectl get jobs -o wide
NAME         DESIRED   SUCCESSFUL   AGE       CONTAINERS   IMAGES           SELECTOR
bigmetajob   100       0            12m       bigmetajob   chenchun/hello   controller-uid=d5a0ff6b-dee7-11e7-9ab9-080027242396

[root@kubernetes-master log]# tail -f kube-controller-manager.log 
E1212 02:58:19.428087   14503 streamwatcher.go:109] Unable to decode an event from the watch stream: object to decode was longer than maximum allowed size

What you expected to happen: The job should run normally or fail early on creating. How to reproduce it (as minimally and precisely as possible):

[root@kubernetes-master yaml]# cat bigmetajob.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: bigmetajob
spec:
  completions: 100
  parallelism: 100
  template:
    metadata:
      name: bigmetajob 
    spec:
      restartPolicy: Never
      containers:
      - name: bigmetajob
        image: chenchun/hello 
        imagePullPolicy: Never
        command: ["date"]
        env:
        - name: longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglongKey
          value: longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglongValue
        - name: longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglongKey
          value: longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglongValue
        - name: longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglongKey
          value: longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglongValue
        ...

[root@kubernetes-master yaml]# du -sh bigmetajob.yaml 
1.9M	bigmetajob.yaml

[root@kubernetes-master yaml]# kubectl create -f bigmetajob.yaml 
job "bigmetajob" created

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 19 (10 by maintainers)

Most upvoted comments

It’s good to see that the controller partially works. The big issue here is consistency, I would rather have the initial API request be rejected then experience inconsistent performance. Anything that uses client-go for watches will experience similar issues that may go undetected for a long time. Thank you for making the PR to fix this at the api layer!