kubernetes: Unable to decode an event from the watch stream: object to decode was longer than maximum allowed size
Is this a BUG REPORT or FEATURE REQUEST?:
Uncomment only one, leave it on its own line:
/kind bug
/kind feature
What happened:
Create a job with lots of envs. The job can’t run normally. Controller keeps logging Unable to decode an event from the watch stream: object to decode was longer than maximum allowed size.
[root@kubernetes-master yaml]# kubectl get jobs -o wide
NAME DESIRED SUCCESSFUL AGE CONTAINERS IMAGES SELECTOR
bigmetajob 100 0 12m bigmetajob chenchun/hello controller-uid=d5a0ff6b-dee7-11e7-9ab9-080027242396
[root@kubernetes-master log]# tail -f kube-controller-manager.log
E1212 02:58:19.428087 14503 streamwatcher.go:109] Unable to decode an event from the watch stream: object to decode was longer than maximum allowed size
What you expected to happen: The job should run normally or fail early on creating. How to reproduce it (as minimally and precisely as possible):
[root@kubernetes-master yaml]# cat bigmetajob.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: bigmetajob
spec:
completions: 100
parallelism: 100
template:
metadata:
name: bigmetajob
spec:
restartPolicy: Never
containers:
- name: bigmetajob
image: chenchun/hello
imagePullPolicy: Never
command: ["date"]
env:
- name: longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglongKey
value: longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglongValue
- name: longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglongKey
value: longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglongValue
- name: longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglongKey
value: longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglongValue
...
[root@kubernetes-master yaml]# du -sh bigmetajob.yaml
1.9M bigmetajob.yaml
[root@kubernetes-master yaml]# kubectl create -f bigmetajob.yaml
job "bigmetajob" created
Anything else we need to know?:
Environment:
- Kubernetes version (use
kubectl version): - Cloud provider or hardware configuration:
- OS (e.g. from /etc/os-release):
- Kernel (e.g.
uname -a): - Install tools:
- Others:
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 19 (10 by maintainers)
It’s good to see that the controller partially works. The big issue here is consistency, I would rather have the initial API request be rejected then experience inconsistent performance. Anything that uses client-go for watches will experience similar issues that may go undetected for a long time. Thank you for making the PR to fix this at the api layer!