test-infra: Pods are OOMKilling itself
senlu@senlu:~/work/src/k8s.io/test-infra/prow$ kubectl get po -n=test-pods -a | grep OOM | wc -l
50
We probably want to set memory limit for each job, like regular e2e jobs use ~1Gi, however bazel job can eat up ~7Gi
"sacrifice child!", said by the node
/area prow /area jobs
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 1
- Comments: 22 (22 by maintainers)
2017-11-14 23:30:02.823312: {'Completed': 238, 'Error': 112, 'Evicted': 2, 'Running': 209}We’re in a pretty good place for the moment.
seems we already have the resources fields in type.go, I’ll add them for jobs using bazel build
This seems to still be a problem. #5457 was deployed before the weekend, right?
FYI, current status:
https://kubernetes.io/docs/tasks/administer-cluster/memory-default-namespace/
we should probably add some configuration for this, as well as look at configuring some of the more intensive jobs (jobs with builds) to request more