kubernetes: Seeing OOMs on small nodes (g1-small) for node agents

Seems to be a regression on the kubelet-side. Ref job - https://k8s-testgrid.appspot.com/google-gce-scale#gce-scale-performance

Run 27 was fine (https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/27/artifacts/PodStartupLatency_density_2017-09-04T10:24:45Z.json) -

      "data": {
        "Perc100": 4482.739935,
        "Perc50": 1776.387197,
        "Perc90": 2446.611714,
        "Perc99": 3139.68287
      },
      "unit": "ms",
      "labels": {
        "Metric": "pod_startup"
      }

Run 28 regressed (https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/28/artifacts/PodStartupLatency_density_2017-09-06T20:20:22Z.json) -

      "data": {
        "Perc100": 25293.880645,
        "Perc50": 1800.683666,
        "Perc90": 2571.957657,
        "Perc99": 16023.080663
      },
      "unit": "ms",
      "labels": {
        "Metric": "pod_startup"
      }

cc @kubernetes/sig-node-bugs @kubernetes/sig-scalability-misc @yujuhong @Random-Liu

I’m looking at the diff to find the fauilty PR. Any leads would be appreciated 😃

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 28 (24 by maintainers)

Most upvoted comments

Event-exporter problem should be fixed now: https://github.com/kubernetes/kubernetes/pull/52263