test-infra: prow.k8s.io: build cluster failing to start jobs

Example job link.

Failing to start the pod with:

  Warning  FailedCreatePodSandBox  4m9s (x153 over 37m)  kubelet, gke-prow-containerd-pool-bigger-170c3937-fzx9  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6769374aadfa0a00be6c6e7b2be9b6da812dcc496c1c2ca0ab4b8ec1c611fe15": failed to allocate for range 0: no IP addresses available in range set: 10.60.28.1-10.60.28.254

Debug info

kubectl get prowjob 
apiVersion: prow.k8s.io/v1
kind: ProwJob
metadata:
  annotations:
    prow.k8s.io/job: pull-test-infra-bazel
  creationTimestamp: 2019-06-14T16:28:09Z
  generation: 3
  labels:
    created-by-prow: "true"
    event-GUID: 64edf720-8ec1-11e9-917a-ebca143d37d5
    preset-bazel-scratch-dir: "true"
    preset-service-account: "true"
    prow.k8s.io/id: 6555d8aa-8ec1-11e9-a433-9ee7625cd1db
    prow.k8s.io/job: pull-test-infra-bazel
    prow.k8s.io/refs.org: kubernetes
    prow.k8s.io/refs.pull: "13038"
    prow.k8s.io/refs.repo: test-infra
    prow.k8s.io/type: presubmit
  name: 6555d8aa-8ec1-11e9-a433-9ee7625cd1db
  namespace: default
  resourceVersion: "235042395"
  selfLink: /apis/prow.k8s.io/v1/namespaces/default/prowjobs/6555d8aa-8ec1-11e9-a433-9ee7625cd1db
  uid: 655622fe-8ec1-11e9-8ebe-42010a800112
spec:
  agent: kubernetes
  cluster: default
  context: pull-test-infra-bazel
  decoration_config:
    gcs_configuration:
      bucket: kubernetes-jenkins
      default_org: kubernetes
      default_repo: kubernetes
      path_strategy: legacy
    gcs_credentials_secret: service-account
    grace_period: 15s
    timeout: 2h0m0s
    utility_images:
      clonerefs: gcr.io/k8s-prow/clonerefs:v20190610-3be53b072
      entrypoint: gcr.io/k8s-prow/entrypoint:v20190610-3be53b072
      initupload: gcr.io/k8s-prow/initupload:v20190610-3be53b072
      sidecar: gcr.io/k8s-prow/sidecar:v20190610-3be53b072
  job: pull-test-infra-bazel
  namespace: test-pods
  pod_spec:
    containers:
    - args:
      - test
      - --config=ci
      - --nobuild_tests_only
      - //...
      command:
      - hack/bazel.sh
      env:
      - name: GOOGLE_APPLICATION_CREDENTIALS
        value: /etc/service-account/service-account.json
      - name: E2E_GOOGLE_APPLICATION_CREDENTIALS
        value: /etc/service-account/service-account.json
      - name: TEST_TMPDIR
        value: /bazel-scratch/.cache/bazel
      image: launcher.gcr.io/google/bazel:0.26.0
      name: ""
      resources: {}
      volumeMounts:
      - mountPath: /etc/service-account
        name: service
        readOnly: true
      - mountPath: /bazel-scratch/.cache
        name: bazel-scratch
    volumes:
    - name: service
      secret:
        secretName: service-account
    - emptyDir: {}
      name: bazel-scratch
  refs:
    base_link: https://github.com/kubernetes/test-infra/commit/f1b872102fc8841673644eca2f32c30b16b4f6ca
    base_ref: master
    base_sha: f1b872102fc8841673644eca2f32c30b16b4f6ca
    org: kubernetes
    pulls:
    - author: stevekuznetsov
      author_link: https://github.com/stevekuznetsov
      commit_link: https://github.com/kubernetes/test-infra/pull/13038/commits/410c774b4a2fbf4e6f1b150e307fc8f9a12082a5
      link: https://github.com/kubernetes/test-infra/pull/13038
      number: 13038
      sha: 410c774b4a2fbf4e6f1b150e307fc8f9a12082a5
    repo: test-infra
    repo_link: https://github.com/kubernetes/test-infra
  report: true
  rerun_command: /test pull-test-infra-bazel
  type: presubmit
status:
  build_id: "1139570293107331073"
  description: Job triggered.
  pod_name: 6555d8aa-8ec1-11e9-a433-9ee7625cd1db
  prev_report_states:
    github-reporter: pending
  startTime: 2019-06-14T16:28:09Z
  state: pending
  url: https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/test-infra/13038/pull-test-infra-bazel/1139570293107331073
kubectl get pod 
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/limit-ranger: 'LimitRanger plugin set: memory request for container
      test; memory request for container sidecar; memory request for init container
      clonerefs; memory request for init container initupload; memory request for
      init container place-entrypoint'
    prow.k8s.io/job: pull-test-infra-bazel
  creationTimestamp: 2019-06-14T16:28:24Z
  labels:
    created-by-prow: "true"
    event-GUID: 64edf720-8ec1-11e9-917a-ebca143d37d5
    preset-bazel-scratch-dir: "true"
    preset-service-account: "true"
    prow.k8s.io/id: 6555d8aa-8ec1-11e9-a433-9ee7625cd1db
    prow.k8s.io/job: pull-test-infra-bazel
    prow.k8s.io/refs.org: kubernetes
    prow.k8s.io/refs.pull: "13038"
    prow.k8s.io/refs.repo: test-infra
    prow.k8s.io/type: presubmit
  name: 6555d8aa-8ec1-11e9-a433-9ee7625cd1db
  namespace: test-pods
  resourceVersion: "1169775978"
  selfLink: /api/v1/namespaces/test-pods/pods/6555d8aa-8ec1-11e9-a433-9ee7625cd1db
  uid: 6ea12de0-8ec1-11e9-a2d6-42010a8000bd
spec:
  automountServiceAccountToken: false
  containers:
  - command:
    - /tools/entrypoint
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /etc/service-account/service-account.json
    - name: E2E_GOOGLE_APPLICATION_CREDENTIALS
      value: /etc/service-account/service-account.json
    - name: TEST_TMPDIR
      value: /bazel-scratch/.cache/bazel
    - name: ARTIFACTS
      value: /logs/artifacts
    - name: BUILD_ID
      value: "1139570293107331073"
    - name: BUILD_NUMBER
      value: "1139570293107331073"
    - name: GOPATH
      value: /home/prow/go
    - name: JOB_NAME
      value: pull-test-infra-bazel
    - name: JOB_SPEC
      value: '{"type":"presubmit","job":"pull-test-infra-bazel","buildid":"1139570293107331073","prowjobid":"6555d8aa-8ec1-11e9-a433-9ee7625cd1db","refs":{"org":"kubernetes","repo":"test-infra","repo_link":"https://github.com/kubernetes/test-infra","base_ref":"master","base_sha":"f1b872102fc8841673644eca2f32c30b16b4f6ca","base_link":"https://github.com/kubernetes/test-infra/commit/f1b872102fc8841673644eca2f32c30b16b4f6ca","pulls":[{"number":13038,"author":"stevekuznetsov","sha":"410c774b4a2fbf4e6f1b150e307fc8f9a12082a5","link":"https://github.com/kubernetes/test-infra/pull/13038","commit_link":"https://github.com/kubernetes/test-infra/pull/13038/commits/410c774b4a2fbf4e6f1b150e307fc8f9a12082a5","author_link":"https://github.com/stevekuznetsov"}]}}'
    - name: JOB_TYPE
      value: presubmit
    - name: PROW_JOB_ID
      value: 6555d8aa-8ec1-11e9-a433-9ee7625cd1db
    - name: PULL_BASE_REF
      value: master
    - name: PULL_BASE_SHA
      value: f1b872102fc8841673644eca2f32c30b16b4f6ca
    - name: PULL_NUMBER
      value: "13038"
    - name: PULL_PULL_SHA
      value: 410c774b4a2fbf4e6f1b150e307fc8f9a12082a5
    - name: PULL_REFS
      value: master:f1b872102fc8841673644eca2f32c30b16b4f6ca,13038:410c774b4a2fbf4e6f1b150e307fc8f9a12082a5
    - name: REPO_NAME
      value: test-infra
    - name: REPO_OWNER
      value: kubernetes
    - name: ENTRYPOINT_OPTIONS
      value: '{"timeout":7200000000000,"grace_period":15000000000,"artifact_dir":"/logs/artifacts","args":["hack/bazel.sh","test","--config=ci","--nobuild_tests_only","//..."],"process_log":"/logs/process-log.txt","marker_file":"/logs/marker-file.txt","metadata_file":"/logs/artifacts/metadata.json"}'
    image: launcher.gcr.io/google/bazel:0.26.0
    imagePullPolicy: IfNotPresent
    name: test
    resources:
      requests:
        memory: 1Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/service-account
      name: service
      readOnly: true
    - mountPath: /bazel-scratch/.cache
      name: bazel-scratch
    - mountPath: /logs
      name: logs
    - mountPath: /tools
      name: tools
    - mountPath: /home/prow/go
      name: code
    workingDir: /home/prow/go/src/github.com/kubernetes/test-infra
  - command:
    - /sidecar
    env:
    - name: JOB_SPEC
      value: '{"type":"presubmit","job":"pull-test-infra-bazel","buildid":"1139570293107331073","prowjobid":"6555d8aa-8ec1-11e9-a433-9ee7625cd1db","refs":{"org":"kubernetes","repo":"test-infra","repo_link":"https://github.com/kubernetes/test-infra","base_ref":"master","base_sha":"f1b872102fc8841673644eca2f32c30b16b4f6ca","base_link":"https://github.com/kubernetes/test-infra/commit/f1b872102fc8841673644eca2f32c30b16b4f6ca","pulls":[{"number":13038,"author":"stevekuznetsov","sha":"410c774b4a2fbf4e6f1b150e307fc8f9a12082a5","link":"https://github.com/kubernetes/test-infra/pull/13038","commit_link":"https://github.com/kubernetes/test-infra/pull/13038/commits/410c774b4a2fbf4e6f1b150e307fc8f9a12082a5","author_link":"https://github.com/stevekuznetsov"}]}}'
    - name: SIDECAR_OPTIONS
      value: '{"gcs_options":{"items":["/logs/artifacts"],"bucket":"kubernetes-jenkins","path_strategy":"legacy","default_org":"kubernetes","default_repo":"kubernetes","gcs_credentials_file":"/secrets/gcs/service-account.json","dry_run":false},"entries":[{"args":["hack/bazel.sh","test","--config=ci","--nobuild_tests_only","//..."],"process_log":"/logs/process-log.txt","marker_file":"/logs/marker-file.txt","metadata_file":"/logs/artifacts/metadata.json"}]}'
    image: gcr.io/k8s-prow/sidecar:v20190610-3be53b072
    imagePullPolicy: IfNotPresent
    name: sidecar
    resources:
      requests:
        memory: 1Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /logs
      name: logs
    - mountPath: /secrets/gcs
      name: gcs-credentials
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  initContainers:
  - command:
    - /clonerefs
    env:
    - name: CLONEREFS_OPTIONS
      value: '{"src_root":"/home/prow/go","log":"/logs/clone.json","git_user_name":"ci-robot","git_user_email":"ci-robot@k8s.io","refs":[{"org":"kubernetes","repo":"test-infra","repo_link":"https://github.com/kubernetes/test-infra","base_ref":"master","base_sha":"f1b872102fc8841673644eca2f32c30b16b4f6ca","base_link":"https://github.com/kubernetes/test-infra/commit/f1b872102fc8841673644eca2f32c30b16b4f6ca","pulls":[{"number":13038,"author":"stevekuznetsov","sha":"410c774b4a2fbf4e6f1b150e307fc8f9a12082a5","link":"https://github.com/kubernetes/test-infra/pull/13038","commit_link":"https://github.com/kubernetes/test-infra/pull/13038/commits/410c774b4a2fbf4e6f1b150e307fc8f9a12082a5","author_link":"https://github.com/stevekuznetsov"}]}]}'
    image: gcr.io/k8s-prow/clonerefs:v20190610-3be53b072
    imagePullPolicy: IfNotPresent
    name: clonerefs
    resources:
      requests:
        memory: 1Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /logs
      name: logs
    - mountPath: /home/prow/go
      name: code
  - command:
    - /initupload
    env:
    - name: INITUPLOAD_OPTIONS
      value: '{"bucket":"kubernetes-jenkins","path_strategy":"legacy","default_org":"kubernetes","default_repo":"kubernetes","gcs_credentials_file":"/secrets/gcs/service-account.json","dry_run":false,"log":"/logs/clone.json"}'
    - name: JOB_SPEC
      value: '{"type":"presubmit","job":"pull-test-infra-bazel","buildid":"1139570293107331073","prowjobid":"6555d8aa-8ec1-11e9-a433-9ee7625cd1db","refs":{"org":"kubernetes","repo":"test-infra","repo_link":"https://github.com/kubernetes/test-infra","base_ref":"master","base_sha":"f1b872102fc8841673644eca2f32c30b16b4f6ca","base_link":"https://github.com/kubernetes/test-infra/commit/f1b872102fc8841673644eca2f32c30b16b4f6ca","pulls":[{"number":13038,"author":"stevekuznetsov","sha":"410c774b4a2fbf4e6f1b150e307fc8f9a12082a5","link":"https://github.com/kubernetes/test-infra/pull/13038","commit_link":"https://github.com/kubernetes/test-infra/pull/13038/commits/410c774b4a2fbf4e6f1b150e307fc8f9a12082a5","author_link":"https://github.com/stevekuznetsov"}]}}'
    image: gcr.io/k8s-prow/initupload:v20190610-3be53b072
    imagePullPolicy: IfNotPresent
    name: initupload
    resources:
      requests:
        memory: 1Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /logs
      name: logs
    - mountPath: /secrets/gcs
      name: gcs-credentials
  - args:
    - /entrypoint
    - /tools/entrypoint
    command:
    - /bin/cp
    image: gcr.io/k8s-prow/entrypoint:v20190610-3be53b072
    imagePullPolicy: IfNotPresent
    name: place-entrypoint
    resources:
      requests:
        memory: 1Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /tools
      name: tools
  nodeName: gke-prow-containerd-pool-bigger-170c3937-fzx9
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: service
    secret:
      defaultMode: 420
      secretName: service-account
  - emptyDir: {}
    name: bazel-scratch
  - emptyDir: {}
    name: logs
  - emptyDir: {}
    name: tools
  - name: gcs-credentials
    secret:
      defaultMode: 420
      secretName: service-account
  - emptyDir: {}
    name: code
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2019-06-14T16:28:24Z
    message: 'containers with incomplete status: [clonerefs initupload place-entrypoint]'
    reason: ContainersNotInitialized
    status: "False"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2019-06-14T16:28:24Z
    message: 'containers with unready status: [test sidecar]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: 2019-06-14T16:28:24Z
    message: 'containers with unready status: [test sidecar]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: 2019-06-14T16:28:24Z
    status: "True"
    type: PodScheduled
  containerStatuses:
  - image: gcr.io/k8s-prow/sidecar:v20190610-3be53b072
    imageID: ""
    lastState: {}
    name: sidecar
    ready: false
    restartCount: 0
    state:
      waiting:
        reason: PodInitializing
  - image: launcher.gcr.io/google/bazel:0.26.0
    imageID: ""
    lastState: {}
    name: test
    ready: false
    restartCount: 0
    state:
      waiting:
        reason: PodInitializing
  hostIP: 10.128.0.121
  initContainerStatuses:
  - image: gcr.io/k8s-prow/clonerefs:v20190610-3be53b072
    imageID: ""
    lastState: {}
    name: clonerefs
    ready: false
    restartCount: 0
    state:
      waiting:
        reason: PodInitializing
  - image: gcr.io/k8s-prow/initupload:v20190610-3be53b072
    imageID: ""
    lastState: {}
    name: initupload
    ready: false
    restartCount: 0
    state:
      waiting:
        reason: PodInitializing
  - image: gcr.io/k8s-prow/entrypoint:v20190610-3be53b072
    imageID: ""
    lastState: {}
    name: place-entrypoint
    ready: false
    restartCount: 0
    state:
      waiting:
        reason: PodInitializing
  phase: Pending
  qosClass: Burstable
  startTime: 2019-06-14T16:28:24Z
kubectl describe pod 
Name:               6555d8aa-8ec1-11e9-a433-9ee7625cd1db
Namespace:          test-pods
Priority:           0
PriorityClassName:  <none>
Node:               gke-prow-containerd-pool-bigger-170c3937-fzx9/10.128.0.121
Start Time:         Fri, 14 Jun 2019 09:28:24 -0700
Labels:             created-by-prow=true
                    event-GUID=64edf720-8ec1-11e9-917a-ebca143d37d5
                    preset-bazel-scratch-dir=true
                    preset-service-account=true
                    prow.k8s.io/id=6555d8aa-8ec1-11e9-a433-9ee7625cd1db
                    prow.k8s.io/job=pull-test-infra-bazel
                    prow.k8s.io/refs.org=kubernetes
                    prow.k8s.io/refs.pull=13038
                    prow.k8s.io/refs.repo=test-infra
                    prow.k8s.io/type=presubmit
Annotations:        kubernetes.io/limit-ranger:
                      LimitRanger plugin set: memory request for container test; memory request for container sidecar; memory request for init container clonere...
                    prow.k8s.io/job: pull-test-infra-bazel
Status:             Pending
IP:                 
Init Containers:
  clonerefs:
    Container ID:  
    Image:         gcr.io/k8s-prow/clonerefs:v20190610-3be53b072
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /clonerefs
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      memory:  1Gi
    Environment:
      CLONEREFS_OPTIONS:  {"src_root":"/home/prow/go","log":"/logs/clone.json","git_user_name":"ci-robot","git_user_email":"ci-robot@k8s.io","refs":[{"org":"kubernetes","repo":"test-infra","repo_link":"https://github.com/kubernetes/test-infra","base_ref":"master","base_sha":"f1b872102fc8841673644eca2f32c30b16b4f6ca","base_link":"https://github.com/kubernetes/test-infra/commit/f1b872102fc8841673644eca2f32c30b16b4f6ca","pulls":[{"number":13038,"author":"stevekuznetsov","sha":"410c774b4a2fbf4e6f1b150e307fc8f9a12082a5","link":"https://github.com/kubernetes/test-infra/pull/13038","commit_link":"https://github.com/kubernetes/test-infra/pull/13038/commits/410c774b4a2fbf4e6f1b150e307fc8f9a12082a5","author_link":"https://github.com/stevekuznetsov"}]}]}
    Mounts:
      /home/prow/go from code (rw)
      /logs from logs (rw)
  initupload:
    Container ID:  
    Image:         gcr.io/k8s-prow/initupload:v20190610-3be53b072
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /initupload
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      memory:  1Gi
    Environment:
      INITUPLOAD_OPTIONS:  {"bucket":"kubernetes-jenkins","path_strategy":"legacy","default_org":"kubernetes","default_repo":"kubernetes","gcs_credentials_file":"/secrets/gcs/service-account.json","dry_run":false,"log":"/logs/clone.json"}
      JOB_SPEC:            {"type":"presubmit","job":"pull-test-infra-bazel","buildid":"1139570293107331073","prowjobid":"6555d8aa-8ec1-11e9-a433-9ee7625cd1db","refs":{"org":"kubernetes","repo":"test-infra","repo_link":"https://github.com/kubernetes/test-infra","base_ref":"master","base_sha":"f1b872102fc8841673644eca2f32c30b16b4f6ca","base_link":"https://github.com/kubernetes/test-infra/commit/f1b872102fc8841673644eca2f32c30b16b4f6ca","pulls":[{"number":13038,"author":"stevekuznetsov","sha":"410c774b4a2fbf4e6f1b150e307fc8f9a12082a5","link":"https://github.com/kubernetes/test-infra/pull/13038","commit_link":"https://github.com/kubernetes/test-infra/pull/13038/commits/410c774b4a2fbf4e6f1b150e307fc8f9a12082a5","author_link":"https://github.com/stevekuznetsov"}]}}
    Mounts:
      /logs from logs (rw)
      /secrets/gcs from gcs-credentials (rw)
  place-entrypoint:
    Container ID:  
    Image:         gcr.io/k8s-prow/entrypoint:v20190610-3be53b072
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/cp
    Args:
      /entrypoint
      /tools/entrypoint
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      memory:     1Gi
    Environment:  <none>
    Mounts:
      /tools from tools (rw)
Containers:
  test:
    Container ID:  
    Image:         launcher.gcr.io/google/bazel:0.26.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /tools/entrypoint
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      memory:  1Gi
    Environment:
      GOOGLE_APPLICATION_CREDENTIALS:      /etc/service-account/service-account.json
      E2E_GOOGLE_APPLICATION_CREDENTIALS:  /etc/service-account/service-account.json
      TEST_TMPDIR:                         /bazel-scratch/.cache/bazel
      ARTIFACTS:                           /logs/artifacts
      BUILD_ID:                            1139570293107331073
      BUILD_NUMBER:                        1139570293107331073
      GOPATH:                              /home/prow/go
      JOB_NAME:                            pull-test-infra-bazel
      JOB_SPEC:                            {"type":"presubmit","job":"pull-test-infra-bazel","buildid":"1139570293107331073","prowjobid":"6555d8aa-8ec1-11e9-a433-9ee7625cd1db","refs":{"org":"kubernetes","repo":"test-infra","repo_link":"https://github.com/kubernetes/test-infra","base_ref":"master","base_sha":"f1b872102fc8841673644eca2f32c30b16b4f6ca","base_link":"https://github.com/kubernetes/test-infra/commit/f1b872102fc8841673644eca2f32c30b16b4f6ca","pulls":[{"number":13038,"author":"stevekuznetsov","sha":"410c774b4a2fbf4e6f1b150e307fc8f9a12082a5","link":"https://github.com/kubernetes/test-infra/pull/13038","commit_link":"https://github.com/kubernetes/test-infra/pull/13038/commits/410c774b4a2fbf4e6f1b150e307fc8f9a12082a5","author_link":"https://github.com/stevekuznetsov"}]}}
      JOB_TYPE:                            presubmit
      PROW_JOB_ID:                         6555d8aa-8ec1-11e9-a433-9ee7625cd1db
      PULL_BASE_REF:                       master
      PULL_BASE_SHA:                       f1b872102fc8841673644eca2f32c30b16b4f6ca
      PULL_NUMBER:                         13038
      PULL_PULL_SHA:                       410c774b4a2fbf4e6f1b150e307fc8f9a12082a5
      PULL_REFS:                           master:f1b872102fc8841673644eca2f32c30b16b4f6ca,13038:410c774b4a2fbf4e6f1b150e307fc8f9a12082a5
      REPO_NAME:                           test-infra
      REPO_OWNER:                          kubernetes
      ENTRYPOINT_OPTIONS:                  {"timeout":7200000000000,"grace_period":15000000000,"artifact_dir":"/logs/artifacts","args":["hack/bazel.sh","test","--config=ci","--nobuild_tests_only","//..."],"process_log":"/logs/process-log.txt","marker_file":"/logs/marker-file.txt","metadata_file":"/logs/artifacts/metadata.json"}
    Mounts:
      /bazel-scratch/.cache from bazel-scratch (rw)
      /etc/service-account from service (ro)
      /home/prow/go from code (rw)
      /logs from logs (rw)
      /tools from tools (rw)
  sidecar:
    Container ID:  
    Image:         gcr.io/k8s-prow/sidecar:v20190610-3be53b072
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /sidecar
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      memory:  1Gi
    Environment:
      JOB_SPEC:         {"type":"presubmit","job":"pull-test-infra-bazel","buildid":"1139570293107331073","prowjobid":"6555d8aa-8ec1-11e9-a433-9ee7625cd1db","refs":{"org":"kubernetes","repo":"test-infra","repo_link":"https://github.com/kubernetes/test-infra","base_ref":"master","base_sha":"f1b872102fc8841673644eca2f32c30b16b4f6ca","base_link":"https://github.com/kubernetes/test-infra/commit/f1b872102fc8841673644eca2f32c30b16b4f6ca","pulls":[{"number":13038,"author":"stevekuznetsov","sha":"410c774b4a2fbf4e6f1b150e307fc8f9a12082a5","link":"https://github.com/kubernetes/test-infra/pull/13038","commit_link":"https://github.com/kubernetes/test-infra/pull/13038/commits/410c774b4a2fbf4e6f1b150e307fc8f9a12082a5","author_link":"https://github.com/stevekuznetsov"}]}}
      SIDECAR_OPTIONS:  {"gcs_options":{"items":["/logs/artifacts"],"bucket":"kubernetes-jenkins","path_strategy":"legacy","default_org":"kubernetes","default_repo":"kubernetes","gcs_credentials_file":"/secrets/gcs/service-account.json","dry_run":false},"entries":[{"args":["hack/bazel.sh","test","--config=ci","--nobuild_tests_only","//..."],"process_log":"/logs/process-log.txt","marker_file":"/logs/marker-file.txt","metadata_file":"/logs/artifacts/metadata.json"}]}
    Mounts:
      /logs from logs (rw)
      /secrets/gcs from gcs-credentials (rw)
Conditions:
  Type              Status
  Initialized       False 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  service:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  service-account
    Optional:    false
  bazel-scratch:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  logs:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  tools:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  gcs-credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  service-account
    Optional:    false
  code:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age                   From                                                    Message
  ----     ------                  ----                  ----                                                    -------
  Normal   Scheduled               39m                   default-scheduler                                       Successfully assigned test-pods/6555d8aa-8ec1-11e9-a433-9ee7625cd1db to gke-prow-containerd-pool-bigger-170c3937-fzx9
  Warning  FailedCreatePodSandBox  39m                   kubelet, gke-prow-containerd-pool-bigger-170c3937-fzx9  Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b44007428cd156be1435328e60724d53bda421d19861469d1619c84963e2310c": failed to allocate for range 0: no IP addresses available in range set: 10.60.28.1-10.60.28.254
  Warning  FailedCreatePodSandBox  39m                   kubelet, gke-prow-containerd-pool-bigger-170c3937-fzx9  Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3492fd3b74f05259384f65c919d09220cb91cc1a7158dfefd501e5ce87a54312": failed to allocate for range 0: no IP addresses available in range set: 10.60.28.1-10.60.28.254
  Warning  FailedCreatePodSandBox  38m                   kubelet, gke-prow-containerd-pool-bigger-170c3937-fzx9  Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "da1db319f6c19f0fced3404ca9eba9231868d151d0a27216ee119dc5a212353c": failed to allocate for range 0: no IP addresses available in range set: 10.60.28.1-10.60.28.254
  Warning  FailedCreatePodSandBox  38m                   kubelet, gke-prow-containerd-pool-bigger-170c3937-fzx9  Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b79a0319862c99e133bc677597e17d6ccaadb06574dc05d7df65c60f602dfd13": failed to allocate for range 0: no IP addresses available in range set: 10.60.28.1-10.60.28.254
  Warning  FailedCreatePodSandBox  38m                   kubelet, gke-prow-containerd-pool-bigger-170c3937-fzx9  Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2a841c485d2cefe61623c15451365fb09b4bc99ab9610e582592ad0087b957e8": failed to allocate for range 0: no IP addresses available in range set: 10.60.28.1-10.60.28.254
  Warning  FailedCreatePodSandBox  38m                   kubelet, gke-prow-containerd-pool-bigger-170c3937-fzx9  Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d90b25d56dc0c66db6bff7c51e5342f1ec8b1466e379b223248fc62309c2ba1f": failed to allocate for range 0: no IP addresses available in range set: 10.60.28.1-10.60.28.254
  Warning  FailedCreatePodSandBox  37m                   kubelet, gke-prow-containerd-pool-bigger-170c3937-fzx9  Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "27e31071c1dd28633a21cf256aa2917b961694500bf9562e24cf4eef3e258ce4": fork/exec /home/containerd/opt/cni/bin/host-local: resource temporarily unavailable
  Warning  FailedCreatePodSandBox  37m                   kubelet, gke-prow-containerd-pool-bigger-170c3937-fzx9  Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3248bb997030fdc2db7598c0b5a0ec580f610d79aa49bee6e0371abfd10522f3": failed to allocate for range 0: no IP addresses available in range set: 10.60.28.1-10.60.28.254
  Warning  FailedCreatePodSandBox  37m                   kubelet, gke-prow-containerd-pool-bigger-170c3937-fzx9  Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "107b9a35a2d07fb2fd1b458ab80a63766a2cbaa7613aef94311823b0cab0f5e7": failed to allocate for range 0: no IP addresses available in range set: 10.60.28.1-10.60.28.254
  Warning  FailedCreatePodSandBox  4m9s (x153 over 37m)  kubelet, gke-prow-containerd-pool-bigger-170c3937-fzx9  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6769374aadfa0a00be6c6e7b2be9b6da812dcc496c1c2ca0ab4b8ec1c611fe15": failed to allocate for range 0: no IP addresses available in range set: 10.60.28.1-10.60.28.254

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 18 (15 by maintainers)

Most upvoted comments

Getting more of these, but without a spike in Disk IOPS or throughput. 😕 (Thanks @cjwagner!)

https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/test-infra/13078/pull-test-infra-verify-file-perms/1141118142936780802

kubectl describe pod -n test-pods 90713123-921c-11e9-ac17-16331ef92eac:

713123-921c-11e9-ac17-16331ef92eac to gke-prow-containerd-pool-bigger-170c3937-glwj
  Warning  FailedCreatePodSandBox  43m                    kubelet, gke-prow-containerd-pool-bigger-170c3937-glwj  Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0ae58c9651cfc164d0e48d2c42faef3f16a5962fe2919acf76adfd4a7ef42ea7": netplugin failed but error parsing its diagnostic message "": unexpected end of JSON input
  Warning  FailedCreatePodSandBox  43m                    kubelet, gke-prow-containerd-pool-bigger-170c3937-glwj  Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7a55ca6430218e323309e2ec9edc0c1cecce29b96d711913bae9f63a601a79a5": fork/exec /home/containerd/opt/cni/bin/loopback: resource temporarily unavailable
  Warning  FailedCreatePodSandBox  42m                    kubelet, gke-prow-containerd-pool-bigger-170c3937-glwj  Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5043a15e8cb3ded675b91f1656727b0c6abab2952d72acf722d5f5d5d7f7c675": netplugin failed but error parsing its diagnostic message "": unexpected end of JSON input
  Warning  FailedCreatePodSandBox  42m                    kubelet, gke-prow-containerd-pool-bigger-170c3937-glwj  Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c22dc99a76efee6f9a11c04902ba8a135a4254d74cc2903a00a4fb4be58f5b98": netplugin failed but error parsing its diagnostic message "": unexpected end of JSON input
  Warning  FailedCreatePodSandBox  42m                    kubelet, gke-prow-containerd-pool-bigger-170c3937-glwj  Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "982f7cf5ac0d62feefc226d43e830c56ad79cb662b41f95b5eaf83e654a39d65": netplugin failed but error parsing its diagnostic message "": unexpected end of JSON input
  Warning  FailedCreatePodSandBox  42m                    kubelet, gke-prow-containerd-pool-bigger-170c3937-glwj  Failed create pod sandbox: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial unix /run/containerd/containerd.sock: connect: connection refused"
  Warning  FailedCreatePodSandBox  42m                    kubelet, gke-prow-containerd-pool-bigger-170c3937-glwj  Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e4af98b15761929487423e8a3b9c780c11dc9147bcc2e03ccdabdc4037ec68e5": netplugin failed but error parsing its diagnostic message "": unexpected end of JSON input
  Warning  FailedCreatePodSandBox  41m                    kubelet, gke-prow-containerd-pool-bigger-170c3937-glwj  Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d6e24979203779a322e7d6240dbfa92c95694bbdfeed7966330d1a4bb77b96dd": netplugin failed but error parsing its diagnostic message "": unexpected end of JSON input
  Warning  FailedCreatePodSandBox  41m                    kubelet, gke-prow-containerd-pool-bigger-170c3937-glwj  Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f5527a6c05e6d185f0f5e56b5804fb7e4a6eacf5a7c51e316197c949076cca07": fork/exec /home/containerd/opt/cni/bin/ptp: resource temporarily unavailable
  Warning  FailedCreatePodSandBox  3m12s (x177 over 41m)  kubelet, gke-prow-containerd-pool-bigger-170c3937-glwj  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "db42c5b264a2f65fa92a2c86b6ae6fc14e045010fb91642abf72d77720727d55": fork/exec /home/containerd/opt/cni/bin/loopback: resource temporarily unavailable

This should have had impact since around 4 PM Pacific Time, but the graphs appear flat.

image

Taking a further look.

+1
michelle192837 on Jun 18, 2019

fork/exec /home/containerd/opt/cni/bin/host-local: resource temporarily unavailable

That sounds an awful lot like we’re hitting I/O issues again, should check the I/O vs the new quotas.

+1
BenTheElder on Jun 14, 2019
Read more comments on GitHub
← test-infra: ProwJobs should have unique names (don't allow branch sharded variants to share a name)
test-infra: Publishing bot cannot push →