kubernetes: [Failing Test] [sig-apps] ReplicaSet should serve a basic image on each replica with a private image, ReplicationController should serve a basic image on each replica with a private image

Which jobs are failing: ci-kubernetes-e2e-gci-gce ci-kubernetes-e2e-gce-cos-k8sbeta-default

Which test(s) are failing: [sig-apps] ReplicaSet should serve a basic image on each replica with a private image [sig-apps] ReplicationController should serve a basic image on each replica with a private image

Since when has it been failing: Started failing between 2:04 and 2:40PM PST Dec 1

Testgrid link: https://k8s-testgrid.appspot.com/sig-release-master-blocking#gce-cos-master-default https://k8s-testgrid.appspot.com/sig-release-1.20-blocking#gce-cos-k8sbeta-default

Reason for failure: pod never run? Looks like both are timing out waiting for containers to be ready

ReplicaSet should serve a basic image on each replica with a private image:

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/replica_set.go:98
Dec  1 22:54:13.321: Unexpected error:
    <*errors.errorString | 0xc0036f8ef0>: {
        s: "pod \"my-hostname-private-cd2ec0df-be38-465e-a00f-f868f9674320-rknrl\" never run (phase: Pending, conditions: [{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2020-12-01 22:49:07 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2020-12-01 22:49:07 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [my-hostname-private-cd2ec0df-be38-465e-a00f-f868f9674320]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2020-12-01 22:49:07 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [my-hostname-private-cd2ec0df-be38-465e-a00f-f868f9674320]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2020-12-01 22:49:07 +0000 UTC Reason: Message:}]): timed out waiting for the condition",
    }
    pod "my-hostname-private-cd2ec0df-be38-465e-a00f-f868f9674320-rknrl" never run (phase: Pending, conditions: [{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2020-12-01 22:49:07 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2020-12-01 22:49:07 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [my-hostname-private-cd2ec0df-be38-465e-a00f-f868f9674320]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2020-12-01 22:49:07 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [my-hostname-private-cd2ec0df-be38-465e-a00f-f868f9674320]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2020-12-01 22:49:07 +0000 UTC Reason: Message:}]): timed out waiting for the condition
occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/replica_set.go:156

ReplicationController should serve a basic image on each replica with a private image

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/rc.go:68
Dec  1 23:07:02.794: Unexpected error:
    <*errors.errorString | 0xc00348b1f0>: {
        s: "pod \"my-hostname-private-3071f600-7524-41d9-b7ea-f7a5cf5011e7-xz94v\" never run (phase: Pending, conditions: [{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2020-12-01 23:02:02 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2020-12-01 23:02:02 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [my-hostname-private-3071f600-7524-41d9-b7ea-f7a5cf5011e7]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2020-12-01 23:02:02 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [my-hostname-private-3071f600-7524-41d9-b7ea-f7a5cf5011e7]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2020-12-01 23:02:02 +0000 UTC Reason: Message:}]): timed out waiting for the condition",
    }
    pod "my-hostname-private-3071f600-7524-41d9-b7ea-f7a5cf5011e7-xz94v" never run (phase: Pending, conditions: [{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2020-12-01 23:02:02 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2020-12-01 23:02:02 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [my-hostname-private-3071f600-7524-41d9-b7ea-f7a5cf5011e7]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2020-12-01 23:02:02 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [my-hostname-private-3071f600-7524-41d9-b7ea-f7a5cf5011e7]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2020-12-01 23:02:02 +0000 UTC Reason: Message:}]): timed out waiting for the condition
occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/rc.go:459

Anything else we need to know:

Example Spyglass links:

• https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/1333903762309255168
• https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sbeta-default/1333958121797718016

Having trouble finding a good Triage link - will drop one if I can find

Wondering whether this has anything to do with the Pod pending timeout errors happening on some of the jobs on the 1.20 boards now?

/sig apps /cc @kubernetes/ci-signal @kubernetes/sig-apps-test-failures