kubernetes: Missing gcr.io/authenticated-image-pulling/windows-nanoserver:v3 image for E2E tests

Which jobs are failing?

capz-windows-containerd-master

Which tests are failing?

[sig-node] Container Runtime blackbox test when running a container with a new image should be able to pull from private registry with secret [NodeConformance]

Since when has it been failing?

Since https://github.com/kubernetes/kubernetes/pull/113900 merged.

Testgrid link

https://testgrid.k8s.io/sig-windows-signal#capz-windows-containerd-master

Reason for failure (if possible)

A recent PR [1] updated the image versions we use for E2E tests. However, the windows-nanoserver image is meant to be in a private authenticated registry: gcr.io/authenticated-image-pulling/windows-nanoserver, which requires credentials to pull images from it. This image is required by the [sig-node] Container Runtime blackbox test when running a container with a new image should be able to pull from private registry with secret [NodeConformance] test for Windows. The v3 image does not exist, there’s no automatic promotion process for that registry. Previously, it was built and pushed manually.

Because of this, the https://testgrid.k8s.io/sig-windows-signal#capz-windows-containerd-master jobs have started to fail.

[1] https://github.com/kubernetes/kubernetes/pull/113900

Anything else we need to know?

The issue can be resolved if the image is built and pushed to the gcr.io/authenticated-image-pulling/ registry.

This can be done by running:

cd kubernetes/test/images
make REGISTRY=gcr.io/authenticated-image-pulling WHAT=windows-nanoserver all-push

Relevant SIG(s)

/sig windows /sig testing

About this issue

Original URL
State: open
Created 2 years ago
Comments: 26 (25 by maintainers)

Most upvoted comments

actually, for that matter, you could just create the service with a hardcoded clusterIP (from the reserved range at the bottom of the service CIDR) and use that for the registry IP…

I don’t think that works, because it has to be reachable by the container runtime on each node? We’re not accessing this from within a pod, the pod image is pulled from it by containerd/cri-o/…

ClusterIPs are reachable from the host network namespace (either a hostNetwork pod or a non-containerized process). That’s required by the e2e tests.

(Of course it wouldn’t be reachable before kube-proxy started, but that would be true for nodeports too so that’s no different.)

danwinship on May 5, 2023

I consider it a misfeature. I’m not sure “networking” does…

Sorry for conflating those, I got the sense network leads generally considered this a misfeature so it might not be the best plan to make the tests leverage it.

At any rate, in this issue we’re only concerned with the behavior of the winkernel proxier anyway, right? I don’t know if it supports localhost NodePorts.

The e2e test may indirectly cover winkernel proxier, but the test is actually “kubernetes can supply image pull credentials to CRI and pull an image that isn’t publicly readable without authentication”. So we need the node container runtime to be able to reach a registry endpoint that requires some known credentials and pull that image.

This node test has been using a GCR registry created in the google.com organization and a fixed exported service account key to access it.

What I’d like is some approach that allows us to avoid hosting a public endpoint for this, so SIG k8s infra doesn’t have to keep up an endpoint with magic permanent fixed creds, and we can stop depending on the endpoint that isn’t controlled by the Kubernetes project.

BenTheElder on May 4, 2023