serving: The Knative Serving controller can not access the same set of private registries as the K8s cluster

Expected Behavior

The Knative Serving controller is able to access the same set of private registries as the K8s cluster.

Actual Behavior

The Knative Serving Controller fails to authenticate itself to the same private registries as the K8s cluster.

Steps to Reproduce the Problem

An example would be a Kops cluster that is able to access a private AWS ECR registry, while the Knative Serving controller can not.

A high-level set of steps to reproduce the issue is provided below:

1. Create a Kops cluster that is able to access a private ECR registry
2. Observe that creating a Pod directly in the cluster that references an image in the private ECR registry successfully pulls the image
3. Observe that the Knative Serving controller will fail to access the image at this point with an error unsupported status code 401; body: Not Authorized

The difference in behavior is due to Kops setting the --cloud-provider flag and that this is not enabled by the Knative Serving controller when executing: https://github.com/kubernetes/kubernetes/blob/36877dafe40495cb43994464e2427355f99042c7/pkg/credentialprovider/aws/aws_credentials.go#L158

Additional Info

Possible solutions:

1. The Knative Serving controller could create a pod in the target namespace and then query (e.g. .status.containerStatuses[0].imageID) the pod status to extract the digest. This would effectively delegate the image pull to the K8s cluster. An example pod spec is provided below:

kind: Pod
metadata:
  name: knative-resolver-{target_image_address}
spec:
    containers:
     - name: knative-resolver-{target_image_address}
       imagePullPolicy: Always
       args: []
       command: []
       image: {target_image_address}

1. The Knative Serving controller image could provide a mechanism to supply Docker credentials helpers at a known path. This would make it possible to use, for example, the ECR credential helper.

Workarounds

1. Specify the digest in the image address which will prevent the controller from attempting to resolve to a digest: https://github.com/knative/serving/blob/b707183d8355544c3cd8e841923eb08ccb91ceba/pkg/reconciler/v1alpha1/revision/resolve.go#L52

2. Configure the registry in registriesToSkip for the controller

The above stated workarounds both lose the tag-to-digest resolution functionality.