serving: Tag to digest resolving is not working in Knative 1.3.0
What version of Knative?
Knative
v1.3.0, AKSv1.21.2(tried on AKS v1.22.6 as well, same issue)
Expected Behavior
I should be able to create a knative service that pulls an image with a defined tag from a private registry, the exact same service definition file was working with knative 0.25.x
Actual Behavior
I get failed to resolve image to digest when deploying an image with a defined tag. I tried it with azurecr as well as registry.gitlab.com, same error. However, when I added the image digest explicitly it worked.
I’m aware that adding the registriesSkippingTagResolving is one workaround, but it won’t be useful for me since users might deploy images in my app from registries I’m not aware of.
Revision "mlserv-4b593f45-42b6-4803-ab88-9aa6d67b1abb-00001" failed with message: Unable to fetch image "xxxx.azurecr.io/my-image:v1.0.2": failed to resolve image to digest: Get "https://xxxx.azurecr.io/v2/": context deadline exceeded.
Steps to Reproduce the Problem
Using knative v1.3.0, attempt deploying a service with an image from a private registry with a defined tag.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 2
- Comments: 26 (14 by maintainers)
Ok new releases are out with the fixes
https://github.com/knative/serving/releases/tag/knative-v1.3.2 https://github.com/knative/serving/releases/tag/knative-v1.2.5
Please let me know if there are any issues
Thanks for the feedback - was finally able to get an azure account and confirmed that it’s just broken.
Here’s the breakdown so far:
K8s Deployment Image Pull
For comparison with a vanilla K8s deployment
Oddly - K8s works even with a partial match - ie.
registry.gitlab.com/dprotKnative Service
I’ve also tested knative serving 1.2.2 and it works well.
Reading the knative-serving Release I’ve noticed that with knative-serving 1.2.3 the github.com/google/go-containerregistry/pkg/authn/k8schain was refactored.
I think that’s what creates the issue.
Although the service account is created correctly, the knative controller cannot schedule the pods because it cannot contact the registry.
I have the same issue upgrading from knative serving 1.1.4 to 1.3.0.
I created the dockerconfigjson secret for gcr registry in my namespace and added as imagePullSecrets in the default service account.
On 1.1.4 It works fine. After the upgrade to 1.3.0 I got this error on knative serving controller: Unable to fetch image “gcr.io/xxxxxxxxx:xxxxxxxx”: failed to resolve image to digest: HEAD https://gcr.io/v2/xxxxx/xxxxxx/manifests/xxxxxx: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)
Reverting to 1.1.4 started working again. I tested also the 1.2.3 and I got the same issue.
I follow the knative serving setup by yaml
From the first tests I did, it seems that the patch has solved the problem. If there are any news, I will update you.
Confirmed the bug - I made a simple mistake of not bumping the sub package (
github.com/google/go-containerregistry/pkg/authn/k8schain)https://github.com/knative/serving/blob/f4ea3ac779621ea133a78a746525f6c6ca9947de/go.mod#L12-L13
I’ll bump the dependency and get patch releases out tomorrow.
Actually on 1.3.0 still needs this workaroud https://github.com/knative/serving/issues/12642 to download the image from the private gcr.