kaniko: Regression: Kaniko 1.7 unstable authentication against GCP Artifact Registry

Actual behavior While building several containers against GCP Artifact Registry via skaffold we are getting intermittent authentication failures.

INFO[0000] Retrieving image gcr.io/kaniko-project/executor:v1.5.1@sha256:c6166717f7fe0b7da44908c986137ecfeab21f31ec3992f6e128fff8a94be8a5 from registry gcr.io 
E0124 14:27:12.856809       1 metadata.go:166] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
INFO[0000] Built cross stage deps: map[]                
INFO[0000] Retrieving image manifest gcr.io/kaniko-project/executor:v1.5.1@sha256:c6166717f7fe0b7da44908c986137ecfeab21f31ec3992f6e128fff8a94be8a5 
INFO[0000] Returning cached image manifest              
INFO[0000] Executing 0 build triggers                   
INFO[0000] Skipping unpacking as no commands require it. 
INFO[0000] Taking snapshot of full filesystem...        
INFO[0000] Pushing image to us-east4-docker.pkg.dev/******/platform/containers/tools/kaniko:abaee2d 
INFO[0001] Pushed image to 1 destinations               
Building [bases/alpine]...
E0124 14:27:20.443958       1 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "us-east4-docker.pkg.dev/******/platform/containers/bases/alpine:abaee2d": creating push check transport for us-east4-docker.pkg.dev failed: GET https://us-east4-docker.pkg.dev/v2/token?scope=repository%3A******%2Fplatform%2Fcontainers%2Fbases%2Falpine%3Apush%2Cpull&service=us-east4-docker.pkg.dev: UNAUTHORIZED: authentication failed

Prior to invoking skaffold we issue:

docker-credential-gcr configure-docker --registries=us-east4-docker.pkg.dev

Expected behavior We expect pushes continue to work throughout the whole build.

Additional Information

  • Google Internal Case 29377744
  • Kaniko Image (fully qualified with digest) gcr.io/kaniko-project/executor:v1.7.0-debug@sha256:88dacc7ea3f5c04709eae96776693c717869405364b19d6e78850fe54c63c6a2

About this issue

Most upvoted comments

@imjasonh

GOOGLE_APPLICATION_CREDENTIALS ENV set with token.json file path no ~/.docker/config.json at all

Version Working?
v1.6.0 ✔️
v1.7.0
v1.8.0 ✔️

GOOGLE_APPLICATION_CREDENTIALS ENV set with token.json file path ~/.docker/config.json loaded with unused third-party external credentials (non-gcr, non-credHelper)

Version Working?
v1.6.0
v1.7.0
v1.8.0 ✔️

GOOGLE_APPLICATION_CREDENTIALS ENV set with token.json file path ~/.docker/config.json loaded with gcr credHelpers for target registry

Version Working?
v1.6.0 ✔️
v1.7.0
v1.8.0

So far, it looks good by removing config.json file. It even works when using a file with unused credentials 👍