kaniko: Pushing images to dockerhub stopped working

We are using Harbor as a self hosted container registry, but I cannot get the authentification to work in the kaniko build. The error is the same as with the official Docker registry:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "hub.domain.com/repo/kaniko:f569d627": POST https://hub.domain.com/v2/repo/kaniko/blobs/uploads/: UNAUTHORIZED: unauthorized to access repository: repo/kaniko, action: push: unauthorized to access repository: repo/kaniko, action: push

I also tried the suggested workaround resulting in config.json like this:

{
        "auths": {
                "hub.domain.com/v1": {
                        "auth": "xxxxxxxxxxxx"
                }
}

But the logs/errors stayed the same, i.e. Kaniko still used the https://hub.domain.com/v2/ endpoint. Any hints how I could adapt the workaround to work with Harbor as well?

This seems to work for docker hub :

$ export DOCKERHUB_AUTH="$(echo -n $DOCKER_HUB_REPOSITORY_USERNAME:$DOCKER_HUB_REPOSITORY_PASSWORD | base64)"
$ echo "{\"auths\":{\"https://index.docker.io/v1/\":{\"auth\":\"${DOCKERHUB_AUTH}\"}}}" > docker.json
$ docker run --rm -v $(pwd):/workspace -v $(pwd)/docker.json:/kaniko/.docker/config.json:ro gcr.io/kaniko-project/executor:v0.22.0 --context=dir:///workspace --dockerfile=Dockerfile --destination=foo/bar:latest

+1. I am able to upload docker images with https://index.docker.io/v1 but not https://index.docker.io/v2 with the latest kaniko debug executor image. Is anyone working on this issue?

i can verify that for us too the lastest working kaniko version is v0.16.0

v0.20.0 is not able to build the image, with the following job output:

$ mkdir -p /kaniko/.docker
$ echo "{\"auths\":{\"index.docker.io\":{\"auth\":\"${DOCKERHUB_AUTH}\"}}}" > /kaniko/.docker/config.json
$ mkdir /docker-tmp
$ echo 'FROM debian:stable' >> /docker-tmp/dockerfile
$ echo 'ENTRYPOINT ["/bin/bash", "-c", "echo hello"]' >> /docker-tmp/dockerfile
$ /kaniko/executor --context /docker-tmp --dockerfile /docker-tmp/dockerfile --destination foo/bar:hello-world-latest
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "foo/bar:hello-world-latest": POST https://index.docker.io/v2/foo/bar/blobs/uploads/: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:mgit/base Type:repository] map[Action:push Class: Name:mgit/base Type:repository]]Running after_script
Uploading artifacts for failed job
ERROR: Job failed: exit code 1
FATAL: exit code 1

thanks @ymage. I was using the v2 docker endpoint instead of v1.

I tried to use https://index.docker.io/v1/ instead of v2 one as docker registry url, which seemed to work for me. Is v2 endpoint really suported by kaniko ?

In the known issues section in readme, it is mentioned that Kanika does not support v1 api https://github.com/GoogleContainerTools/kaniko#known-issues. These are contradictory items in the document.

OK the doc is https://github.com/GoogleContainerTools/kaniko#pushing-to-docker-hub

Use a config.json file for docker as:

{
	"auths": {
		"https://index.docker.io/v1/": {
			"auth": "xxxxxxxxxxxxxxx"
		}
	}
}

After hours upon hours on this issue and keep getting the Unauthorized error, I did a docker login and found that the config looked as follows:

{
        "auths": {
                "https://index.docker.io/v1/": {
                        "auth": "REDACTED_T0="
                },
                "https://index.docker.io/v2/": {
                        "auth": "REDACTED_T0K"
                }
        }
}

When doing a base64 -d on both, it showed the same value. Thus the issue was that docker login uses a different base64 encode method for v1 and v2. (my password ended with an =) When using Kaniko I removed the v2 auth and it now works.

@nlamirault how did you changed the v1/v2 endpoint for reaching dockerhub ?

I don’t understand why it’s so complicated to have this basic feature working… (I’m just starting with Kaniko…)

works for me:

https://index.docker.io/v1/ + debug-v0.16.0

UP. It seems the real reason was UserAgent in config.json:

"HttpHeaders": {
  "User-Agent": "Docker-Client/19.03.12 (linux)"
}

After removing this section I haven’t have any problems with pushing (even with original debug-539ddefcae3fd6b411a95982a830d987f4214251)

This is also happening to us using the lastest version of “debug” (May 6, 2020) attempting to push to GCR. Seeing this error:

E0506 17:32:28.676141      16 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Changing to tag debug-v0.19.0 things start working again.

We are using Harbor as a self hosted container registry, but I cannot get the authentification to work in the kaniko build. The error is the same as with the official Docker registry:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "hub.domain.com/repo/kaniko:f569d627": POST https://hub.domain.com/v2/repo/kaniko/blobs/uploads/: UNAUTHORIZED: unauthorized to access repository: repo/kaniko, action: push: unauthorized to access repository: repo/kaniko, action: push

I also tried the suggested workaround resulting in config.json like this:

{
        "auths": {
                "hub.domain.com/v1": {
                        "auth": "xxxxxxxxxxxx"
                }
}

But the logs/errors stayed the same, i.e. Kaniko still used the https://hub.domain.com/v2/ endpoint. Any hints how I could adapt the workaround to work with Harbor as well?

I’m having the exact same problem, did you ever find the fix to this?

Ok sooo, since v1 is deprecated, I don’t believe using v1 is the safe option here or is it? I’m just confused. Isn’t v1 supported only by older docker clients? (and thus meaning it would stop working someday)

Which makes me think that, from debug-v0.16.0 to debug-v0.19.0 something change that made the registries think kaniko is an old docker client? and thus blocking kaniko from pushing/pulling on v2? I don’t know.

Does kaniko actually supports docker registry v2 API? I have private registry with v2 api exposed and cannot pull images from there by their tags (e.g. latest / stable / specific x.x.x version). Pull directly from docker command line works perfectly. Kaniko can only pull if I provide the exact image digest. Your docs at README.md says that v1 is not supported - so how to make this pull work with v2 API?

I try with Kaniko v1.0.0 on Docker Hub v2 endpoint, it fails. It works using v1.

After being struggled all the day with the issue, trying to push to dockerhub with a previous version of kaniko, debug-v0.18.0, which was fine few months ago in the same context, as @gebi, I was able to push the image using --registry-mirror index.docker.io.

@tejal29 may it be related to dockerhub hostname or default image path that has maybe changed and is no more compatible with kaniko (in old versions at least)?

I’m using auths with https://index.docker.io/v1/ in config.json.

KO:

$ docker run --rm --entrypoint "" -v /host/path/to/kaniko/config.json:/kaniko/.docker/config.json -v /host/path/to/dockerfile/directory/kaniko/20200825-001/build1:/workspace gcr.io/kaniko-project/executor:debug-v0.18.0 /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile --destination index.docker.io/tanguydelignieres/kaniko_bugs_20200825-001_build1:debug-v0.18.0
INFO[0003] Resolved base name alpine:3.9 to alpine:3.9
INFO[0003] Resolved base name alpine:3.9 to alpine:3.9
INFO[0003] Retrieving image manifest alpine:3.9
INFO[0005] Retrieving image manifest alpine:3.9
INFO[0009] Built cross stage deps: map[]
INFO[0009] Retrieving image manifest alpine:3.9
INFO[0011] Retrieving image manifest alpine:3.9
INFO[0014] Skipping unpacking as no commands require it.
INFO[0014] Taking snapshot of full filesystem...
INFO[0014] Resolving paths
INFO[0014] CMD echo "OK"
error pushing image: failed to push to destination index.docker.io/tanguydelignieres/kaniko_bugs_20200825-001_build1:debug-v0.18.0: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:tanguydelignieres/kaniko_bugs_20200825-001_build1 Type:repository] map[Action:push Class: Name:tanguydelignieres/kaniko_bugs_20200825-001_build1 Type:repository] map[Action:pull Class: Name:library/alpine Type:repository]]

$

OK with --registry-mirror index.docker.io:

$ docker run --rm --entrypoint "" -v /host/path/to/kaniko/config.json:/kaniko/.docker/config.json -v /host/path/to/dockerfile/directory/kaniko/20200825-001/build1:/workspace gcr.io/kaniko-project/executor:debug-v0.18.0 /kaniko/executor --registry-mirror index.docker.io --context /workspace --dockerfile /workspace/Dockerfile --destination index.docker.io/tanguydelignieres/kaniko_bugs_20200925-001_build1:debug-v0.18.0
INFO[0002] Resolved base name alpine:3.9 to alpine:3.9
INFO[0002] Resolved base name alpine:3.9 to alpine:3.9
INFO[0002] Retrieving image manifest alpine:3.9
INFO[0004] Retrieving image manifest alpine:3.9
INFO[0007] Built cross stage deps: map[]
INFO[0007] Retrieving image manifest alpine:3.9
INFO[0008] Retrieving image manifest alpine:3.9
INFO[0010] Skipping unpacking as no commands require it.
INFO[0010] Taking snapshot of full filesystem...
INFO[0010] Resolving paths
INFO[0010] CMD echo "OK"

$

I confirm I did not face the problem with debug-v1.0.0.

And there i was sitting the last sunday half a day thinking to be that stupid to build a simple image which i wanted to push to my private docker hub.

auths: [https://index.docker.io/v**2**/]

did NOT work for me.

• debug (without version)
• debug-v0.22.0
• debug-v0.19.0

did work for me.

• debug-v0.16.0

auths: [https://index.docker.io/v**1**/]

did work for me.

• debug (without version)
• debug-v0.22.0
• debug-v0.19.0
• debug-v0.16.0

What combination should i use, since i have no idea what the difference it’s making?

• v1
• v2

This is also happening to us using the lastest version of “debug” (May 6, 2020) attempting to push to GCR. Seeing this error:

E0506 17:32:28.676141      16 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Changing to tag debug-v0.19.0 things start working again.

Version 0.22.0 fixed my issue.

@gebi thanks, I confirm with the mgit/base:kaniko-executor-debug-stable image I was able to push to DockerHub but with gcr.io/kaniko-project/executor:debug I was experiencing the same error. Thanks!

This is my script

- export BASE64_CREDENTIALS="$(echo -n $DOCKER_HUB_REPOSITORY_USERNAME:$DOCKER_HUB_REPOSITORY_PASSWORD | base64)"
# @see https://github.com/GoogleContainerTools/kaniko/issues/1227 where folder may not exist
- mkdir -p /kaniko/.docker
- echo "{\"auths\":{\"https://index.docker.io/v2/\":{\"auth\":\"$BASE64_CREDENTIALS\"}}}" > /kaniko/.docker/config.json
- /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/$IMAGE_TAG/Dockerfile --destination $IMAGE_NAME:$IMAGE_TAG

Used variables to make it more meaningful for newbies (like myself 1 hour ago)