kaniko: Kaniko failing to push to ECR

Hi all and @tiru1930

Issue is fix for me. Stupid problem but I dont’t know if it’s a bug or not…

As a reminder

• With gitlabci and multi-stage image builder my gitlab.com push : OK .

• With gitlabci and simple-stage image builder my ECR push : OK .

• With gitlabci and multi-stage image builder my ECR push : FAILED .

For my ECR authentification I used the following line :

echo “{"credHelpers":{"$ECR_REGISTRY":"ecr-login"}}” > /kaniko/.docker/config.json

It’s important not to use kaniko user (home directory) for config.json storage. In my case I used /root/.docker/config.json after this issue and my multistage to ECR was good.

          mkdir -p /kaniko/.docker
          cat <<EOF | tee /kaniko/.docker/config.json
          {
              "auths": {
                  "${CI_REGISTRY_IMAGE}": {
                      "username": "${CI_REGISTRY_USER}",
                      "password": "${CI_REGISTRY_PASSWORD}"
                  }
              },
              "credHelpers": {
                  "${aws_account_id}.dkr.ecr.${region}.amazonaws.com": "ecr-login"
              }
          }
          EOF
          export AWS_PROFILE=${stage}

this appears to work even without having the $HOME/.aws/config file set I reckon kaniko only checks if the PROFILE is set or not…

• It would be great to know if kaniko supports the actual aws cli env vars AWS_ACCOUNT_ID and so forth. In the example above I explicitly set the lowercase vars in the CI/CD job
• The .aws/config is not set in the working example…

I don’t think I understand the issue well enough to comment. I need to find some time to use Kaniko with ECR myself.

If folks think that documenting AWS_PROFILE behavior would be helpful, I can approve a PR to add that to the README until I can debug it myself.

@Jasper-Ben issue is with AWS_PROFILE due to some reason profile is not working i change the name of profile to default and it start working.

I also have the same problem. Single stage or multi stage pipelines make no difference and both fail with an EOF error: error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "<accountid>.dkr.ecr.<region>.amazonaws.com/<reponame>:latest": Post "https://<accountid>dkr.ecr.<region>.amazonaws.com/v2/<reponame>/blobs/uploads/": EOF This problem has also been mentioned here but the solution is not clear: #1935

Are you using aws credentials or IAM roles for authentication? Are you using kaniko v1.8.0?

Thank you for your fast reply. I use aws credentials for the authentication and kaniko v1.8.0 in a gitlab CICD pipeline

I have a similar issue, but setting credHelpers does not seem to help 😞. However, I am getting an “EOF” error. Maybe I am doing something wrong?

Error message:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "<account_id>.dkr.ecr.eu-central-1.amazonaws.com/<image_name>:<image_tag>": Post "https://<account_id>.dkr.ecr.eu-central-1.amazonaws.com/<image_name>/blobs/uploads/": EOF

My terraform config:

# use a unix timestamp as image tag
resource "time_static" "image_tag" {
  triggers = {
    repository_url = var.ecr_repository_url
  }
}

resource "kubernetes_config_map" "kaniko_config" {
  metadata {
    name      = "${var.prefix}${var.project_name}-kaniko-config"
    namespace = var.namespace
  }
  data = {
    "config.json" = jsonencode(
      {
        credsStore = "ecr-login"
        credHelpers = {
          "${data.aws_caller_identity.current.account_id}.dkr.ecr.${var.aws_region}.amazonaws.com" = "ecr-login"
        }
      }
    )
  }
}

# build and push container image
resource "kubernetes_job" "build_image" {
  metadata {
    name      = "${var.prefix}${var.project_name}-build"
    namespace = var.namespace
    labels    = local.common_labels_k8s
  }
  spec {
    template {
      metadata {}
      spec {
        container {
          name = "kaniko"
          image             = "gcr.io/kaniko-project/executor:v1.6.0-debug"
          image_pull_policy = "IfNotPresent"
          args = [
            "--dockerfile=/mnt/Dockerfile",
            "--context=/mnt",
            "--destination=${time_static.image_tag.triggers.repository_url}:${time_static.image_tag.unix}"
          ]
          volume_mount {
            mount_path = "/mnt"
            name       = kubernetes_secret.build_files.metadata[0].name
          }
          volume_mount {
            mount_path = "/kaniko/.docker"
            name       = kubernetes_config_map.kaniko_config.metadata[0].name
          }
        }
        service_account_name = kubernetes_service_account.service_account.metadata[0].name
        restart_policy       = "Never"
        volume {
          name = kubernetes_secret.build_files.metadata[0].name
          secret {
            secret_name = kubernetes_secret.build_files.metadata[0].name
          }
        }
        volume {
          name = kubernetes_config_map.kaniko_config.metadata[0].name
          config_map {
            name = kubernetes_config_map.kaniko_config.metadata[0].name
          }
        }
      }
    }
  }
}

I am giving the following IAM permissions using OIDC:

      "ecr:GetAuthorizationToken",
      "ecr:BatchGetImage",
      "ecr:BatchCheckLayerAvailability",
      "ecr:CompleteLayerUpload",
      "ecr:GetDownloadUrlForLayer",
      "ecr:InitiateLayerUpload",
      "ecr:PutImage",
      "ecr:UploadLayerPart"

@jordan85 Weird, if it’s only during a multi-stage that indeed sounds like a bug. Sorry I’m not a kaniko dev, just trying to help. 😦 Likely pasting/pastebin-ing a verbose log would be helpful for devs debugging it though. Can’t hurt.

Best of luck!