jx: DevPod can't be deployed because of deprecated skaffold digest hex

Summary

DevPods worked for me until a few days ago, when suddenly they wouldn’t be deployed anymore. I’d just get an ingress error 503. The problem is rooted in the skaffold.yaml, I think. It says:

    ...
    profiles:
    - name: dev
      build:
        artifacts:
        - docker: {}
        tagPolicy:
          envTemplate:
            template: '{{.DOCKER_REGISTRY}}/organisation/repo:{{.DIGEST_HEX}}'
        local: {}
      deploy:
        helm:
          releases:
          - name: brownbag
            chartPath: charts/brownbag
            setValueTemplates:
              image.repository: '{{.DOCKER_REGISTRY}}/organisation/repo'
              image.tag: '{{.DIGEST_HEX}}'

Since Skaffold v0.23.0, DIGEST_HEX and other DIGEST variables are deprecated and replaced by _DEPRECATED_<value_name>_, which affects pushing the images to the docker registry. So when running watch.sh, they are not tagged with a digest hex, but with the string literal _DEPRECATED_DIGEST_HEX_:

   curl docker-registry.jx.<ip>.nip.io/v2/organisation/repo/tags/list | jq .
    {
      "name": "organisation/repo",
      "tags": [
        "_DEPRECATED_DIGEST_HEX_",
        "0.0.1"
      ]
    }

Nonetheless, when retrieving the same value for the helm chart a few lines down, the replacement works as intended, and so the deployments expects the actual digest hex value as the tag. The pod creation fails with ImagePullBackOff:

   
   kubectl describe pod repo-repo-589d4d987-5sbqk -n=jx-edit-simon

   Events:
   Type     Reason     Age                     From                                                Message
   ----     ------     ----                    ----                                                -------
   Normal   Scheduled  8m44s                   default-scheduler                                   Successfully assigned  jx-edit-simon/repo-repo-589d4d987-5sbqk to gke-beakripple-default-pool-21d1dc97-p7gp
   Normal   Pulling    7m14s (x4 over 8m43s)   kubelet, gke-beakripple-default-pool-21d1dc97-p7gp  pulling image  "10.35.243.48:5000/organisation/repo:03efa48a18fd3b9310db6637abcb8e1f3cc1656ff0d8939b71713f194a3b97a6"
   Warning  Failed     7m14s (x4 over 8m43s)   kubelet, gke-beakripple-default-pool-21d1dc97-p7gp  Failed to pull image  "10.35.243.48:5000/organisation/repo:03efa48a18fd3b9310db6637abcb8e1f3cc1656ff0d8939b71713f194a3b97a6": rpc error: code =  Unknown desc = Error response from daemon: manifest for 10.35.243.48:5000/organisation/repo:03efa48a18fd3b9310db6637abcb8e1f3cc1656ff0d8939b71713f194a3b97a6 not found
   Warning  Failed     7m14s (x4 over 8m43s)   kubelet, gke-beakripple-default-pool-21d1dc97-p7gp  Error: ErrImagePull
   Warning  Failed     6m46s (x7 over 8m42s)   kubelet, gke-beakripple-default-pool-21d1dc97-p7gp  Error: ImagePullBackOff
   Normal   BackOff    3m33s (x21 over 8m42s)  kubelet, gke-beakripple-default-pool-21d1dc97-p7gp  Back-off pulling image  "10.35.243.48:5000/organisation/repo:03efa48a18fd3b9310db6637abcb8e1f3cc1656ff0d8939b71713f194a3b97a6"

Steps to reproduce the behavior

Using the versions specified below, set up a DevPod by running jx create devpod -l go --reuse --sync and start both watch.sh in the DevPod and jx sync locally.

Expected behavior

When files change, the images are tagged correctly, so that they can be found during deployment into my dev environment.

Actual behavior

The image is pushed to the registry with the tag _DEPRECATED_DIGEST_HEX_ and the helm chart expects the value of the digest hex when deploying into in the dev environment.

Jx version

The output of jx version is:

    NAME               VERSION
    jx                 1.3.1072
    Kubernetes cluster v1.11.7-gke.12
    kubectl            v1.14.0
    helm client        v2.13.0+g79d0794
    helm server        v2.13.0+g79d0794
    git                git version 2.17.1
    Operating System   Ubuntu 18.04.2 LTS

The putput of skaffold version in the devpod is:

    WARN[0000] Using SKAFFOLD_DEPLOY_NAMESPACE env variable is deprecated. Please use SKAFFOLD_NAMESPACE instead.
    WARN[0000] Using SKAFFOLD_DEPLOY_NAMESPACE env variable is deprecated. Please use SKAFFOLD_NAMESPACE instead.
    WARN[0000] Using SKAFFOLD_DEPLOY_NAMESPACE env variable is deprecated. Please use SKAFFOLD_NAMESPACE instead.
    WARN[0000] Using SKAFFOLD_DEPLOY_NAMESPACE env variable is deprecated. Please use SKAFFOLD_NAMESPACE instead.
    WARN[0000] Using SKAFFOLD_DEPLOY_NAMESPACE env variable is deprecated. Please use SKAFFOLD_NAMESPACE instead.
    v0.25.0

Jenkins type

Classic Jenkins

Kubernetes cluster

GKE, using jx create cluster --provider gke

Operating system / Environment

see above

Workaround

I came up with a unpractical workaround: I added an environment variable containing the datetime at the skaffold run, which replaces the digest hex. I would much rather fix this with one of the supported tag policies, like SHA256. Unfortunately, this can’t be combined with the envTemplate approach, so you’d have to hard-code the docker registry address. Can you evaluate if my assumptions are correct and if there are strategies to fix this without breaking the current workflow? Let me know, I’d be happy to fix this if possible.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 5
Comments: 19 (5 by maintainers)

Most upvoted comments

Workaround:

#!/usr/bin/env bash

# watch the java files and continously deploy the service
mvn clean install
export UUID=$(uuidgen)
skaffold run -p dev
reflex -r "\.java$" -- bash -c 'export UUID=$(uuidgen) && mvn install && skaffold run -p dev'

Then on the skaffold.yaml

        template: '{{.DOCKER_REGISTRY}}/foxsports/bedrock-k8s:{{.UUID}}'

        setValueTemplates:
          #      TODO: CHANGE_ME
          image.repository: '{{.DOCKER_REGISTRY}}/foxsports/bedrock-k8s'
          image.tag: '{{.UUID}}'

sajid2045 on May 5, 2019

May I ask what you practically wrote into the skaffold.yaml to achieve this? If I just use dateTime, I am getting an “Error response from daemon: invalid reference format”.

in my skaffold.yaml, I changed DIGEST_HEX to CURRENT_TIME, and then after making the devpod, from the pod’s shell (before building with ./watch.sh) export CURRENT_TIME="201904250817a"

jeremy-seiu on Apr 25, 2019

It is described here in the Skaffold releases notes. https://github.com/GoogleContainerTools/skaffold/blob/master/CHANGELOG.md#v0230-release---2142019

Which makes me think more and more people will run into this. Let’s see if we can find a proper fix for this that works for everyone.

joostvdg on May 16, 2019

Those workarounds are simply not necessary. Here’s my first try (tested, working on GKE + GCR, Skaffold 0.29) which I came up with just by reading Skaffold documentation.

DIGEST_HEX etc are deprecated for a reason: a new tagging mechanism exists, that does this without using environment variables (envTemplate -> tagPolicy: sha256).

apiVersion: skaffold/v1beta10
kind: Config
build:
  artifacts:
  - image: YOUR_PROJECTNAMESPACE/IMAGE
    context: .
    docker: {}
  tagPolicy:
    envTemplate:
      template: '{{.DOCKER_REGISTRY}}/{{.IMAGE_NAME}}:{{.VERSION}}'
  local: {}
deploy:
  kubectl: {}
profiles:
- name: dev
  build:
    tagPolicy:
        sha256: {}
    artifacts:
    - image: YOUR_DOCKER_REGISTRY/NAMESPACE/APP
    local: {}
  deploy:
    helm:
      releases:
      - name: goqs
        chartPath: charts/goqs
        setValueTemplates:
          # NOT CHANGED! This may threw a warning.
          image.repository: '{{.DOCKER_REGISTRY}}/{{.IMAGE_NAME}}'
          image.tag: '{{.DIGEST}}'

Removal of DIGEST_HEX etc is approaching - an issue mentions “not before 15/05/2019”. The quickstarts should be updated.

freeo on May 28, 2019