skaffold: Order of deployers doesn't match order listed in `skaffold.yaml`

Expected Behavior

I have a skaffold.yaml file that contains the lines:

apiVersion: skaffold/v2alpha3
kind: Config

metadata:
  name: discover

profiles:
  - name: local
    activation:
      - kubeContext: minikube
    build:
      local:
        useBuildkit: true
        push: false
      artifacts:
        - blah
    deploy:
      kubectl:
        manifests:
          - create-discover-app-namespace.json
      helm:
        releases:
          - name: dev
            chartPath: ./kubernetes/discover-app/
            etc: "etc"

Because helm3 does not automatically create namespaces (anymore/yet), I have a kubectl deployer that is intended to create the namespace for the subsequent helm deployer. (I also have in mind some other things I’d like to do with kubectl before helm runs - basically, I think there are other use cases besides creating namespaces.)

The kubectl deploy doesn’t always run first, then the helm deploy fails. Following is a very trimmed debug output showing that.

I expect the kubectl deployer to run to completion before the helm deployer runs.

I have not at all investigated the skaffold source to see what changes could be made, sorry. If this issue is a low priority I may get to that and augment this issue with more information.

Actual Behavior

See these logs; the helm deployer is running before the kubectl deployer and then nothing is working.

(minikube:discover-app)➜  discover git:(acaird/AT-233-env-vars) ✗ skaffold -v debug run
INFO[0000] Skaffold &{Version:v1.6.0-docs ConfigVersion:skaffold/v2beta1 GitVersion: GitCommit:b74e2f94f628b16a866abddc2ba8f05ce0bf956c GitTreeState:clean BuildDate:2020-03-25T00:09:12Z GoVersion:go1.14 Compiler:gc Platform:linux/amd64}
DEBU[0000] config version (skaffold/v2alpha3) out of date: upgrading to latest (skaffold/v2beta1)
INFO[0000] applying profile: local
DEBU[0000] overlaying profile on config for field Build
 [...]
INFO[0000] Using kubectl context: minikube
DEBU[0000] Using builder: local
DEBU[0000] Running command: [minikube docker-env --shell none]
DEBU[0000] Command output: [DOCKER_TLS_VERIFY=1
DOCKER_HOST=tcp://127.0.0.1:32769
DOCKER_CERT_PATH=/home/acaird/.minikube/certs
MINIKUBE_ACTIVE_DOCKERD=minikube
]
DEBU[0000] setting Docker user agent to skaffold-v1.6.0-docs
Generating tags...
 - discover-db-migrator -> DEBU[0000] Running command: [git describe --tags --always]
DEBU[0000] Running command: [git describe --tags --always]
 [...]
DEBU[0006] Running command: [helm version]
INFO[0006] Deploying with helm v3.1.2 ...
DEBU[0006] Executing template &{envTemplate 0xc0003fc900 0xc00003b7c0  } with environment map[COLORTERM:truecolor...
DEBU[0006] Running command: [helm --kube-context minikube get all --namespace discover-app dev]
Helm release dev not installed. Installing...
DEBU[0006] EnvVarMap: map[DIGEST:discover-db-migrator:95cbaac2cf5f353ff87ca131d242f305f0a4b449b330b9faec0f63173fa7ed73...
DEBU[0006] Executing template &{envTemplate 0xc00011f600 0xc000a560c0  } with environment map[COLORTERM:truecolor...
DEBU[0006] Executing template &{envTemplate 0xc00062e000 0xc000a561c0  } with environment map[COLORTERM:truecolor...
 [...]
DEBU[0006] Executing template &{envTemplate 0xc00062f200 0xc000a574c0  } with environment map[COLORTERM:truecolor...
DEBU[0006] Executing template &{envTemplate 0xc00062f300 0xc000a57640  } with environment map[COLORTERM:truecolor...
DEBU[0006] Running command: [helm --kube-context minikube install dev ./kubernetes/discover-app/ --namespace discover-app --set-string api.image.fullName=discover-web-server:797b046c95f1d9ad8a088fe841c62f5f385d8e4fed27a6cdb914317c7d414419 [...]
coalesce.go:196: warning: cannot overwrite table with non table for nodeSelector (map[node_pool:association-node-pool])
coalesce.go:196: warning: cannot overwrite table with non table for nodeSelector (map[node_pool:association-node-pool])
Error: create: failed to create: namespaces "discover-app" not found
DEBU[0007] Running command: [helm --kube-context minikube get all --namespace discover-app dev]
WARN[0007] exit status 1
DEBU[0007] getting client config for kubeContext: ``
DEBU[0007] getting client config for kubeContext: ``
DEBU[0007] Running command: [kubectl version --client -ojson]
DEBU[0007] Command output: [{
  "clientVersion": {
    "major": "1",
    "minor": "17",
    "gitVersion": "v1.17.4",
    "gitCommit": "8d8aa39598534325ad77120c120a22b3a990b5ea",
    "gitTreeState": "clean",
    "buildDate": "2020-03-12T21:03:42Z",
    "goVersion": "go1.13.8",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}
]
DEBU[0007] Running command: [kubectl --context minikube create --dry-run -oyaml -f /home/acaird/git/discover/create-discover-app-namespace.json]
DEBU[0007] Command output: [apiVersion: v1
kind: Namespace
metadata:
  labels:
    name: discover-app
  name: discover-app
]
WARN[0007] image [discover-db-migrator] is not used by the deployment
 [...]
DEBU[0007] manifests with tagged images: apiVersion: v1
kind: Namespace
metadata:
  labels:
    name: discover-app
  name: discover-app
DEBU[0007] manifests with labels apiVersion: v1
kind: Namespace
metadata:
  labels:
    app.kubernetes.io/managed-by: skaffold-v1.6.0-docs
    name: discover-app
    skaffold.dev/builder: local
    skaffold.dev/cleanup: "true"
    skaffold.dev/deployer: kubectl
    skaffold.dev/docker-api-version: "1.40"
    skaffold.dev/run-id: 2bfe88b8-fe10-437f-8182-b82b55589ab2
    skaffold.dev/tag-policy: git-commit
    skaffold.dev/tail: "true"
  name: discover-app
DEBU[0007] 1 manifests to deploy. 1 are updated or new
DEBU[0007] Running command: [kubectl --context minikube apply -f -]
 - namespace/discover-app created
INFO[0007] Deploy complete in 688.331603ms
Waiting for deployments to stabilize
DEBU[0007] getting client config for kubeContext: ``
Deployments stabilized in 11.318892ms
You can also run [skaffold run --tail] to get the logs

Information

Skaffold Version:

~ skaffold version
v1.6.0-docs

and also:

INFO[0000] Skaffold &{Version:v1.6.0-docs ConfigVersion:skaffold/v2beta1 GitVersion: GitCommit:b74e2f94f628b16a866abddc2ba8f05ce0bf956c GitTreeState:clean BuildDate:2020-03-25T00:09:12Z GoVersion:go1.14 Compiler:gc Platform:linux/amd64}

Operating System:

~ cat /etc/os-release
NAME=Fedora
VERSION="31 (Workstation Edition)"
ID=fedora
VERSION_ID=31
VERSION_CODENAME=""
PLATFORM_ID="platform:f31"
PRETTY_NAME="Fedora 31 (Workstation Edition)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:31"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f31/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=31
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=31
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation

~ uname -a
Linux primary 5.5.10-200.fc31.x86_64 #1 SMP Wed Mar 18 14:21:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

See above for part of the skaffold.yaml file.

Steps to Reproduce the Behavior

I think that any example of a kubectl deployer (maybe a slow one, like create ns) followed by a helm deployer should show this. TBH I haven’t tried to create a minimal reproducible example, so let me know if that is needed.

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 17
Comments: 38 (17 by maintainers)

Most upvoted comments

Hi all, just wanted to give a quick update on this issue. We’re currently working through some broad sweeping changes in skaffold and are figuring out how to address this issue properly. We’d really like to have a good working solution for user’s needing this functionality, and will try to keep updated on this. Thanks for being patient 😃

MarlonGamez on Jan 26, 2021

The deploy node is a map, and YAML does not maintain an order on map keys. And the underlying data structure is a Go struct with named fields. So we do not have any information on the order of the deployers.

The actual deployment code orders deployers by Helm, Kubectl, and finally Kustomize.

Two possibilities:

Add an explicit mechanism to specify an order. For example:

 deploy:
    helm: ...
    kubectl: ...
    order: ["kubectl", "helm"]

Turn the deploy into a sequence of deployers. This would allow us to chain operations, including doing things like first installing a CRD, then configuring a namespace, then install some charts with helm, and …
```
 deploy:
 - kubectl: ...
 - helm: ...
 - kubectl: ...
```

briandealwis on Apr 3, 2020

update: this is getting prioritized in Q3

gsquared94 on Jun 21, 2021

@foobarbecue @SizZiKe 😄

just for transparency we did/are still prioritizing this, but it wasn’t resolved last quarter because we identified a few different aspects of the problem that we wanted to address individually. we’re currently designing a way to better support arbitrary CRDs in skaffold - this is planned to ship in Q4, and we’ll try and keep this issue updated with progress on that (though it’s probably more related to https://github.com/GoogleContainerTools/skaffold/issues/4081).

also, one potential solution i haven’t seen mentioned here: skaffold supports specifying dependencies between separate pipeline configurations (we sometimes refer to these as “modules”). what this will essentially let you do is define logical groupings of resources (modules), and then basically tell skaffold “module B requires module A to be deployed and available before attempting to deploy”. the --iterative-status-check flag will ensure that the required module’s status check has passed before starting any subsequent deployments.

nkubala on Oct 28, 2021

Any news on this? I’m considering giving up on skaffold due to this issue.

foobarbecue on Mar 30, 2021

+1 I need this too.

glenbot on Sep 18, 2020

I also need this feature, because I need a CRD to be set up before I install resources of that kind. Adding namespace creation abilities to skaffold wouldn’t help me. Either option 1 or 2 would be great.

foobarbecue on May 23, 2020

thanks @tejal29 . @gsquared94 is this prioritized in Q4 too?

foobarbecue on Oct 12, 2021

Sorry about that @acaird. We have plans for supporting order of deployments. We trying to revamp skaffold config to make this more explicit. We recently send out a survey with the same feature in mind. https://forms.gle/b5XETJcPRDRVu7p79

Sorry this is taking longer than usual but we want to address the crux of the problem here which is strong coupling between project configuration and phases in which they get used.

tejal29 on Apr 6, 2021

Thanks for a great tool! Here is a really silly workaround for the CRD case. One can add a phony build artifact 🙈

- image: phony
   custom:
     buildCommand: kubectl apply -f crd.yaml && echo FROM alpine | docker build -t phony -
     dependencies:
       paths:
         - crd.yaml

aleeve on Feb 23, 2021

@cpaika https://github.com/GoogleContainerTools/skaffold/pull/4765. this was just released with v1.15.0

nkubala on Sep 30, 2020

I think there have been several themes where having multiple instances of the same deployer kind would be necessary (e.g., install a CRD, or configure a PV). Is it possible to have deploy take either an object/map or array?

briandealwis on Jul 22, 2020

Hey @briandealwis I’d like to chime in with support for the second option you proposed. I also have the need to setup a few CRDs before deploying resources and would like to be able to do so in an automated fashion using Skaffold. The current workaround is to just manually apply the CRDs before using Skaffold.

I’m not familiar with the Skaffold code base, would you be able to offer any insight on the difficulty of implementing something like this?

mattsre on Jun 11, 2020

thanks @tejal29 . @gsquared94 is this prioritized in Q4 too?

savage +1 to this, being able to create existing claims, resources, etc via kubectl before deploying helm would be nice.

SizZiKe on Oct 21, 2021

When depending on CRDs to be present, I’ve begun to install those in a pre-deploy hook:

    deploy:
      kubectl:
        hooks:
          before:
            - host:
                command:
                  - bash
                  - -c
                  - |
                    set -euo pipefail
                    echo "Installing Contour CRDs ..."
                    kubectl apply -f https://raw.githubusercontent.com/projectcontour/contour/v1.18.1/examples/contour/01-crds.yaml
        manifests:
          - https://projectcontour.io/quickstart/contour.yaml
          - k8s/httpproxy.yml

mamachanko on Sep 24, 2021

In the meantime, can someone explain what the order is now? Is it essentially randomized every run?

foobarbecue on Jul 21, 2021

I am experiencing what I think is a similar issue. My skaffold file includes a CRD creation (for Traefik). In order to have that custom resources available when skaffold will create resources based on them, I placed explicitly the manifest (for CRD) as first one. However skaffold (kubectl) exits with error, it is complaining that names of some resources are unknonwn (exactly the ones that I expect to be created by CRD yaml).

Whether I manually create the CRD with kubectl, I can run skaffold that completes without any error.

Am I falling in this issue? Will the workaroud proposed by @RetWolf work for me?

I agree with @briandealwis that apparently order in kubectl references is referred to kustomize.

sw360cab on Jun 22, 2020

A third alternative is to address the specifics of this situation: have Skaffold be able to create and deploy into a namespace. dev could create and tear down the namespace, whereas run (and deploy?) would create and then leave it.

briandealwis on Apr 3, 2020