skaffold: Skipping deploy due to sync error: copying files: didn't sync any files

TL;DR

Skaffold Sync depends on the current namespace in $KUBECONFIG. When you develop locally with skaffold dev and switch the current namespace in $KUBECONFIG as part of the dev flow, Skaffold Sync becomes unreliable.

This issue has been reported earlier but issues were just closed without a fix:

• https://github.com/GoogleContainerTools/skaffold/issues/3685
• https://github.com/GoogleContainerTools/skaffold/issues/1668

Details and a workaround: https://github.com/GoogleContainerTools/skaffold/issues/1668#issuecomment-595752550

Expected behavior

Skaffold Sync should be able to copy files regardless of what happens to be the current namespace in $KUBECONFIG.

Actual behavior

Skaffold Sync will fail if artefact is deployed in a different namespace than which is defined in $KUBECONFIG as the current namespace.

Information

• Skaffold version: 1.10.1 (and older)
• Operating system: MacOS
• Contents of skaffold.yaml:

apiVersion: skaffold/v2alpha2
kind: Config
build:
  local:
    push: false
  artifacts:
    - image: lolcat
      context: lolcat
      sync:
        manual:
          - src: "app/**/*"
            dest: /

PROBLEM: Namespace in $KUBECONFIG affects Skaffold Sync

Skaffold sync will only work when the Kubernetes namespace in $KUBECONFIG, at the time of starting Skaffold, matches the namespace of the artifact (application) being deployed.

The critical moment is when Skaffold starts. After Skaffold is running, I can change my current namespace in $KUBECONFIG and Skaffold sync works neatly. But, if the namespace in $KUBECONFIG is different from the hot-sync-artifact’s namespace when Skaffold starts, then sync will fail with error WARN[0032] Skipping deploy due to sync error: copying files: didn't sync any files .

For Skaffold Sync to work, the namespace in $KUBECONFIG must either be undefined or it must equal to the hot-sync-artifact’s namespace.

Steps to reproduce the behavior

EXAMPLE: Namespace undefined

Without namespace being defined, Skaffold Sync works for artifacts in any namespace:

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /Users/pre/.minikube/ca.crt
    server: https://192.168.64.100:8443
  name: rex
contexts:
- context:
    cluster: rex
    user: rex
  name: rex
current-context: rex
kind: Config
preferences: {}
users:
- name: rex
  user:
    client-certificate: /Users/pre/.minikube/client.crt
    client-key: /Users/pre/.minikube/client.key

EXAMPLE: Namespace defined

Here namespace is defined and Skaffold Sync will fail for all artifacts (applications) which are not deployed in dev namespace:

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /Users/pre/.minikube/ca.crt
    server: https://192.168.64.100:8443
  name: rex
contexts:
- context:
    cluster: rex
    namespace: dev
    user: rex
  name: rex
current-context: rex
kind: Config
preferences: {}
users:
- name: rex
  user:
    client-certificate: /Users/pre/.minikube/client.crt
    client-key: /Users/pre/.minikube/client.key

Workaround

1. Reset the current namespace in $KUBECONFIG
2. Run skaffold dev without a namespace being defined in $KUBECONFIG
3. Create a subshell which switches back to the original namepace after Skaffold has started.

_dev() {
  local current_ns; current_ns="$(kubectl config view --minify --output 'jsonpath={..namespace}')"
  kubectl config set-context --current --namespace=
  (sleep 3 && kubectl config set-context --current --namespace="${current_ns}" ) &
    
  skaffold dev
}

Best things are achieved with a decent amount of sleep.