skaffold: skaffold build fails in specific Linux box with - copying sources: uploading build context:

This issue seems very similar to issue 3944 , but doesn’t help. It appears that any additional logging that was added as part of 3944 does not cover my scenario. Note that the same build works on windows and some other Linux machines (with same os version), but fails on this particular one.

$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)

Here’s my test Dockerfile

#Redhat Linux base image
FROM mycontainerregistry-############redhat7:latest
RUN yum install -y wget

Expected behavior

skaffold build should succeed. Even if it fails there should be a descriptive error. The issue seems machine specific so adding more logs would help in pin-pointing the real issue.

Actual behavior

skaffold build fails with “copying sources: uploading build context:”

$ skaffold build -p local --verbosity=debug
INFO[0000] Skaffold &{Version:v1.14.0 ConfigVersion:skaffold/v2beta7 GitVersion: GitCommit:4b3ca59af505c4ab5d5b6960a194e1f6887018f8 GitTreeState:clean BuildDate:2020-09-02T19:38:54Z GoVersion:go1.14.2 Compiler:gc Platform:linux/amd64}
DEBU[0000] config version "skaffold/v2beta5" out of date: upgrading to latest "skaffold/v2beta7"
INFO[0000] Loaded Skaffold defaults from "/home/#####/.skaffold/config"
INFO[0000] applying profile: local
DEBU[0000] overlaying profile on config for field Build
DEBU[0000] overlaying profile on config for field artifacts
DEBU[0000] overlaying profile on config for field insecureRegistries
DEBU[0000] overlaying profile on config for field tagPolicy
INFO[0000] no values found in profile for field TagPolicy, using original config values
DEBU[0000] overlaying profile on config for field BuildType
INFO[0000] no values found in profile for field BuildType, using original config values
DEBU[0000] overlaying profile on config for field Test
DEBU[0000] overlaying profile on config for field Deploy
DEBU[0000] overlaying profile on config for field DeployType
DEBU[0000] overlaying profile on config for field helm
DEBU[0000] overlaying profile on config for field kpt
DEBU[0000] overlaying profile on config for field kubectl
DEBU[0000] overlaying profile on config for field kustomize
DEBU[0000] overlaying profile on config for field statusCheckDeadlineSeconds
DEBU[0000] overlaying profile on config for field kubeContext
DEBU[0000] overlaying profile on config for field logs
DEBU[0000] overlaying profile on config for field prefix
DEBU[0000] overlaying profile on config for field PortForward
INFO[0000] applying profile: local
DEBU[0000] overlaying profile on config for field Build
DEBU[0000] overlaying profile on config for field artifacts
DEBU[0000] overlaying profile on config for field insecureRegistries
DEBU[0000] overlaying profile on config for field tagPolicy
INFO[0000] no values found in profile for field TagPolicy, using original config values
DEBU[0000] overlaying profile on config for field BuildType
INFO[0000] no values found in profile for field BuildType, using original config values
DEBU[0000] overlaying profile on config for field Test
DEBU[0000] overlaying profile on config for field Deploy
DEBU[0000] overlaying profile on config for field DeployType
DEBU[0000] overlaying profile on config for field helm
DEBU[0000] overlaying profile on config for field kpt
DEBU[0000] overlaying profile on config for field kubectl
DEBU[0000] overlaying profile on config for field kustomize
DEBU[0000] overlaying profile on config for field statusCheckDeadlineSeconds
DEBU[0000] overlaying profile on config for field kubeContext
DEBU[0000] overlaying profile on config for field logs
DEBU[0000] overlaying profile on config for field prefix
DEBU[0000] overlaying profile on config for field PortForward
INFO[0000] applying profile: local
DEBU[0000] overlaying profile on config for field Build
DEBU[0000] overlaying profile on config for field artifacts
DEBU[0000] overlaying profile on config for field insecureRegistries
DEBU[0000] overlaying profile on config for field tagPolicy
INFO[0000] no values found in profile for field TagPolicy, using original config values
DEBU[0000] overlaying profile on config for field BuildType
INFO[0000] no values found in profile for field BuildType, using original config values
DEBU[0000] overlaying profile on config for field Test
DEBU[0000] overlaying profile on config for field Deploy
DEBU[0000] overlaying profile on config for field DeployType
DEBU[0000] overlaying profile on config for field helm
DEBU[0000] overlaying profile on config for field kpt
DEBU[0000] overlaying profile on config for field kubectl
DEBU[0000] overlaying profile on config for field kustomize
DEBU[0000] overlaying profile on config for field statusCheckDeadlineSeconds
DEBU[0000] overlaying profile on config for field kubeContext
DEBU[0000] overlaying profile on config for field logs
DEBU[0000] overlaying profile on config for field prefix
DEBU[0000] overlaying profile on config for field PortForward
INFO[0000] applying profile: local
DEBU[0000] overlaying profile on config for field Build
DEBU[0000] overlaying profile on config for field artifacts
DEBU[0000] overlaying profile on config for field insecureRegistries
DEBU[0000] overlaying profile on config for field tagPolicy
INFO[0000] no values found in profile for field TagPolicy, using original config values
DEBU[0000] overlaying profile on config for field BuildType
INFO[0000] no values found in profile for field BuildType, using original config values
DEBU[0000] overlaying profile on config for field Test
DEBU[0000] overlaying profile on config for field Deploy
DEBU[0000] overlaying profile on config for field DeployType
DEBU[0000] overlaying profile on config for field helm
DEBU[0000] overlaying profile on config for field kpt
DEBU[0000] overlaying profile on config for field kubectl
DEBU[0000] overlaying profile on config for field kustomize
DEBU[0000] overlaying profile on config for field statusCheckDeadlineSeconds
DEBU[0000] overlaying profile on config for field kubeContext
DEBU[0000] overlaying profile on config for field logs
DEBU[0000] overlaying profile on config for field prefix
DEBU[0000] overlaying profile on config for field PortForward
INFO[0000] applying profile: local
DEBU[0000] overlaying profile on config for field Build
DEBU[0000] overlaying profile on config for field artifacts
DEBU[0000] overlaying profile on config for field insecureRegistries
DEBU[0000] overlaying profile on config for field tagPolicy
INFO[0000] no values found in profile for field TagPolicy, using original config values
DEBU[0000] overlaying profile on config for field BuildType
INFO[0000] no values found in profile for field BuildType, using original config values
DEBU[0000] overlaying profile on config for field Test
DEBU[0000] overlaying profile on config for field Deploy
DEBU[0000] overlaying profile on config for field DeployType
DEBU[0000] overlaying profile on config for field helm
DEBU[0000] overlaying profile on config for field kpt
DEBU[0000] overlaying profile on config for field kubectl
DEBU[0000] overlaying profile on config for field kustomize
DEBU[0000] overlaying profile on config for field statusCheckDeadlineSeconds
DEBU[0000] overlaying profile on config for field kubeContext
DEBU[0000] overlaying profile on config for field logs
DEBU[0000] overlaying profile on config for field prefix
DEBU[0000] overlaying profile on config for field PortForward
INFO[0000] applying profile: local
DEBU[0000] overlaying profile on config for field Build
DEBU[0000] overlaying profile on config for field artifacts
DEBU[0000] overlaying profile on config for field insecureRegistries
DEBU[0000] overlaying profile on config for field tagPolicy
INFO[0000] no values found in profile for field TagPolicy, using original config values
DEBU[0000] overlaying profile on config for field BuildType
INFO[0000] no values found in profile for field BuildType, using original config values
DEBU[0000] overlaying profile on config for field Test
DEBU[0000] overlaying profile on config for field Deploy
DEBU[0000] overlaying profile on config for field DeployType
DEBU[0000] overlaying profile on config for field helm
DEBU[0000] overlaying profile on config for field kpt
DEBU[0000] overlaying profile on config for field kubectl
DEBU[0000] overlaying profile on config for field kustomize
DEBU[0000] overlaying profile on config for field statusCheckDeadlineSeconds
DEBU[0000] overlaying profile on config for field kubeContext
DEBU[0000] overlaying profile on config for field logs
DEBU[0000] overlaying profile on config for field prefix
DEBU[0000] overlaying profile on config for field PortForward
INFO[0000] applying profile: local
DEBU[0000] overlaying profile on config for field Build
DEBU[0000] overlaying profile on config for field artifacts
DEBU[0000] overlaying profile on config for field insecureRegistries
DEBU[0000] overlaying profile on config for field tagPolicy
INFO[0000] no values found in profile for field TagPolicy, using original config values
DEBU[0000] overlaying profile on config for field BuildType
INFO[0000] no values found in profile for field BuildType, using original config values
DEBU[0000] overlaying profile on config for field Test
DEBU[0000] overlaying profile on config for field Deploy
DEBU[0000] overlaying profile on config for field DeployType
DEBU[0000] overlaying profile on config for field helm
DEBU[0000] overlaying profile on config for field kpt
DEBU[0000] overlaying profile on config for field kubectl
DEBU[0000] overlaying profile on config for field kustomize
DEBU[0000] overlaying profile on config for field statusCheckDeadlineSeconds
DEBU[0000] overlaying profile on config for field kubeContext
DEBU[0000] overlaying profile on config for field logs
DEBU[0000] overlaying profile on config for field prefix
DEBU[0000] overlaying profile on config for field PortForward
INFO[0000] applying profile: local
DEBU[0000] overlaying profile on config for field Build
DEBU[0000] overlaying profile on config for field artifacts
DEBU[0000] overlaying profile on config for field insecureRegistries
DEBU[0000] overlaying profile on config for field tagPolicy
INFO[0000] no values found in profile for field TagPolicy, using original config values
DEBU[0000] overlaying profile on config for field BuildType
INFO[0000] no values found in profile for field BuildType, using original config values
DEBU[0000] overlaying profile on config for field Test
DEBU[0000] overlaying profile on config for field Deploy
DEBU[0000] overlaying profile on config for field DeployType
DEBU[0000] overlaying profile on config for field helm
DEBU[0000] overlaying profile on config for field kpt
DEBU[0000] overlaying profile on config for field kubectl
DEBU[0000] overlaying profile on config for field kustomize
DEBU[0000] overlaying profile on config for field statusCheckDeadlineSeconds
DEBU[0000] overlaying profile on config for field kubeContext
DEBU[0000] overlaying profile on config for field logs
DEBU[0000] overlaying profile on config for field prefix
DEBU[0000] overlaying profile on config for field PortForward
INFO[0000] applying profile: local
DEBU[0000] overlaying profile on config for field Build
DEBU[0000] overlaying profile on config for field artifacts
DEBU[0000] overlaying profile on config for field insecureRegistries
DEBU[0000] overlaying profile on config for field tagPolicy
INFO[0000] no values found in profile for field TagPolicy, using original config values
DEBU[0000] overlaying profile on config for field BuildType
INFO[0000] no values found in profile for field BuildType, using original config values
DEBU[0000] overlaying profile on config for field Test
DEBU[0000] overlaying profile on config for field Deploy
DEBU[0000] overlaying profile on config for field DeployType
DEBU[0000] overlaying profile on config for field helm
DEBU[0000] overlaying profile on config for field kpt
DEBU[0000] overlaying profile on config for field kubectl
DEBU[0000] overlaying profile on config for field kustomize
DEBU[0000] overlaying profile on config for field statusCheckDeadlineSeconds
DEBU[0000] overlaying profile on config for field kubeContext
DEBU[0000] overlaying profile on config for field logs
DEBU[0000] overlaying profile on config for field prefix
DEBU[0000] overlaying profile on config for field PortForward
DEBU[0000] Defaulting deploy type to kubectl
INFO[0000] Using kubectl context: ########################@##########################
DEBU[0000] Using builder: cluster
DEBU[0000] setting Docker user agent to skaffold-v1.14.0
Generating tags...
 - mycontainerregistry-##########/############/############myimage -> mycontainerregistry-##########/#############/#############myimage:latest
INFO[0000] Tags generated in 71.597µs
Creating docker config secret [build-secret]...
DEBU[0000] getting client config for kubeContext: ``
DEBU[0000] No docker config specified. Checking for one in the cluster.
Building [mycontainerregistry-##########/#############/###########myimage]...
DEBU[0000] getting client config for kubeContext: ``
INFO[0000] Waiting for kaniko-wgzp9 to be initialized
DEBU[0005] Running command: [kubectl --context ########################@########################## exec -i kaniko-wgzp9 -c kaniko-init-container -n build-dev -- tar -xf - -C /kaniko/buildcontext]
DEBU[0005] FIXME: Got an status-code for which error does not match any expected type!!!: -1  module=api status_code=-1
couldn't build "mycontainerregistry-##########/#############/##########myimage": copying sources: uploading build context:

Information

• Skaffold version: v1.14.0
• Operating system: Red Hat Enterprise Linux Server release 7.7 (Maipo)
• Contents of skaffold.yaml:

apiVersion: skaffold/v2beta5
kind: Config
build:
  artifacts:
    - image: mycontainerregistry-##########/############/##########myimage
      kaniko:
        initImage: mycontainerregistry-##########/############/##################redhat:7-stable
        image: mycontainerregistry-##########/############/##################/kaniko-executor:latest
  tagPolicy:
    sha256: {}

  cluster:
    namespace: build-dev
profiles:
  - name: local
    patches:
      - op: add
        path: /build/cluster/dockerConfig
        value: {secretName: "build-secret"}

Multiple people have independently looked into it and found no obvious issue with the environment , so any additional logs when the build fails would be helpful. The behavior is same for 1.10.1 version as well.