skaffold: skaffold returns x509 while using buildx script, while buildx works without skaffold

Expected behavior

skaffold dev with custom build script to build and push to secured private repo like docker buildx does independently of skaffold.

Actual behavior

buildx cmd successfully pushes to secured private repo
skaffold dev returns error: x509 certificate signed by unknown authority

Information

Skaffold version: v1.34.0
Operating system: macOS Big Sur 11.4
Installed via: homebrew
Contents of skaffold.yaml:

apiVersion: skaffold/v2beta25
kind: Config
build:
  artifacts:
  - image: skaffold-example
    context: .
    custom:
      buildCommand: ./build.sh
deploy:
  kubectl:
    manifests:
      - k8s-*

Contents of build.sh:

#!/usr/bin/env bash

docker buildx build \
  --platform linux/amd64 \
  --tag $IMAGE \
  --push \
  $BUILD_CONTEXT

Steps to reproduce the behavior

skaffold/examples/getting-started
skaffold dev --default-repo my-secured-private-repo

Additionally

I’ve attempted the same with the skaffold/examples/custom-buildx and received the same results
The cmd:

docker buildx build --platform linux/amd64 --tag my-secured-private-repo/skaffold-example --push .

successfully pushes to the private repo.

skaffold dev without the custom build script also pushes to the repo successfully. Suspect it has something to do with how skaffold checks for certs with a custom build cmd.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 16 (4 by maintainers)

Most upvoted comments

It sounds like either your Harbor certificate chain is missing an intermediary certificate, or one the chain’s roots or intermediates is marked as untrusted. If missing, you need to add this certificate to your configured Harbor certificate list.

Skaffold (via Golang) should uses the OS certificate store for its trusted roots, as does openssl (I’m fairly certain), whereas I believe Chrome maintains its own set of trusted roots (but maybe also accepts the OS roots too?). So it may be that Chrome includes the missing intermediate certificate.

I believe you said you’re on a Mac? Safari uses the OS certificates and may help more easily diagnose the problem. Apparently curl -v will show certificate information.

briandealwis on Jan 8, 2022

Skaffold should use the same mechanisms used by Docker — using credential helpers defined in ~/.docker/config.json — to authenticate to the registry.

I’ve been using buildx to build multi-platform images for Skaffold’s support images for skaffold debug, and haven’t had any problems like this.

So I’m baffled as to why it would not work for the custom builder, but does for other builders. If you could please run with -vdebug and redact any sensitive information, that would be helpful.

briandealwis on Nov 15, 2021

Confirmed running with custom buildx cmd works as expected, post certificate alteration.

Thanks for all the help, esp since it wasn’t a skaffold issue at all. 🙏🏻

ianmcfall on Feb 21, 2022

@ianbuilds does the cert chain as seen by Chrome match what you see when you run the openssl command? What does the chain look like when viewed from the browser (i.e. by clicking on the lock icon and viewing the cert that way)? It would be interesting to see what Chrome sees and trusts for the root cert.

dinofizz on Jan 8, 2022

I realize the Docker daemon normally returns the resulting image digest and so Skaffold doesn’t need to query the remote registry explicitly like we do with the custom builder. So I suspect you have an issue with your local certificate store — perhaps something to do with Let’s Encrypt’s root expiry?

Skaffold leverages both [GGCR](https://github.com/google/go-containerregistry] and Docker libraries for fetching digests from remote registries. I suspect that you would have the same problem with connecting via either a Go CLI program.

Try connecting using openssl to your remote registry and see there is an error. Below I show connecting to gcr.io and to badssl.

$ openssl s_client -showcerts -connect gcr.io:443
CONNECTED(00000005)
depth=3 C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
verify return:1
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = *.googlecode.com
verify return:1
---
...

$ openssl s_client -showcerts -connect untrusted-root.badssl.com:443
depth=0 C = US, ST = California, L = San Francisco, O = BadSSL Fallback. Unknown subdomain or no SNI., CN = badssl-fallback-unknown-subdomain-or-no-sni
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = BadSSL Fallback. Unknown subdomain or no SNI., CN = badssl-fallback-unknown-subdomain-or-no-sni
verify error:num=21:unable to verify the first certificate
verify return:1
---
...

If that works, then try GGCR’s crane:

$ go install github.com/google/go-containerregistry/cmd/crane@latest
$ crane catalog --verbose <your-host>

briandealwis on Dec 13, 2021

Not self-signed certs; repo = harbor.io v2 instance with certs from godaddy.
Results from the --insecure-registry flag:

cmd: skaffold dev --insecure-registry my-repo --default-repo my-repo -vdebug:

skaffold dev --default-repo my-repo --insecure-registry my-repo -vdebug
DEBU[0000] skaffold API not starting as it's not requested  subtask=-1 task=DevLoop
INFO[0000] Skaffold &{Version:v1.34.0 ConfigVersion:skaffold/v2beta25 GitVersion: GitCommit:22cfab75ffb305e7af220910af2f48d0a5c0e6af BuildDate:2021-10-27T00:20:49Z GoVersion:go1.17.2 Compiler:gc Platform:darwin/arm64 User:}  subtask=-1 task=DevLoop
INFO[0000] Loaded Skaffold defaults from "/path/.skaffold/config"  subtask=-1 task=DevLoop
DEBU[0000] parsed 1 configs from configuration file /path/Developer/playground/hellok8s/skaffold/examples/getting-started/skaffold.yaml  subtask=-1 task=DevLoop
DEBU[0000] Defaulting build type to local build          subtask=-1 task=DevLoop
INFO[0000] Using kubectl context: falcon                 subtask=-1 task=DevLoop
DEBU[0000] Running command: [minikube version --output=json]  subtask=-1 task=DevLoop
DEBU[0000] setting Docker user agent to skaffold-v1.34.0  subtask=-1 task=DevLoop
DEBU[0000] Using builder: local                          subtask=-1 task=DevLoop
DEBU[0000] push value not present in NewBuilder, defaulting to true because cluster.PushImages is true  subtask=-1 task=DevLoop
INFO[0000] build concurrency first set to 1 parsed from *local.Builder[0]  subtask=-1 task=DevLoop
INFO[0000] final build concurrency value is 1            subtask=-1 task=DevLoop
Listing files to watch...
 - skaffold-example
INFO[0000] List generated in 539.25µs                    subtask=-1 task=DevLoop
Generating tags...
 - skaffold-example -> DEBU[0000] Running command: [git describe --tags --always]  subtask=-1 task=Build
DEBU[0000] Command output: [bfc43b0
]                    subtask=-1 task=Build
DEBU[0000] Running command: [git status . --porcelain]   subtask=-1 task=Build
DEBU[0000] Command output: [ M examples/getting-started/skaffold.yaml
?? examples/getting-started/build.sh
?? "examples/getting-started/sample output"
]  subtask=-1 task=Build
my-repo/skaffold-example:bfc43b0-dirty
INFO[0000] Tags generated in 68.772708ms                 subtask=-1 task=Build
Checking cache...
DEBU[0000] Could not import artifact from Docker, building instead (import of missing images disabled)  subtask=-1 task=Build
 - skaffold-example: Not found. Building
INFO[0000] Cache check completed in 1.455459ms           subtask=-1 task=Build
Starting build...
Building [skaffold-example]...
DEBU[0000] Executing template &{envars here]  subtask=-1 task=DevLoop
DEBU[0000] Running command: [sh -c ./build.sh]           subtask=skaffold-example task=Build
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 32B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#4 [internal] load metadata for docker.io/library/golang:1.15
#4 ...

#3 [internal] load metadata for docker.io/library/alpine:3
#3 DONE 0.8s

#4 [internal] load metadata for docker.io/library/golang:1.15
#4 DONE 0.9s

#5 [stage-1 1/2] FROM docker.io/library/alpine:3@sha256:635f0aa53d99017b38d1a0aa5b2082f7812b03e3cdb299103fe77b5c8a07f1d2
#5 DONE 0.0s

#6 [builder 1/3] FROM docker.io/library/golang:1.15@sha256:ea080cc817b02a946461d42c02891bf750e3916c52f7ea8187bccde8f312b59f
#6 DONE 0.0s

#7 [internal] load build context
#7 transferring context: 29B done
#7 DONE 0.0s

#8 [builder 2/3] COPY main.go .
#8 CACHED

#9 [builder 3/3] RUN go build -gcflags="${SKAFFOLD_GO_GCFLAGS}" -o /app main.go
#9 CACHED

#10 [stage-1 2/2] COPY --from=builder /app .
#10 CACHED

#11 exporting to image
#11 exporting layers done
#11 writing image sha256:abe0a6105645847f72148b932fbf8539f291e6a79083a0bc10a352ef13cd1bef done
#11 naming to my-repo/skaffold-example:bfc43b0-dirty done
#11 DONE 0.0s

#12 pushing my-repo/skaffold-example:bfc43b0-dirty with docker
#12 pushing layer 40c62b6900a2
#12 pushing layer 1a058d5342cc
#12 pushing layer 40c62b6900a2 131.58kB / 2.03MB 2.2s
#12 pushing layer 40c62b6900a2 1.38MB / 2.03MB 2.6s
#12 pushing layer 40c62b6900a2 1.57MB / 2.03MB 3.1s
#12 pushing layer 40c62b6900a2 2.04MB / 2.03MB 3.3s
#12 pushing layer 40c62b6900a2 5.8s done
#12 pushing layer 1a058d5342cc 6.7s done
#12 DONE 7.3s
DEBU[0009] Running command: [tput colors]                subtask=-1 task=DevLoop
DEBU[0009] Command output: [256
]                        subtask=-1 task=DevLoop
getting image: Get "https://my-repo/v2/": x509: certificate signed by unknown authority
DEBU[0009] exporting metrics                             subtask=-1 task=DevLoop

This cmd is being run from an M1 iMac, btw. Not sure if that’s relevant anyhow.

ianmcfall on Nov 20, 2021

getting image: Get "https://my-repo/v2/": x509: certificate signed by unknown authority

This seems to be failing when skaffold is pulling the image to check the digest. https://github.com/GoogleContainerTools/skaffold/blob/af75d9e6ca558b4b9207c6ed0fab3016af7c68c9/pkg/skaffold/docker/remote.go#L54

did you use a self-signed certificate to setup the image registry? Can you try setting this registry as an insecure registry and retrying?

gsquared94 on Nov 19, 2021