porter: Error on publish to Harbor 2.0 (OCI-compliant) registry

Describe the bug

Harbor 2.0 should be fully OCI-compliant (https://goharbor.io/blog/harbor-2.0/) registry, but Porter fails to publish CNAB bundles to it. In the Harbor UI, there are specific instructions on how to push CNAB packages using cnab-to-oci which makes even a stronger case why Porter should be able to publish to it. The push command is: cnab-to-oci push CNAB_PATH --target 172.31.33.xxx/library/REPOSITORY[:TAG] --auto-update-bundle

Expected behavior

CNAB bundle is successfully pushed to Harbor and available for deployment.

Porter Command and Output

# porter publish --debug
DEBUG name:    arm
DEBUG pkgDir: /root/.porter/mixins/arm
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/arm/arm version --output json --debug
DEBUG name:    aws
DEBUG pkgDir: /root/.porter/mixins/aws
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/aws/aws version --output json --debug
DEBUG name:    az
DEBUG pkgDir: /root/.porter/mixins/az
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/az/az version --output json --debug
DEBUG name:    exec
DEBUG pkgDir: /root/.porter/mixins/exec
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/exec/exec version --output json --debug
DEBUG name:    gcloud
DEBUG pkgDir: /root/.porter/mixins/gcloud
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/gcloud/gcloud version --output json --debug
DEBUG name:    helm
DEBUG pkgDir: /root/.porter/mixins/helm
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/helm/helm version --output json --debug
DEBUG name:    kubernetes
DEBUG pkgDir: /root/.porter/mixins/kubernetes
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/kubernetes/kubernetes version --output json --debug
DEBUG name:    terraform
DEBUG pkgDir: /root/.porter/mixins/terraform
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/terraform/terraform version --output json --debug
Pushing CNAB invocation image...
The push refers to repository [172.31.33.xxx/library/infra-cnab-installer]
08d3279ecc54: Preparing
b92dc245d8c2: Preparing
8363e3460c66: Preparing
4eb4a6386afd: Preparing
219410cd4347: Preparing
90ca8ca0957c: Preparing
8e7363b9dcd4: Preparing
9b7657a76455: Preparing
4e38024e7e09: Preparing
90ca8ca0957c: Waiting
8e7363b9dcd4: Waiting
9b7657a76455: Waiting
4e38024e7e09: Waiting
219410cd4347: Mounted from library/infra-install-cnab-installer
8363e3460c66: Pushed
08d3279ecc54: Pushed
90ca8ca0957c: Mounted from library/infra-install-cnab-installer
8e7363b9dcd4: Mounted from library/infra-install-cnab-installer
9b7657a76455: Mounted from library/infra-install-cnab-installer
4e38024e7e09: Mounted from library/infra-install-cnab-installer
4eb4a6386afd: Pushed
b92dc245d8c2: Pushed
0.0.1: digest: sha256:b1f065f32e9ca151fc47d6905c901ffb306ca7d01b92a096c130cb0d51989dca size: 2214

Rewriting CNAB bundle.json...
DEBUG name:    arm
DEBUG pkgDir: /root/.porter/mixins/arm
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/arm/arm version --output json --debug
DEBUG name:    aws
DEBUG pkgDir: /root/.porter/mixins/aws
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/aws/aws version --output json --debug
DEBUG name:    az
DEBUG pkgDir: /root/.porter/mixins/az
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/az/az version --output json --debug
DEBUG name:    exec
DEBUG pkgDir: /root/.porter/mixins/exec
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/exec/exec version --output json --debug
DEBUG name:    gcloud
DEBUG pkgDir: /root/.porter/mixins/gcloud
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/gcloud/gcloud version --output json --debug
DEBUG name:    helm
DEBUG pkgDir: /root/.porter/mixins/helm
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/helm/helm version --output json --debug
DEBUG name:    kubernetes
DEBUG pkgDir: /root/.porter/mixins/kubernetes
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/kubernetes/kubernetes version --output json --debug
DEBUG name:    terraform
DEBUG pkgDir: /root/.porter/mixins/terraform
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/terraform/terraform version --output json --debug
Starting to copy image 172.31.33.xxx/library/infra-cnab-installer:0.0.1...
Failed to copy image 172.31.33.xxx/library/infra-cnab-installer:0.0.1: failed to resolve or push image for service "InvocationImage"
Error: failed to resolve or push image for service "InvocationImage"

Version

porter v0.27.2 (aee93e98)

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 20 (9 by maintainers)

Most upvoted comments

The latest version of harbor works as a charm https://github.com/goharbor/harbor/releases/tag/v2.1.0 . Just follow the installation steps and you should be good to go. image image

I can confirm that downgrading to Porter v0.27.2 solves this issue:

# porter publish --insecure-registry
Pushing CNAB invocation image...
The push refers to repository [172.31.33.128:80/cnab/infra-cnab-installer]
6e350c9f6a7f: Preparing
5ef651ff6703: Preparing
fd450e0398e7: Preparing
076e2fbff553: Preparing
219410cd4347: Preparing
90ca8ca0957c: Preparing
8e7363b9dcd4: Preparing
9b7657a76455: Preparing
4e38024e7e09: Preparing
8e7363b9dcd4: Waiting
9b7657a76455: Waiting
4e38024e7e09: Waiting
90ca8ca0957c: Waiting
219410cd4347: Layer already exists
90ca8ca0957c: Layer already exists
fd450e0398e7: Pushed
8e7363b9dcd4: Layer already exists
4e38024e7e09: Layer already exists
9b7657a76455: Layer already exists
6e350c9f6a7f: Pushed
5ef651ff6703: Pushed
076e2fbff553: Pushed
1.0.0: digest: sha256:d0d130b51fa35a6fbe745cd7e79be90c502777270d7736403607c97b8d31ef54 size: 2214

Rewriting CNAB bundle.json...
Starting to copy image 172.31.33.128:80/cnab/infra-cnab-installer:1.0.0...
Completed image 172.31.33.128:80/cnab/infra-cnab-installer:1.0.0 copy
Starting to copy image busybox:latest...
Completed image busybox:latest copy
Starting to copy image kiwigrid/k8s-sidecar:0.1.20...
Completed image kiwigrid/k8s-sidecar:0.1.20 copy
Starting to copy image nginx:1.17.3-alpine...
Completed image nginx:1.17.3-alpine copy
Starting to copy image squareup/ghostunnel:v1.4.1...
Completed image squareup/ghostunnel:v1.4.1 copy
Starting to copy image rancher/rancher:v2.3.5...
Completed image rancher/rancher:v2.3.5 copy
Starting to copy image quay.io/jetstack/cert-manager-cainjector:v0.12.0...
Completed image quay.io/jetstack/cert-manager-cainjector:v0.12.0 copy
Starting to copy image chartmuseum/chartmuseum:v0.11.0...
Completed image chartmuseum/chartmuseum:v0.11.0 copy
Starting to copy image k8s.gcr.io/defaultbackend-amd64:1.5...
Completed image k8s.gcr.io/defaultbackend-amd64:1.5 copy
Starting to copy image grafana/grafana:6.3.5...
Completed image grafana/grafana:6.3.5 copy
Starting to copy image gcr.io/rec-repo/grafana-plugins:master...
Completed image gcr.io/rec-repo/grafana-plugins:master copy
Starting to copy image rancher/local-path-provisioner:v0.0.12...
Completed image rancher/local-path-provisioner:v0.0.12 copy
Starting to copy image nvidia/dcgm-exporter:1.4.6...
Completed image nvidia/dcgm-exporter:1.4.6 copy
Starting to copy image quay.io/jetstack/cert-manager-controller:v0.12.0...
Completed image quay.io/jetstack/cert-manager-controller:v0.12.0 copy
Starting to copy image quay.io/jetstack/cert-manager-webhook:v0.12.0...
Completed image quay.io/jetstack/cert-manager-webhook:v0.12.0 copy
Starting to copy image gcr.io/rec-repo/container-tools:1.0.7...
Completed image gcr.io/rec-repo/container-tools:1.0.7 copy
Starting to copy image quay.io/gravitational/debian-tall:stretch...
Completed image quay.io/gravitational/debian-tall:stretch copy
Starting to copy image quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.32.0...
Completed image quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.32.0 copy
Starting to copy image jettech/kube-webhook-certgen:v1.0.0...
Completed image jettech/kube-webhook-certgen:v1.0.0 copy
Bundle tag 172.31.33.128:80/cnab/infra-cnab:1.0.0 pushed successfully, with digest "sha256:e650842d764c4cd53a81dbcd940f17c47dfa841c3b76410d72f89e3100b76c31"

However, I had to comment-out one image docker.io/metallb/speaker:v0.9.3, because otherwise I am facing a different issue which I think is related to https://github.com/deislabs/porter/issues/1224:

# porter publish --insecure-registry
Pushing CNAB invocation image...
The push refers to repository [172.31.33.128:80/cnab/infra-cnab-installer]
a24d18766e7d: Preparing
ab26aade72c5: Preparing
0f7465cfb4e2: Preparing
8a3fdc0430dc: Preparing
219410cd4347: Preparing
90ca8ca0957c: Preparing
8e7363b9dcd4: Preparing
9b7657a76455: Preparing
4e38024e7e09: Preparing
8e7363b9dcd4: Waiting
9b7657a76455: Waiting
4e38024e7e09: Waiting
90ca8ca0957c: Waiting
219410cd4347: Layer already exists
90ca8ca0957c: Layer already exists
8e7363b9dcd4: Layer already exists
a24d18766e7d: Pushed
9b7657a76455: Layer already exists
4e38024e7e09: Layer already exists
0f7465cfb4e2: Pushed
ab26aade72c5: Pushed
8a3fdc0430dc: Pushed
1.0.0: digest: sha256:b35dcd79f7d744fcb8247a3b46a65a435a3faade0e6235806a89b4d1a518e017 size: 2214

Rewriting CNAB bundle.json...
Starting to copy image 172.31.33.128:80/cnab/infra-cnab-installer:1.0.0...
Completed image 172.31.33.128:80/cnab/infra-cnab-installer:1.0.0 copy
Starting to copy image gcr.io/rec-repo/container-tools:1.0.7...
Completed image gcr.io/rec-repo/container-tools:1.0.7 copy
Starting to copy image quay.io/gravitational/debian-tall:stretch...
Completed image quay.io/gravitational/debian-tall:stretch copy
Starting to copy image docker.io/metallb/speaker:v0.9.3...
Failed to copy image docker.io/metallb/speaker:v0.9.3: failed to copy: httpReaderSeeker: failed open: failed to do request: context canceled
Error: failed to copy: httpReaderSeeker: failed open: failed to do request: context canceled

Thanks for the extra information, all. If anyone is/was able to capture logs from the registry being published to, it’d be much appreciated. Also, I’m curious if using 0.27.2 leads to the same error(s)? edit: it sounds like 0.27.2 works fine

In the meantime, I’ll keep attempting to reproduce the issue… looking into spinning up a Harbor registry now.

@jeffersonbenson here’s my output

I just ran into this one today using version 0.28.1. I reverted back to 0.27.0 and the problem went away.

Extracting bundle from archive /home/ocpadmin/my/my.tgz… Beginning bundle publish to <internal ocp ip>:5000/my/my-cnab:0.1.0. This may take some time. Starting to copy image <internal ocp ip>:5000/my/my-porter-installer@sha256:f12da10ce3f308808c939f87d754cd5bfa7223cc3f806471baa077b933a09e57… Failed to copy image <internal ocp ip>:5000/my/my-installer@sha256:f12da10ce3f308808c939f87d754cd5bfa7223cc3f806471baa077b933a09e57: failed to resolve or push image for service “InvocationImage” Error: failed to resolve or push image for service “InvocationImage”

Sounds like this is resolved. Let me know if it’s still a problem and we can investigate.