harbor: Harbor Replication Issue: Invalid Image Manifest Causes Transfer Failure and Vulnerability Scan Error

Expected behavior and actual behavior: Since I use harbor I replicate (cache) my images into our registry and mirror them whereever I need them. Since OCI there seem to be some issues rising atm.

Steps to reproduce the problem: Sync OCI any linkerd image from current stable or quay.io/jetstack/trust-manager:v0.5.0 into your registry using a replica job.

Versions: Version

harbor version: v2.8.2-d4c34dcc
docker engine version: 24.0.2
docker-compose version: v2.18.1

Additional context: Replication into the registry:

Job succeeded

Replicate out of the registry:

2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:139]: client for source registry [type: harbor, URL: http://core:8080, insecure: true] created
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:149]: client for destination registry [type: docker-registry, URL: https://swr.eu-de.otc.t-systems.com, insecure: false] created
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:182]: copying cache-common/cert-manager/trust-manager:[v0.5.0](source registry) to [...]/cert-manager/trust-manager:[v0.5.0](destination registry)...
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying cache-common/cert-manager/trust-manager:v0.5.0(source registry) to [...]/cert-manager/trust-manager:v0.5.0(destination registry)...
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact cache-common/cert-manager/trust-manager:v0.5.0 ...
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact cache-common/cert-manager/trust-manager:v0.5.0 pulled
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying cache-common/cert-manager/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a(source registry) to [...]/cert-manager/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a(destination registry)...
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact cache-common/cert-manager/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a ...
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact cache-common/cert-manager/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a pulled
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:6be20a52f0e9ea097d4e334ca45ef7ad61b316374bb5aeff8fda9c5fe1b2a7fe(the 1th running)...
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:339]: the blob sha256:6be20a52f0e9ea097d4e334ca45ef7ad61b316374bb5aeff8fda9c5fe1b2a7fe already exists on the destination registry, skip
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:6be20a52f0e9ea097d4e334ca45ef7ad61b316374bb5aeff8fda9c5fe1b2a7fe completed
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:de11bf62c62d8e59c187692126a4f833a8647d1f1cfd70deecd50305b5a20202(the 1th running)...
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:339]: the blob sha256:de11bf62c62d8e59c187692126a4f833a8647d1f1cfd70deecd50305b5a20202 already exists on the destination registry, skip
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:de11bf62c62d8e59c187692126a4f833a8647d1f1cfd70deecd50305b5a20202 completed
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:496]: pushing the manifest of artifact ops-[...]/cert-manager/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a ...
2023-06-12T15:10:35Z [ERROR] [/controller/replication/transfer/image/transfer.go:504]: failed to push manifest of artifact ops-[...]/cert-manager/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a: http status code: 400, body: {"errors":[{"code":"MANIFEST_INVALID","message":"Invalid image, fail to parse 'manifest.json'"}]}
2023-06-12T15:10:35Z [ERROR] [/controller/replication/transfer/image/transfer.go:194]: http status code: 400, body: {"errors":[{"code":"MANIFEST_INVALID","message":"Invalid image, fail to parse 'manifest.json'"}]}
2023-06-12T15:10:35Z [ERROR] [/controller/replication/transfer/image/transfer.go:200]: got error during the whole transfer period, mark the job failure

Trivy logs

2023-06-12T15:02:40Z [INFO] [/pkg/scan/job.go:387]: {
  "uuid": "932f8b23-87a8-11eb-a090-0242ac1a0004",
  "name": "Trivy",
  "description": "The Trivy scanner adapter",
  "url": "http://trivy-adapter:8080",
  "disabled": false,
  "is_default": true,
  "health": "healthy",
  "auth": "",
  "access_credential": "[HIDDEN]",
  "skip_certVerify": false,
  "use_internal_addr": true,
  "adapter": "Trivy",
  "vendor": "Aqua Security",
  "version": "v0.42.0",
  "create_time": "2021-03-18T05:12:45.855453Z",
  "update_time": "2021-03-18T05:12:45.855455Z"
}
2023-06-12T15:02:40Z [INFO] [/pkg/scan/job.go:387]: {
  "registry": {
    "url": "http://core:8080",
    "authorization": "[HIDDEN]"
  },
  "artifact": {
    "namespace_id": 3211,
    "repository": "cache-common/cert-manager/trust-manager",
    "tag": "",
    "digest": "sha256:06c88ccf61e2d5f1c6beca7feca4b12c9b69a37ed7eb27dedc0a1a6db392d358",
    "mime_type": "application/vnd.oci.image.manifest.v1+json"
  }
}
2023-06-12T15:02:40Z [INFO] [/pkg/scan/job.go:167]: Report mime types: [application/vnd.security.vulnerability.report; version=1.1]
2023-06-12T15:02:40Z [INFO] [/pkg/scan/job.go:224]: Get report for mime type: application/vnd.security.vulnerability.report; version=1.1
2023-06-12T15:02:42Z [INFO] [/pkg/scan/job.go:245]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-06-12T15:02:47Z [ERROR] [/pkg/scan/job.go:294]: check scan report with mime type application/vnd.security.vulnerability.report; version=1.1: running trivy wrapper: running trivy: exit status 1: 2023-06-12T15:02:42.365Z	[34mINFO[0m	Vulnerability scanning is enabled
2023-06-12T15:02:42.552Z	[31mFATAL[0m	image scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:80b00ffd2d00d0c46ed7ad0a13877402398d0c3a9617423d6e4694c7bfed5964): walk error: failed to extract the archive: archive/tar: invalid tar header
: general response handler: unexpected status code: 500, expected: 200

The OCI in the registry looks like so

See issue on OCI https://github.com/opencontainers/image-spec/issues/1025

About this issue

Original URL
State: closed
Created a year ago
Comments: 38 (14 by maintainers)

Most upvoted comments

BTW, I think the attestations should ideally be linked to the artifact via OCI reference API rather than mixed in the image manifest list.

Yeah, I agree with you on that. And if these non-image artifacts are linked to subject artifacts via OCI reference API, they will be stored in Harbor as accessories, which will not be scanned as expected. It seems this should be the best practice, rather than putting these attestations in the image index file.

zyyw on Nov 13, 2023

BTW, I think the attestations should ideally be linked to the artifact via OCI reference API rather than mixed in the image manifest list.

knqyf263 on Nov 13, 2023

It is not a container image. It makes sense to fail to scan it. What does Harbor expect in this case? IMO, Harbor should not show the “Scan” button and trigger scanning on the provenance. It is not only provenance technically. The scanning should not be triggered on anything other than supported types, images (and SBOM). What do you think?

$ crane manifest quay.io/jetstack/trust-manager@sha256:06c88ccf61e2d5f1c6beca7feca4b12c9b69a37ed7eb27dedc0a1a6db392d358
{
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:6ddd1691a4cc2cf22f77434879f3b07450315e5574002b993c3d7b4471bd1d7d",
    "size": 167
  },
  "layers": [
    {
      "mediaType": "application/vnd.in-toto+json",
      "digest": "sha256:80b00ffd2d00d0c46ed7ad0a13877402398d0c3a9617423d6e4694c7bfed5964",
      "size": 1147,
      "annotations": {
        "in-toto.io/predicate-type": "https://slsa.dev/provenance/v0.2"
      }
    }
  ]
}

$ crane blob quay.io/jetstack/trust-manager@sha256:80b00ffd2d00d0c46ed7ad0a13877402398d0c3a9617423d6e4694c7bfed5964 | jq .
{
  "_type": "https://in-toto.io/Statement/v0.1",
  "predicateType": "https://slsa.dev/provenance/v0.2",
  "subject": [
    {
      "name": "pkg:docker/quay.io/jetstack/trust-manager@v0.5.0?platform=linux%2Famd64",
      "digest": {
        "sha256": "d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a"
      }
    }
  ],
  "predicate": {
    "builder": {
      "id": ""
    },
    "buildType": "https://mobyproject.org/buildkit@v1",
    "materials": [
      {
        "uri": "pkg:docker/golang@1.19?platform=linux%2Famd64",
        "digest": {
          "sha256": "9613596d7405705447f36440a59a3a2a1d22384c7568ae1838d0129964c5ba13"
        }
      }
    ],
    "invocation": {
      "configSource": {
        "entryPoint": "Dockerfile"
      },
      "parameters": {
        "frontend": "dockerfile.v0",
        "locals": [
          {
            "name": "context"
          },
          {
            "name": "dockerfile"
          }
        ]
      },
      "environment": {
        "platform": "linux/amd64"
      }
    },
    "metadata": {
      "buildInvocationID": "sdjwl6omebhb1sqxmrw6w93ug",
      "buildStartedOn": "2023-05-19T10:55:50.912447143Z",
      "buildFinishedOn": "2023-05-19T11:19:36.090418153Z",
      "completeness": {
        "parameters": false,
        "environment": true,
        "materials": false
      },
      "reproducible": false,
      "https://mobyproject.org/buildkit@v1#metadata": {
        "vcs": {
          "revision": "858eced8cb85116ebb406ebd71e238df8c37ed91",
          "source": "git@github.com:SgtCoDFish/cert-manager-trust-manager.git"
        }
      }
    }
  }
}

knqyf263 on Nov 13, 2023

From my perspective, the issue may relate with remote registry because from the replication logs it shows the process is normal, and eventually it failed due to receive the error from remote registry. You can try to replicate the image to other registries such as DockerHub for verification.

chlins on Jul 11, 2023