grype: grype cannot scan a local image that was not archived by tar or pushed to a registry

I have the following pipeline in Tekton:

  params:
    - name: IMAGE
    - name: BASE_IMAGE_TRIVY
      type: string
      default: "aquasec/trivy:0.41.0"
    - name: BASE_IMAGE_GRYPE
      type: string
      default: "anchore/grype:v0.62.1-debug"
    - name: PATH_CONTEXT
      description: The build context.
      default: "."
  steps:
    - name: trivy
      image: $(params.BASE_IMAGE_TRIVY)
      workingDir: $(workspaces.source.path)/$(params.PATH_CONTEXT)
      script: |
        #!/usr/bin/env sh
        set -ex

        trivy image $(params.IMAGE)

    - name: grype
      image: $(params.BASE_IMAGE_GRYPE)
      workingDir: $(workspaces.source.path)/$(params.PATH_CONTEXT)
      script: |
        #!/busybox/sh
        set -ex

        /grype $(params.IMAGE)

Trivy manages to scan the image built with Kaniko, while grype causes an error (it does not find this image locally and tries to pull it from remote):

+ /grype some_aws_image:latest
1 error occurred:
	* failed to catalog: could not fetch image "some_aws_image:latest": unable to use OciRegistry source: failed to get image descriptor from registry: GET https://some_aws_image_link: unexpected status code 401 Unauthorized: Not Authorized

I also tried grype locally. Although, when I install this until via brew, it works alright and can scan images present on the machine. But I cannot scan local images with the grype docker container. The option --volume /var/run/docker.sock:/var/run/docker.sock does not help.

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 22 (9 by maintainers)

Most upvoted comments

@mirekphd thanks for the additional info, unfortunately I still have not been able to reproduce:

❯ echo 'FROM ubuntu:latest
RUN touch /foo' > Dockerfile

❯ docker build -t mirekphd/test-image:20231005_unpushed .
[+] Building 0.1s (6/6) FINISHED                                                                          docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                      0.0s
 => => transferring dockerfile: 108B                                                                                      0.0s
 => [internal] load .dockerignore                                                                                         0.0s
 => => transferring context: 2B                                                                                           0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                                          0.0s
 => [1/2] FROM docker.io/library/ubuntu:latest                                                                            0.0s
 => [2/2] RUN touch /foo                                                                                                  0.1s
 => exporting to image                                                                                                    0.0s
 => => exporting layers                                                                                                   0.0s
 => => writing image sha256:6d2fed83a95399186ebde47403c5fa1a334b850632d0882b70c8d85efab6cd3f                              0.0s
 => => naming to docker.io/mirekphd/test-image:20231005_unpushed

❯ docker run -it \
  -e GRYPE_DB_CACHE_DIR=/grype-db \
  --rm --name grype-test \
  -v /Users/wagoodman/Library/Caches/grype/db:/grype-db \
  -v /var/run/docker.sock:/var/run/docker.sock \
      anchore/grype:latest \
          mirekphd/test-image:20231005_unpushed
 ✔ Vulnerability DB                [no update available]
 ✔ Loaded image                                                                         mirekphd/test-image:20231005_unpushed
 ✔ Parsed image                                       sha256:6d2fed83a95399186ebde47403c5fa1a334b850632d0882b70c8d85efab6cd3f
 ✔ Cataloged packages              [102 packages]
 ✔ Scanned for vulnerabilities     [23 vulnerability matches]
   ├── by severity: 0 critical, 2 high, 6 medium, 12 low, 3 negligible
   └── by status:   2 fixed, 21 not-fixed, 0 ignored
NAME              INSTALLED                 FIXED-IN         TYPE  VULNERABILITY   SEVERITY
bash              5.1-6ubuntu1                               deb   CVE-2022-3715   Low
coreutils         8.32-4.1ubuntu1                            deb   CVE-2016-2781   Low
...

If I run with verbose logging (-vvv) I see:

[0000] DEBUG image: source=DockerDaemon location=mirekphd/test-image:20231005_unpushed
[0000] DEBUG image metadata: digest=sha256:6d2fed83a95399186ebde47403c5fa1a334b850632d0882b70c8d85efab6cd3f mediaType=application/vnd.docker.distribution.manifest.v2+json tags=[mirekphd/test-image:20231005_unpushed]
[0000] DEBUG layer metadata: index=0 digest=sha256:61bbeda53374249345217a09cd78040f124ef8bc07a1612a1a186d2468f67d54 mediaType=application/vnd.docker.image.rootfs.diff.tar.gzip
[0000] DEBUG layer metadata: index=1 digest=sha256:629d1dd2c23ff92a4de510698828bfade99e5f5caa85cf09a2d0be073f9332cf mediaType=application/vnd.docker.image.rootfs.diff.tar.gzip
...

Which shows that the underlying lib is selecting the DockerDaemon as expected.

I can use the registry scheme in the command to force getting the same behavior you’re seeing:

[0000] DEBUG image: source=OciRegistry location=mirekphd/test-image:20231005_unpushed
[0000] DEBUG pulling image info directly from registry image="mirekphd/test-image:20231005_unpushed"
[0000] DEBUG no registry credentials configured for "index.docker.io", using the default keychain
[0002]  INFO new version of grype is available: 0.69.1 (currently running: 0.69.0)
[0006] DEBUG found database update candidate: Listing(url=https://toolbox-data.anchore.io/grype/databases/vulnerability-db_v5_2023-10-05T01:24:28Z_17b232c51ce3734f92f5.tar.gz)
[0006] DEBUG existing database is already up to date
[0006] DEBUG no database update available
[0006] TRACE worker stopped component=eventloop
[0006] TRACE signal exit component=eventloop
1 error occurred:
	* failed to catalog: unable to load image: unable to use OciRegistry source: failed to get image descriptor from registry: GET https://index.docker.io/v2/mirekphd/test-image/manifests/20231005_unpushed: MANIFEST_UNKNOWN: manifest unknown; unknown tag=20231005_unpushed

… however, that’s because this scheme forces grype down this path, but the example before this shows the correct default behavior (it reaches out to the local docker daemon).

At a surface level this looks closer to what @tgerla mentioned earlier, where the docker socket might be inaccessible.

If I try again but use a different socket path, I can also get the same (bad) behavior:

# note the "-v /var/run/docker.sock:/var/run/docker.sock-oopswrongpath"...

❯ docker run -it -e GRYPE_DB_CACHE_DIR=/grype-db --rm --name grype-test -v /Users/wagoodman/Library/Caches/grype/db:/grype-db -v /var/run/docker.sock:/var/run/docker.sock-oopswrongpath anchore/grype:latest mirekphd/test-image:20231005_unpushed -vvv
[0000] TRACE trying podman socket path=/run/user/0/podman/podman.sock
[0000] TRACE trying podman socket path=/run/podman/podman.sock
[0000] TRACE unable to connect to podman via unix socket error=no socket address
github.com/anchore/stereoscope/internal/podman.init
	/home/runner/go/pkg/mod/github.com/anchore/stereoscope@v0.0.0-20230925132944-bf05af58eb44/internal/podman/client.go:18
runtime.doInit1
	/opt/hostedtoolcache/go/1.21.1/x64/src/runtime/proc.go:6740
runtime.doInit
	/opt/hostedtoolcache/go/1.21.1/x64/src/runtime/proc.go:6707
runtime.main
	/opt/hostedtoolcache/go/1.21.1/x64/src/runtime/proc.go:249
runtime.goexit
	/opt/hostedtoolcache/go/1.21.1/x64/src/runtime/asm_arm64.s:1197
[0000] TRACE trying containerd socket path=/run/containerd/containerd.sock
[0000] DEBUG image: source=OciRegistry location=mirekphd/test-image:20231005_unpushed
[0000] DEBUG pulling image info directly from registry image="mirekphd/test-image:20231005_unpushed"
[0000] DEBUG no registry credentials configured for "index.docker.io", using the default keychain
[0000] DEBUG no new grype update available
[0000] DEBUG found database update candidate: Listing(url=https://toolbox-data.anchore.io/grype/databases/vulnerability-db_v5_2023-10-05T01:24:28Z_17b232c51ce3734f92f5.tar.gz)
[0000] DEBUG existing database is already up to date
[0000] DEBUG no database update available
[0000] TRACE worker stopped component=eventloop
[0000] TRACE signal exit component=eventloop
1 error occurred:
	* failed to catalog: unable to load image: unable to use OciRegistry source: failed to get image descriptor from registry: GET https://index.docker.io/v2/mirekphd/test-image/manifests/20231005_unpushed: MANIFEST_UNKNOWN: manifest unknown; unknown tag=20231005_unpushed

… which is expected since the docker socket simply isn’t accessible.

When the docker socket can’t be found it searches for a podman socket, then containerd socket, then finally attempts to reach out to the registry.

@mirekphd can you post the results of your failed run with verbose logging?

I think you will see similar output as above, showing that it’s attempting these local options before getting to OCI. Since docker is the first default it doesn’t show “trying docker socket” in the log (we should add that to make it a little more clear though).

Do those steps fail for you?

They work fine, but are not sufficient. What differs in real-life settings when the error is raised:

  • the repo user already exists in the remote registry (as e.g. mirekphd in the Docker Hub)
  • the scanned image is not merely an alias of an image whose all layers have been already fully pushed to the remote registry (because then the pushed alias would be successfully scanned, not the new local perfect alias that we think we are scanning).

So starting from the above example let’s trigger the issue. I’ve built an and pushed as mirekphd/test-image:20231005 a test image based on alpine:latest as above. As expected from pushed images no amount of tagging could then stop Grype scans from working (even those including full remote repo name):

SUCCESS:

  1. pushed images (see: https://hub.docker.com/r/mirekphd/test-image/tags) :
  • 20231005,
  • latest,
  • 20230423
  1. aliases of pushed images:
  • the remaining images names and tags below (some with hidden docker.io tags):
registry.hub.docker.com/mirekphd/test-image                           20231005_v3                    607e4d2f5006   About an hour ago   114MB
registry.hub.docker.com/mirekphd/test-image                           20231005_v5                    607e4d2f5006   About an hour ago   114MB
registry.hub.docker.com/mirekphd/test-image                           20231005_v6                    607e4d2f5006   About an hour ago   114MB
test                                                                  whatever                       607e4d2f5006   About an hour ago   114MB
mirekphd/test-image                                                   20231005                       607e4d2f5006   About an hour ago   114MB
mirekphd/test-image                                                   20231005_v2                    607e4d2f5006   About an hour ago   114MB
mirekphd/test-image                                                   20231005_v3                    607e4d2f5006   About an hour ago   114MB
mirekphd/test-image                                                   20231005_v4                    607e4d2f5006   About an hour ago   114MB
mirekphd/test-image                                                   20231005_v5                    607e4d2f5006   About an hour ago   114MB
mirekphd/test-image                                                   latest                         aaf63ea74b74   5 months ago        27.1MB
mirekphd/test-image                                                   20230423                       57b1b70d29fc   5 months ago        27.1MB

However, when I changed the base image to include a never-before pushed (to my namespace) base image with ubuntu:latest (I have pushed only ubuntu:22.04), so that the image ID changed to a never-pushed 96aa83446e66, then the Grype scan revealed that it needs after all to pull the image from the remote repo before the scan…

More info in the log below:

$ echo 'FROM ubuntu:latest
RUN touch /foo' > Dockerfile

$ docker build -t mirekphd/test-image:20231005_unpushed .
[+] Building 0.4s (6/6) FINISHED                                                                                                                                                             docker:default
 => [internal] load .dockerignore                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                        0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                   0.0s
 => => transferring dockerfile: 64B                                                                                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                                                                                                                       0.4s
 => [1/2] FROM docker.io/library/ubuntu@sha256:9b8dec3bf938bc80fbe758d856e96fdfab5f56c39d44b0cff351e847bb1b01ea                                                                                        0.0s
 => CACHED [2/2] RUN touch /foo                                                                                                                                                                        0.0s
 => exporting to image                                                                                                                                                                                 0.0s
 => => exporting layers                                                                                                                                                                                0.0s
 => => writing image sha256:96aa83446e6604b507f09fcfd3d08de047b4c779b3882e7d836b39effd60e5cf                                                                                                           0.0s
 => => naming to docker.io/mirekphd/test-image:20231005_unpushed                                                                                                                                       0.0s

$ docker images | grep "whatever\|test-image"
mirekphd/test-image                                                   20231005_unpushed              96aa83446e66   17 minutes ago      77.8MB
test                                                                  whatever                       607e4d2f5006   55 minutes ago      114MB
mirekphd/test-image                                                   20231005                       607e4d2f5006   55 minutes ago      114MB
mirekphd/test-image                                                   20231005_v2                    607e4d2f5006   55 minutes ago      114MB
mirekphd/test-image                                                   20231005_v3                    607e4d2f5006   55 minutes ago      114MB
mirekphd/test-image                                                   20231005_v4                    607e4d2f5006   55 minutes ago      114MB
mirekphd/test-image                                                   20231005_v5                    607e4d2f5006   55 minutes ago      114MB
registry.hub.docker.com/mirekphd/test-image                           20231005_v3                    607e4d2f5006   55 minutes ago      114MB
registry.hub.docker.com/mirekphd/test-image                           20231005_v5                    607e4d2f5006   55 minutes ago      114MB
registry.hub.docker.com/mirekphd/test-image                           20231005_v6                    607e4d2f5006   55 minutes ago      114MB
mirekphd/test-image                                                   latest                         aaf63ea74b74   5 months ago        27.1MB
mirekphd/test-image                                                   20230423                       57b1b70d29fc   5 months ago        27.1MB

$ ~/git/build-server-scripts/grype/grype-starter.sh mirekphd/test-image:20251005_unpushed | grep High
1 error occurred:
	* failed to catalog: unable to load image: unable to use DockerDaemon source: pull failed: Error response from daemon: manifest for mirekphd/test-image:20251005_unpushed not found: manifest unknown: manifest unknown

Note I use a dockerized grype. So the contents of the used scanning script are:

$ cat ~/git/build-server-scripts/grype/grype-starter.sh
IMAGE=$1

docker run --rm --name grype-test -v /var/run/docker.sock:/var/run/docker.sock anchore/grype:latest $IMAGE

@YevheniiPokhvalii it’s not restricted to Kanico, you can update the issue title, as the classic docker build is sufficient to reproduce the issue (see scenarios below). Looks like grype pulls manifest data (or even entire images) from remote registry, and hence it scan only remote images, i.e. ones that have been already pushed to a registry.

If grype OSS version had a --container_type (or --container-type) flag like the grype fork made by Harness apparently does, it would be permanently set to: --container_type="docker_v2" rather than the setting needed for scanning locally build, unpushed images, which was long possible under Clair (and e.g. clair-scanner) and is now also possible with Trivy (as per OP report): --container_type="local"

One consequence is that grype scans won’t work on air-gapped servers that have no access to any registry, which is different from how Clair and its clients work - they let you scan local images without pushing them first to a registry. But as a redeeming feature (a side note here, but something that needed to be excluded first), it does not have a root requirement (all of the tests below were executed using the official anchore/grype:latest container run with docker under a non-root user belonging to the docker group).

@wagoodman to reproduce the issue:

I. FAIL scenario

  1. docker build an image; do not push it; In my case I can use mirekphd/ml-cache:20230726 as an example of an image available locally (e.g. for docker run) but never pushed to the Docker Hub.
  2. Try to scan the locally available image with grype and you will get something like this:
$ ./scan-with-grype-dockerized.sh mirekphd/ml-cache:20230726
[0000]  INFO grype version: 0.64.2
1 error occurred:
	* failed to catalog: unable to load image: unable to use OciRegistry source: failed to get image descriptor from registry: GET https://index.docker.io/v2/mirekphd/ml-cache/manifests/20230726: MANIFEST_UNKNOWN: manifest unknown; unknown tag=20230726

vs.:

II. SUCCESS scenario

  1. docker build an image;
  2. push it to the Docker Hub; in my case I can use mirekphd/ml-cache:20230727 as an example of an image already pushed to the Docker Hub (to a public repo)
  3. Try to scan with grype and you will likely succeed, like this:
$ ./scan-with-grype-dockerized.sh mirekphd/ml-cache:20230727
[0000]  INFO grype version: 0.64.2
[0002]  INFO identified distro: Alpine Linux v3.16 from-lib=syft
[0002]  INFO cataloging an image from-lib=syft
[0002]  INFO found 28 vulnerabilities for 18 packages
[0002]  INFO ignoring 12 matches due to user-provided ignore rules
libcrypto1.1  1.1.1s-r0  CVE-2022-4450  High    /lib/apk/db/installed
libcrypto1.1  1.1.1s-r0  CVE-2023-0215  High    /lib/apk/db/installed
libcrypto1.1  1.1.1s-r0  CVE-2023-0286  High    /lib/apk/db/installed
libcrypto1.1  1.1.1s-r0  CVE-2023-0464  High    /lib/apk/db/installed
libcrypto1.1  1.1.1s-r0  CVE-2023-2650  High    /lib/apk/db/installed
libssl1.1     1.1.1s-r0  CVE-2022-4450  High    /lib/apk/db/installed
libssl1.1     1.1.1s-r0  CVE-2023-0215  High    /lib/apk/db/installed
libssl1.1     1.1.1s-r0  CVE-2023-0286  High    /lib/apk/db/installed
libssl1.1     1.1.1s-r0  CVE-2023-0464  High    /lib/apk/db/installed
libssl1.1     1.1.1s-r0  CVE-2023-2650  High    /lib/apk/db/installed
libcrypto1.1  1.1.1s-r0  CVE-2022-4304  Medium  /lib/apk/db/installed
libcrypto1.1  1.1.1s-r0  CVE-2023-0465  Medium  /lib/apk/db/installed
libssl1.1     1.1.1s-r0  CVE-2022-4304  Medium  /lib/apk/db/installed
libssl1.1     1.1.1s-r0  CVE-2023-0465  Medium  /lib/apk/db/installed