trivy: Trivy image scans inside docker hangs

Description

Hey,

In my jenkins pipeline I run trivy image scans before publishing the images. Trivy is running as a container:

docker.image('aquasec/trivy:latest').inside("""-v /var/run/docker.sock:/var/run/docker.sock -u 0 --entrypoint=''""") {
    trivyScanResult = sh(script: "/usr/local/bin/trivy image ...", returnStdout: true)
}

In the last couple of days my jobs are failing when they reach the scan due to one of the following:

trivy consuming 100% CPU
trivy reaching timeout of 5 or 10 minutes

This happens with small (30 MB) and large (> 1GB) images. All images are stored locally on the machine during the scan.

When I login to the machines and start the container manually, I observe the same issue. If I download trivy to the machines and run it outside of docker, everything works fine.

One of the machines I am using was running trivy, with the same code, without any issues for a few months.

What did you expect to happen?

trivy scans usually finish in a matter of seconds to a minute.

What happened instead?

trivy hangs / timesout.

Output of run with `-debug`:

/usr/local/bin/trivy --debug image --severity HIGH,CRITICAL <image>
2022-04-26T07:41:56.279Z	DEBUG	Severities: HIGH,CRITICAL

Output of `trivy -v`:

Version: 0.27.0
Vulnerability DB:
  Version: 2
  UpdatedAt: 2022-04-26 06:06:39.247959855 +0000 UTC
  NextUpdate: 2022-04-26 12:06:39.247959555 +0000 UTC
  DownloadedAt: 2022-04-26 07:05:35.00901385 +0000 UTC

Additional details (base image name, container registry info…):

This does not seems to happen in version 0.25.3. with version 0.25.3 the scan process takes 10-20 seconds.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 15

Most upvoted comments

So I was experiencing the same issue when running trivy on a docker image. We basically ran the following commands in our bamboo-build.sh: scan --exit-code 0 --format template --template "@contrib/html.tpl" -o /reports/report.html scan --exit-code 1 --ignore-unfixed --severity CRITICAL --offline-scan

The first of these seemed to be doing fine, and exiting the scan as expected. The second one however would get stuck without letting know why or on which files. after checking the files it was accessing in a different window on the same image with the lsof command we would see that it would not access anything after the following 25 /usr/local/bin/trivy 9 /cache/db/trivy.db

After this we went through each individual part of the commands we were executing, which eventually led us to the following solution turning the second command into the following: scan --exit-code 1 --format template --template "@contrib/html.tpl" -o /reports/report-CRITICAL.html --ignore-unfixed --severity CRITICAL --offline-scan and letting our builds succeed.

I hope this might help some of you or maybe the developers in solving this issue of the trivy scans getting stuck in docker.

StijnNoot on Apr 26, 2022