trivy: FATAL error in image scan: failed analysis: analyze error: timeout: context deadline exceeded

Description

Scanning some images consistently results in timeout and “context deadline exceeded” failure, seemingly with no pattern.

That is – for a given image docker.prod.skytap.com:5000/skytap/continuous_queries:*, scanning docker.prod.skytap.com:5000/skytap/continuous_queries:b835ff37ac1b8304d57060e12b917561387dd2f0 results in timeout / context deadline exceeded, but scanning the prior tag (barely any difference) docker.prod.skytap.com:5000/skytap/continuous_queries:4f9ee82b3f59792480212d18a4ea283b62c87879 will succeed just fine.

Both images have been docker pulled beforehand.

What did you expect to happen?

Trivy should successfully scan the image, or give a clearer actionable reason for failure.

What happened instead?

Trivy timed out, reported “context deadline exceeded”, with seemingly no rhyme or reason.

Output of run with `-debug`:

$ sudo trivy -d image docker.prod.skytap.com:5000/skytap/continuous_queries:b835ff37ac1b8304d57060e12b917561387dd2f0
2021-01-07T21:17:11.165Z	DEBUG	Severities: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
2021-01-07T21:17:14.423Z	DEBUG	cache dir:  /highland/.cache/trivy
2021-01-07T21:17:14.424Z	DEBUG	DB update was skipped because DB is the latest
2021-01-07T21:17:14.425Z	DEBUG	DB Schema: 1, Type: 1, UpdatedAt: 2021-01-07 12:25:41.888540058 +0000 UTC, NextUpdate: 2021-01-08 00:25:41.888539858 +0000 UTC, DownloadedAt: 2021-01-07 21:10:29.831187878 +0000 UTC
2021-01-07T21:17:15.448Z	DEBUG	Vulnerability type:  [os library]
2021-01-07T21:19:14.426Z	FATAL	error in image scan:
    github.com/aquasecurity/trivy/internal/artifact.run
        /home/circleci/project/internal/artifact/run.go:88
  - failed analysis:
    github.com/aquasecurity/trivy/pkg/scanner.Scanner.ScanArtifact
        /home/circleci/project/pkg/scanner/scan.go:97
  - analyze error:
    github.com/aquasecurity/fanal/artifact/image.Artifact.Inspect
        /go/pkg/mod/github.com/aquasecurity/fanal@v0.0.0-20201218050947-981a0510f9cb/artifact/image/image.go:50
  - timeout:
    github.com/aquasecurity/fanal/artifact/image.Artifact.inspect
        /go/pkg/mod/github.com/aquasecurity/fanal@v0.0.0-20201218050947-981a0510f9cb/artifact/image/image.go:90
  - context deadline exceeded

Output of `trivy -v`:

Version: 0.15.0

Additional details (base image name, container registry info…):

Note that other context deadline exceeded tickets exist, but none of them match the specific details of this error.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 10
Comments: 18

Most upvoted comments

Just in case google brings someone here, here’s how I solved:

Call trivy using the --timeout argument.

Example: --timeout 10m for 10 minutes (currently the default is 5 minutes).

+13

meleu on Sep 21, 2021

We’re facing the same issue for many repos now. Built from scratch, doesn’t matter 😞

dinvlad on Jan 12, 2021

I forgot to report back but it did help, at least in our case!

dinvlad on Jan 14, 2021

Doesn’t --timeout option fix the issue?

knqyf263 on Jan 14, 2021

@Alexc0007 Had the same issue as well, but even though I’m also using client / server mode, the --timeout 10m flag did have an impact on my side (v0.28.1), since it returned successful after 8 Minutes, while it terminated after 5 Minutes on previous runs without using the flag. The fact that it works after several runs with the same setting may be related to caching behaviour, since trivy seems to cache results for each docker layer individually. If you have say 2 layers, each of which takes 4 minutes scan time, the run potentially fails for the first time after exceeding 5m timeout, but may succeed on a subsequent run, since layer #1 has been already successfully cached on the first run.

tillkuhn on Jun 14, 2022