trivy: trivy image scan suddenly timing out

https://status.maven.org/incidents/mblzqd69jjpz this looks relevant to the issues people are seeing here

Would it be possible to add a flag to just switch to offline mode after 3 attempts and throw a warning in the end?

I think any flag that results in inconsistent behavior isn’t a good idea. A given invocation of trivy should always produce the same result or it should fail.

Thanks all for your patience. We managed to include the improvement in v0.37.0 🎉🎉 We’re grateful for everyone who helped make it happen.

Trivy no longer relies on the third-party API and instead uses the pre-built database for Java scanning. The Java index database will be downloaded when JAR files are found. The scan results should be almost the same as before. You don’t have to use --offline-scan anymore. You can see more detail below. https://github.com/aquasecurity/trivy/discussions/3518

It can be used even in air-gapped environments💯 https://aquasecurity.github.io/trivy/v0.37/docs/advanced/air-gap/#download-the-java-index-database1

guess that the mirror could help with that, but it’s a huge effort, since you have to keep it synced and you will want to sync only your deps, not the entire repo.

It’s quite common to have a local mirror of maven central on your network (e.g. Nexus). It’s not required for trivy to mirror Maven central itself - it really should not. What’s needed is a way to decouple trivy from search.maven.org and point trivy to other mirrors.

@knqyf263 please urgently update the trivy-action as well with this new version. This is a big breaking change in our pipeline since you release trivy v0.37.2

I created already a PR for it and wait for approval so that the pipeline is running. 😆

Is that intended (or have I misconfigured something)?

Good question. Yes, we’re aware of that and it is intended for now as most users are running Trivy in standalone mode. We needed to get the Java scan working as soon as possible, so we put client/server aside. Improving client/server will be the next task.

Can you try --offline-scan then? It doesn’t try to connect to search.maven.org.

FYI, the intermittent timeout errors from search.maven.org has just been resolved, see their update https://status.maven.org/incidents/mblzqd69jjpz.

@bpfoster Thanks. We started working on that. https://github.com/aquasecurity/trivy-java-db

@candrews We’re not maintainers of Grype. Seems like they parse pom.properties like Trivy also does at first, but we’re not sure.

@knqyf263

@WhyJee --debug helps. Many JAR files don’t correctly define the identity. The warning will overwhelm the output. That is why they are debugging messages.

As I wrote in my comment almost all JARs in my project provide such information. However, if --debug shows that JARs that’s helpful. I will try it and check my project again.

Instead of deactivating the search would it possible to cache the search results like the database? The SHA-1 digest should never change. Trivy should only call search.maven.org if there is a new JAR without a cached SHA-1 digest. That would also ease the rate limit problem.

If your jar file doesn’t have information about GroupID, ArtifactID and Version in pom.properties or MANIFEST.MF then: We use maven repository to try to get GroupID, ArtifactID and Version by SHA-1 digest or get GroupID by artifactID(artifactID got from file name)

I don’t this is 100% accurate, it is also used to complement the db, docs state in bold

And I just confirmed a couple of CVE that are not reported when running with --offline-scan.

Imo --offline-scan involves some risk.

What happens if we set --offline-scan? The documentation is not very clear about it. If it stops to refresh the CVE database, this isn’t the best constant solution.

We are probably facing a similar issue — starting from today (the first fail was at Jan 12, 2023, 11:30 AM GMT), the GitLab pipelines started to fail.

• Trivy version was not changed, remains 0.36.1.
• Same configuration was successfully working in around 2-3.5 minutes.
• First it was a timeout issue.
• I increased the timeout to 10m and then to 30m.
• Now it’s failing on log4j or other artifacts like this, after 7-25 minutes of execution:

$ trivy --cache-dir .trivycache/ image --exit-code 0 --no-progress --format template --template "@/contrib/junit.tpl" --output "$CI_PROJECT_DIR/container-scan-junit.xml" "$FULL_IMAGE_NAME" --timeout 30m0s
2023-01-12T18:01:31.130Z	INFO	Need to update DB
2023-01-12T18:01:31.130Z	INFO	DB Repository: ghcr.io/aquasecurity/trivy-db
2023-01-12T18:01:31.130Z	INFO	Downloading DB...
2023-01-12T18:01:33.467Z	INFO	Vulnerability scanning is enabled
2023-01-12T18:01:33.467Z	INFO	Secret scanning is enabled
2023-01-12T18:01:33.467Z	INFO	If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2023-01-12T18:01:33.467Z	INFO	Please see also https://aquasecurity.github.io/trivy/v0.36/docs/secret/scanning/#recommendation for faster secret detection
2023-01-12T18:06:55.053Z	FATAL	image scan error: scan error: scan failed: failed analysis: analyze error: failed to analyze layer:  : walk error: failed to process the file: failed to analyze file: failed to analyze app/BOOT-INF/lib/log4j-core-2.19.0.jar: unable to open app/BOOT-INF/lib/log4j-core-2.19.0.jar: failed to open: unable to read the file: read tcp 172.17.0.2:47734->3.5.139.107:443: read: connection reset by peer
Uploading artifacts for failed job

Setting --offline-scan option is helping, but I am not sure whether it’s a good long-term solution.

Hello,

maybe it is related to search.maven.org: 2023-01-12T12:55:15.461Z DEBUG request failed {"error": "Get \"https://search.maven.org/solrsearch/select?q=1%3A%22185ae7422deb18aa14464b18cf9ce997d4710a57%22&rows=1&wt=json\": net/http: TLS handshake timeout", "method": "GET", "url": "https://search.maven.org/solrsearch/select?q=1%3A%22185ae7422deb18aa14464b18cf9ce997d4710a57%22&rows=1&wt=json"}

Can someone explain how Trivy checks if a given image is a Java image and takes the decision to invoke the Maven endpoint which seems to be going down this last week? Also…

Agreed on this. The true solution would be for search maven to be stable or dependency on it removed(not recommended).

Why is it not recommended to remove the Maven dependency? (Truly out of curiosity, does it plays a great part into Java vulnerabilities?)

Correct me if I am wrong. I think we want to keep the maven search functionality as needed as mentioned above for jar identification method sake. Removing the maven search dependency doesn’t really help unless there is a better way for jar identification as durcon mentioned above.

@DmitriyLewen What is search.maven.org used for?

If your jar file doesn’t have information about GroupID, ArtifactID and Version in pom.properties or MANIFEST.MF then: We use maven repository to try to get GroupID, ArtifactID and Version by SHA-1 digest or get GroupID by artifactID(artifactID got from file name)

I checked for a given jar file, all the above details are present in manifest.mf and pom.properties Then why Trivy tried to connect to search.maven.org to retrieve these details and hitting 504 I think such errors need to be gracefully handled (with warnings perhaps) without erroring out completely and impacting the execution.

One comment regarding this issue. The logic to try to get missing properties from maven is a good solution, but shall perhaps be considered as a workaround to a guilty software not properly identifying itself. Thus, would it be possible to have an option to activate a warning on each jar missing identification ? It could help to raise issue against each of these software and ensure that in the future every opensource component has set appropriate identification.

One note to others looking at this ticket. We’re predominantly a Ruby shop (and cRuby at that) but were getting the same timeouts described here. It turns out that the concurrent-ruby gem includes a jar file in its install, and that jar was causing us to hit search.maven.org even though everything else being scanned was not Java. So, be sure to turn on -d debug output before deciding this can’t be what’s causing your problem.

If you run with --offline-scan then you even get additional features 😉

In addition, if you want to scan Java dependencies such as JAR and pom.xml, you need to specify --offline-scan since Trivy tries to issue API requests for scanning Java applications by default.

https://aquasecurity.github.io/trivy/v0.36/docs/advanced/air-gap/

What about people using “skip-update” in an off-line environment?

db:
  # We are air-gapped
  skip-update: true 

Edit: Oh… Apparently we now also need to download a separate “Java index database”…

https://aquasecurity.github.io/trivy/v0.37/docs/advanced/air-gap/#download-the-java-index-database1

v0.37.1 works for us

Thankyou.

v0.37.0 has a critical issue. Please try v0.37.1 instead. https://github.com/aquasecurity/trivy/releases/tag/v0.37.1

@13CSherman I think that would be a great feature.

We are also facing trivy image scan error due to timeout, below is the trace. Please let us know if we have any resolution for the same. We are blocked with this issue.

All info is in the thread above.

@wenqiglantz unfortunately maven still has problems - https://status.maven.org/incidents/lfdx0m1kq3qt

@bpfoster Thanks for the info. Do you know a way to convert the Lucene index to a format that can be easily parsed such as XML, JSON, etc.?

@knqyf263 The maven indexer project contains a (relatively) straightforward example of how to programmatically read the index (in Java). The example prints out fields to stdout but the same approach could be used to created a formatted file.

https://github.com/apache/maven-indexer/blob/a7aeb9f2fe34e6c9f5222bcc2e31f43442cdf99c/indexer-examples/indexer-examples-basic/src/main/java/org/apache/maven/indexer/examples/BasicUsageExample.java#L198-L217

hello @kumarsandeep590 This is problem with maven repository. You can use the offline-scan flag, but there are some risks to doing so- https://github.com/aquasecurity/trivy/issues/3421#issuecomment-1381514618

@durcon take a look on #3427 we are talking about it here.

In a lot of cases there is also the pom.xml in den JAR.

I’m not sure there is resource with a list of jars that we can use to save to the database.

That’s why we use maven repository to analyze similar jars.

Central does provide their index, but it’s pretty big.

Is it possible to reduce / configure the query timeout (20s seems to be too high) for external sources like maven.org? Also, it would be nice to skip query the external source if n-tries failed before for an current scan process.

A quick request to https://search.maven.org from a browser gives you a 504 Gateway Timeout error When the website was still available but the searches were already failing yesterday, I saw a banner on top saying their new search repository will be at https://central.sonatype.dev/ So Maven may have moved the search services to this new domain which, as I’ve tried a few simple searches on it, seems to work fine.

seems like search.maven.org is still unstable or has a more aggressive rate limiting. After some requests (where the response is really fast) I get “504 Gateway Timeout”.

What is search.maven.org used for? Only for checking whether an update fixing a vulnerability of maven dependency is available?

@dmitry-weirdo Please see my comment above. https://github.com/aquasecurity/trivy/issues/3421#issuecomment-1380380681

Ref. https://aquasecurity.github.io/trivy/v0.36/docs/references/troubleshooting/#maven-rate-limiting-inconsistent-jar-vulnerability-reporting

Is everyone scanning images with JAR files? If so, maven might be relevant like @dusik66 pointed out. --offline-scan may help until the server is back.

Seeing the same issue with Trivy 0.32.1.

Hello, I am having the same issue there. Worked perfectly fine this morning until the Vulnerability DB updates:

trivy -v
Version: 0.33.0
Vulnerability DB:
  Version: 2
  UpdatedAt: 2023-01-12 06:07:03.840852771 +0000 UTC
  NextUpdate: 2023-01-12 12:07:03.840852271 +0000 UTC
  DownloadedAt: 2023-01-12 06:41:12.347924267 +0000 UTC