ort: ORT, GitLab Pipeline Gradle - DownloadException: Download failed for 'Maven:org.hamcrest:hamcrest-core:1.3'.

Hello

Recently I struggle with test over ORT scanner for sample Gradle project When I execute the analyzer, scanner locallyon my linux machine using ORT image I can sucessfully scan the dependancies but when I try to scan the same project from a gitlab pipeline repository over corporate proxy (proxy is provided in http_proxy and https_proxy variables) I obtain following warnings in the analyzer part:

16:28:35.287 [Analyzer-1] INFO  org.ossreviewtoolkit.analyzer.PackageManager - Resolving Gradle dependencies for '/builds/opensource/oss-compliance-sample-projects/gradle-01/build.gradle'...
Downloading https://services.gradle.org/distributions/gradle-6.6.1-bin.zip
.................................................................................................
16:28:59.696 [Analyzer-1] WARN  org.ossreviewtoolkit.analyzer.managers.utils.MavenSupport - Unable to find 'joda-time:joda-time:jar:2.2' in any of [https://repo.maven.apache.org/maven2, https://repo.maven.apache.org/maven2/].
16:29:00.015 [Analyzer-1] WARN  org.ossreviewtoolkit.analyzer.managers.utils.MavenSupport - Unable to find 'joda-time:joda-time:jar:sources:2.2' in any of [https://repo.maven.apache.org/maven2, https://repo.maven.apache.org/maven2/].
16:29:00.821 [Analyzer-1] WARN  org.ossreviewtoolkit.analyzer.managers.utils.MavenSupport - Unable to find 'junit:junit:jar:4.12' in any of [https://repo.maven.apache.org/maven2, https://repo.maven.apache.org/maven2/].
16:29:00.834 [Analyzer-1] WARN  org.ossreviewtoolkit.analyzer.managers.utils.MavenSupport - Unable to find 'junit:junit:jar:sources:4.12' in any of [https://repo.maven.apache.org/maven2, https://repo.maven.apache.org/maven2/].
16:29:00.868 [Analyzer-1] WARN  org.ossreviewtoolkit.analyzer.managers.utils.MavenSupport - Unable to find 'org.hamcrest:hamcrest-core:jar:1.3' in any of [https://repo.maven.apache.org/maven2, https://repo.maven.apache.org/maven2/].
16:29:00.878 [Analyzer-1] WARN  org.ossreviewtoolkit.analyzer.managers.utils.MavenSupport - Unable to find 'org.hamcrest:hamcrest-core:jar:sources:1.3' in any of [https://repo.maven.apache.org/maven2, https://repo.maven.apache.org/maven2/].
16:29:08.465 [FileStoreAttributeReader-2] WARN  org.eclipse.jgit.util.FS - locking FileBasedConfig[/root/.config/jgit/config] failed after 5 retries
16:29:08.493 [Analyzer-1] INFO  org.ossreviewtoolkit.analyzer.PackageManager - Resolving Gradle dependencies for 'build.gradle' took 33.199490799s.
Found 1 project(s) in total.
Writing analyzer result to '/builds/opensource/oss-compliance-sample-projects/gradle-01/ort/analyzer/analyzer-result.yml'.

scanner part with an ERROR at the end which fails the process:

16:31:11.249 [ScanCode-1] INFO  org.ossreviewtoolkit.downloader.Downloader - Trying to download source artifact for 'Maven:org.hamcrest:hamcrest-core:1.3' from ...
16:31:11.250 [ScanCode-1] ERROR org.ossreviewtoolkit.scanner.LocalScanner - Could not download 'Maven:org.hamcrest:hamcrest-core:1.3': DownloadException: Download failed for 'Maven:org.hamcrest:hamcrest-core:1.3'.
Suppressed: DownloadException: Unable to determine a revision to checkout.

Do you have any idea if this can be a proxy issue? If yes how could I pass the proxy settings?

PS: the same pipeline works ok for reular maven and nodejs projects

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 22 (13 by maintainers)

Most upvoted comments

With a curations.yml file

- id: "Maven:com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava"
  curations:
    comment: "no sources available, skipping"
    is_meta_data_only: true

the scans works ok

/opt/ort/bin/ort -c /root/.ort.conf --info analyze -i "$ORT_PROJECT_DIR" -o "$ORT_PROJECT_DIR/ort/analyzer" --package-curations-file $ORT_PROJECT_DIR/curations.yml --clearly-defined-curations

thank you Sebastian

woznik on Oct 30, 2020

I see that the dependency 9999.0-empty-to-avoid-conflict-with-guava in maven repository

Yes, but what you don’t see is a -sources.jar artifact, right? 😉 So there is no published sources artifact and no SCM information in the POM, hence the downloader cannot download the source code without further help.

Given that the binary JAR artifact is empty (and that’s also what the package description states), I’d simply create a curation for this artifact that sets is_meta_data_only to true.

Ping @MarcelBochtler, this is also of interest to you.

sschuberth on Oct 30, 2020

“Unable to determine a revision to checkout” DownloadException isthrown by ORT if the tool can clone the repository but is unable to find a tag for package version. Looking at https://github.com/hamcrest/JavaHamcrest/tags there are multiple tags with 1.3 in the name whihc may be the issues adding a curation.

You can easily test this by adding a VCS revision curation to your curations.yml

- id: "Maven:org.hamcrest:hamcrest-core:1.3"
    vcs:
      revision: "hamcrest-java-1.3"

tsteenbe on Sep 29, 2020