ort: Scanner freezes when using ClearlyDefined as a scan storage provider

When i run the scanner using clearlydefined with a file based storage, the scanner seem to hang.

I don’t know if its a problem using the clearlydefined api or if my configuration file is incorrect ?

Is there any flag to see more information from the scanner, like debug messages ?

The process doesn’t seem to do much

sudo sudo strace -f -e trace=network -s 10000 -p `pidof java`
strace: Process 110620 attached with 91 threads
[pid 110770] recvfrom(295, 
 <unfinished ...>
[pid 110767] sendto(295, "\27\3\3\0#\315K\17\366~\3\365\5\37\r\327.N.\274\215y>\264\10\355u\244\277\35K^{9\69\233\305i\"", 40, 0, NULL, 0) = 40
[pid 110767] sendto(295, "\27\3\3\0#\326?|j\201\320\224\257I\316\261\207\333(\\\311>\10\2217x1\255\227\223J\271\377,R\224!>\236\301", 40, 0, NULL, 0) = 40
[pid 110767] shutdown(295, SHUT_WR)     = 0
[pid 110767] shutdown(295, SHUT_RD)     = 0
[pid 110770] <... recvfrom resumed>"", 5, 0, NULL, NULL) = 0
[pid 110770] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_ACCERR, si_addr=0x7f231368e008} ---
[pid 110767] +++ exited with 0 +++
[pid 110770] +++ exited with 0 +++
[pid 110620] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_USER, si_pid=0, si_uid=0} ---
[pid 110620] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_USER, si_pid=0, si_uid=0} ---
[pid 110620] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_USER, si_pid=0, si_uid=0} ---
[pid 110620] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_USER, si_pid=0, si_uid=0} ---
[pid 110620] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_USER, si_pid=0, si_uid=0} ---
[pid 110620] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_USER, si_pid=0, si_uid=0} ---
[pid 110620] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_USER, si_pid=0, si_uid=0} ---
[pid 110620] --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_USER, si_pid=0, si_uid=0} ---

~/scripts/scan.sh 
20:06:35.331 [main] INFO  org.ossreviewtoolkit.model.config.OrtConfiguration - Using ORT configuration file '/ort-home/.ort/ort.conf'.
20:06:35.638 [main] INFO  org.ossreviewtoolkit.model.config.LicenseFilenamePatterns - Configuring the license file patterns.
________ _____________________
\_____  \\______   \__    ___/ the OSS Review Toolkit, version DOCKER-SNAPSHOT.
 /   |   \|       _/ |    |
/    |    \    |   \ |    |    Running 'scan' under Java 11.0.8 on Linux with
\_______  /____|_  / |____|    8 CPUs and a maximum of 7942 MiB of memory.
        \/       \/
Environment variables:
ORT_CONFIG_DIR = /ort-home/.ort
ORT_DATA_DIR = /ort-home/.ort
JAVA_HOME = /opt/java/openjdk
ANDROID_HOME = /opt/android-sdk
GOPATH = /go

20:06:35.682 [main] INFO  org.ossreviewtoolkit.scanner.ScanResultsStorage - Using file based storage with local directory '/ort-home/.ort/scan-results'.
20:06:35.689 [main] INFO  org.ossreviewtoolkit.scanner.ScanResultsStorage - Using ClearlyDefined storage with URL 'https://api.clearlydefined.io'.
20:06:35.694 [main] INFO  org.ossreviewtoolkit.scanner.ScanResultsStorage - Using composite storage with readers FileBasedStorage with XZCompressedLocalFileStorage backend, ClearlyDefinedStorage and writers FileBasedStorage with XZCompressedLocalFileStorage backend.
20:06:35.695 [main] INFO  org.ossreviewtoolkit.scanner.ScanResultsStorage - ScanResultStorage has been configured to composite[readers:[FileBasedStorage with XZCompressedLocalFileStorage backend, ClearlyDefinedStorage], writers:[FileBasedStorage with XZCompressedLocalFileStorage backend]].
Using scan storage 'composite[readers:[FileBasedStorage with XZCompressedLocalFileStorage backend, ClearlyDefinedStorage], writers:[FileBasedStorage with XZCompressedLocalFileStorage backend]]'.
Using scanner 'ScanCode'.
20:06:36.733 [main] INFO  org.ossreviewtoolkit.scanner.scanners.scancode.ScanCode - Searching scan results for 1 packages.
20:06:36.739 [main] INFO  org.ossreviewtoolkit.scanner.scanners.scancode.ScanCode - Searching scan results for 380 packages.
20:06:37.199 [DefaultDispatcher-worker-34] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/cavaliercoder/badio:v0.0.0-20160213150051-ce5280129e9e'.
20:06:37.199 [DefaultDispatcher-worker-9] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/anmitsu/go-shlex:v0.0.0-20161002113705-648efa622239'.
20:06:37.199 [DefaultDispatcher-worker-63] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.62.0'.
20:06:37.200 [DefaultDispatcher-worker-7] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/asaskevich/govalidator:v0.0.0-20200907205600-7a23bdc65eef'.
20:06:37.200 [DefaultDispatcher-worker-37] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/Masterminds/squirrel:v1.4.0'.
20:06:37.200 [DefaultDispatcher-worker-61] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go/storage:v1.6.0'.
20:06:37.200 [DefaultDispatcher-worker-13] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.65.0'.
20:06:37.200 [DefaultDispatcher-worker-41] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/Masterminds/goutils:v1.1.0'.
20:06:37.200 [DefaultDispatcher-worker-45] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/cavaliercoder/go-rpm:v0.0.0-20200122174316-8cb9fd9c31a8'.
20:06:37.200 [DefaultDispatcher-worker-38] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/alecthomas/jsonschema:v0.0.0-20200530073317-71f438968921'.
20:06:37.200 [DefaultDispatcher-worker-10] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/BurntSushi/toml:v0.3.1'.
20:06:37.200 [DefaultDispatcher-worker-59] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go/storage:v1.5.0'.
20:06:37.200 [DefaultDispatcher-worker-46] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::code.gitea.io/sdk/gitea:v0.13.0'.
20:06:37.200 [DefaultDispatcher-worker-60] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.76.0'.
20:06:37.200 [DefaultDispatcher-worker-43] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/Masterminds/sprig:v2.22.0+incompatible'.
20:06:37.201 [DefaultDispatcher-worker-48] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.34.0'.
20:06:37.200 [DefaultDispatcher-worker-18] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/asaskevich/govalidator:v0.0.0-20200428143746-21a406dcc535'.
20:06:37.201 [DefaultDispatcher-worker-24] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go/storage:v1.0.0'.
20:06:37.201 [DefaultDispatcher-worker-50] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.72.0'.
20:06:37.201 [DefaultDispatcher-worker-30] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go/storage:v1.10.0'.
20:06:37.201 [DefaultDispatcher-worker-33] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.26.0'.
20:06:37.198 [DefaultDispatcher-worker-35] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/cavaliercoder/go-cpio:v0.0.0-20180626203310-925f9528c45e'.
20:06:37.198 [DefaultDispatcher-worker-36] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/Masterminds/semver:v1.5.0'.
20:06:37.198 [DefaultDispatcher-worker-64] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go/storage:v1.8.0'.
20:06:37.198 [DefaultDispatcher-worker-62] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.74.0'.
20:06:37.210 [DefaultDispatcher-worker-8] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.57.0'.
20:06:37.225 [DefaultDispatcher-worker-34] INFO  org.ossreviewtoolkit.utils.OrtAuthenticator - Authenticator was successfully installed.
20:06:37.240 [DefaultDispatcher-worker-34] INFO  org.ossreviewtoolkit.utils.OrtProxySelector - Proxy selector was successfully installed.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by retrofit2.Platform (file:/opt/ort/lib/retrofit-2.9.0.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of retrofit2.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
20:06:37.446 [DefaultDispatcher-worker-30] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.54.0'.
20:06:37.446 [DefaultDispatcher-worker-34] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.44.3'.
20:06:37.446 [DefaultDispatcher-worker-35] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.46.3'.
20:06:37.447 [DefaultDispatcher-worker-30] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.44.2'.
20:06:37.447 [DefaultDispatcher-worker-34] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.44.1'.
20:06:37.447 [DefaultDispatcher-worker-35] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.38.0'.
20:06:37.446 [DefaultDispatcher-worker-61] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.50.0'.
20:06:37.447 [DefaultDispatcher-worker-34] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::github.com/cespare/xxhash/v2:v2.1.1'.
20:06:37.446 [DefaultDispatcher-worker-41] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.52.0'.
20:06:37.447 [DefaultDispatcher-worker-30] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.45.1'.
20:06:37.446 [DefaultDispatcher-worker-50] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.56.0'.
20:06:37.446 [DefaultDispatcher-worker-60] INFO  org.ossreviewtoolkit.scanner.storages.ClearlyDefinedStorage - Looking up results for 'GoMod::cloud.google.com/go:v0.53.0'.

cat ~/scripts/scan.sh 
#!/bin/bash

export CURRENT_DIRNAME=$(basename "$PWD")

sudo rm -rf ~/.ort/project/packaged/ort/scanner/native-scan-results

docker run -e ORT_CONFIG_DIR=/ort-home/.ort \
           -e ORT_DATA_DIR=/ort-home/.ort \
           -v $HOME:/ort-home \
           -v $(pwd):/project \
	   -v ~/.gitconfig:/root/.gitconfig \
           ort:latest --info scan -i /ort-home/.ort/project/$CURRENT_DIRNAME/ort/analyzer/analyzer-result.json -o /ort-home/.ort/project/$CURRENT_DIRNAME/ort/scanner

 cat ~/.ort/ort.conf
ort {
  scanner {
    storages {
      clearlyDefined {
        serverUrl = "https://api.clearlydefined.io"
      }
      fileBasedStorage {
        backend {
          localFileStorage {
            directory = "/ort-home/.ort/scan-results"
            compression = true
          }
        }
      }
    }

    storageReaders: [
       "fileBasedStorage", "clearlyDefined"
    ]

    storageWriters: [
      "fileBasedStorage"
    ]
  }
}