tern: Error if Tern calls Scancode-Toolkit

@Jeeppler Working on this today. Should be out EOD.

Thanks, @rnjudge, it works now with the branch you provided! 😃

Confirmed that there was a big change to license_detection reporting which was causing the error difference in v31 vs v32.

@Jeeppler @armintaenzertng Hopefully this fixes all the errors you are seeing. Please give it a try: https://github.com/rnjudge/tern/tree/issue-1202

I still have some parsing to fix from the scancode updates but if it fixes your issues I’d like to get a new release cut with the fixes.

I was able to reproduce the issue by using the example commands from the readme. That is,

python3 -m venv scanenv
cd scanenv/
source bin/activate
pip install tern scancode-toolkit
tern report -x scancode -i golang:1.12-alpine

Ok, working on this. What version of scancode are you running here? I can’t reproduce the error with 31.2.6 and my latest changes, but let me try upgrading to 32.0 to see if they changed their data model again.

UPDATE: I can reproduce with 32.0.4 so scancode dictionary key names have changed again. Sigh.

I was able to reproduce the issue by using the example commands from the readme. That is,

python3 -m venv scanenv
cd scanenv/
source bin/activate
pip install tern scancode-toolkit
tern report -x scancode -i golang:1.12-alpine

results in

Traceback (most recent call last):
  File "/home/armin/python_venvs/scanenv/bin/tern", line 8, in <module>
    sys.exit(main())
  File "/home/armin/python_venvs/scanenv/lib/python3.10/site-packages/tern/__main__.py", line 311, in main
    do_main(args)
  File "/home/armin/python_venvs/scanenv/lib/python3.10/site-packages/tern/__main__.py", line 123, in do_main
    crun.execute_image(args)
  File "/home/armin/python_venvs/scanenv/lib/python3.10/site-packages/tern/analyze/default/container/run.py", line 80, in execute_image
    cimage.default_analyze(full_image, args)
  File "/home/armin/python_venvs/scanenv/lib/python3.10/site-packages/tern/analyze/default/container/image.py", line 71, in default_analyze
    passthrough.run_extension_layer(image_obj.layers[0], options.extend,
  File "/home/armin/python_venvs/scanenv/lib/python3.10/site-packages/tern/analyze/passthrough.py", line 92, in run_extension_layer
    return mgr.driver.execute_layer(image_layer, redo)
  File "/home/armin/python_venvs/scanenv/lib/python3.10/site-packages/tern/extensions/scancode/executor.py", line 219, in execute_layer
    file_list, package_list = collect_layer_data(image_layer)
  File "/home/armin/python_venvs/scanenv/lib/python3.10/site-packages/tern/extensions/scancode/executor.py", line 162, in collect_layer_data
    files.append(get_scancode_file(f))
  File "/home/armin/python_venvs/scanenv/lib/python3.10/site-packages/tern/extensions/scancode/executor.py", line 61, in get_scancode_file
    if file_dict['licenses']:
KeyError: 'licenses'

Image:

image.zip

Tern command used:

tern report -f spdxjson -w "sechub-test-alpine.tar" -o "my_result.spdx.json" -x scancode

Alpine Dockerfile:

# SPDX-License-Identifier: MIT

FROM alpine:3.18

ENV DATA="/data"

COPY data "$DATA"

COPY shared/run.sh /run.sh
RUN chmod +x /run.sh

CMD ["/run.sh"]

Data is a folder with some source files with licenses annotations: https://github.com/mercedes-benz/sechub/tree/develop/sechub-pds-solutions/tern/tests/data

run.sh

#!/usr/bin/env sh
# SPDX-License-Identifier: MIT

while true
do
  echo "Press [CTRL+C] to stop.."
  sleep 120
done

In general, you can find all the test images/files for the containers here: https://github.com/mercedes-benz/sechub/tree/develop/sechub-pds-solutions/tern/tests.

@Jeeppler thanks. Can you provide the image you’re working with so I can reproduce? The alpine base image doesn’t fail.

Once I get confirmation from you that it’s fixed, I’ll cut a release.

@Jeeppler I still cannot install scanacode which is quite concerning for me wrt Tern’s integration with the library. I will try to setup a linux environment to debug on my old mac. Stay tuned.

@Jeeppler let me try to install the latest release of scancode. It’s hard for me to debug and fix this when I can’t install scancode 😕

@rnjudge It seems only some images produce this error, I wonder what do they have in common:

# tern report -x scancode -i docker-daemon:node:18.12.1 -o bitnami-node.18.12.1-prod.tern.$(date +"%d.%m.%Y").txt
2023-01-19 15:17:33,198 - DEBUG - __main__ - Starting...                                                                     [21/535]2023-01-19 15:17:33,198 - DEBUG - prep - Setting up...                                                                               2023-01-19 15:17:37,099 - DEBUG - rootfs - Running command: chmod +x /root/ternenv/lib/python3.10/site-packages/tern/tools/fs_hash.sh2023-01-19 15:17:37,101 - DEBUG - run - Starting analysis...                                                                         2023-01-19 15:17:37,102 - DEBUG - skopeo - Attempting to pull image "docker-daemon:node:18.12.1"                                     2023-01-19 15:17:37,102 - DEBUG - rootfs - Running command: skopeo copy docker-daemon:node:18.12.1 dir:/root/.tern/temp
2023-01-19 15:18:05,324 - DEBUG - rootfs - Running command: tar -tf /root/.tern/temp/58861e9c7a9b85fcb52553413d8725a7ae8d215349b6325cfdb9d0c38b7f5583                                                                                                                     2023-01-19 15:18:05,375 - DEBUG - rootfs - Running command: tar -x -f /root/.tern/temp/58861e9c7a9b85fcb52553413d8725a7ae8d215349b6325cfdb9d0c38b7f5583 -C /root/.tern/temp/1/contents                                                                                    2023-01-19 15:18:06,022 - DEBUG - rootfs - Running command: /root/ternenv/lib/python3.10/site-packages/tern/tools/fs_hash.sh /root/.tern/temp/1/contents                                                                                                                  2023-01-19 15:18:18,438 - DEBUG - rootfs - Running command: tar -tf /root/.tern/temp/8a8dfd8ce7a77963ad1d0d8009a79f22f71c674bcf6198699bfd20ce868ff27b                                                                                                                     2023-01-19 15:18:18,601 - DEBUG - rootfs - Running command: tar -x -f /root/.tern/temp/8a8dfd8ce7a77963ad1d0d8009a79f22f71c674bcf6198699bfd20ce868ff27b -C /root/.tern/temp/2/contents
2023-01-19 15:18:21,603 - DEBUG - rootfs - Running command: /root/ternenv/lib/python3.10/site-packages/tern/tools/fs_hash.sh /root/.tern/temp/2/contents
2023-01-19 15:19:33,438 - DEBUG - common - Loading packages from cache: layer "9d22d5e891"
2023-01-19 15:19:33,932 - DEBUG - common - Loading files from cache: layer "9d22d5e891"
2023-01-19 15:19:35,235 - DEBUG - rootfs - Running command: /root/ternenv/bin/scancode -ilpcu --quiet --timeout 300 -n 1 --json - /root/.tern/temp/1/contents
Traceback (most recent call last):
  File "/root/ternenv/bin/tern", line 8, in <module>
    sys.exit(main())
  File "/root/ternenv/lib/python3.10/site-packages/tern/__main__.py", line 311, in main
    do_main(args)
  File "/root/ternenv/lib/python3.10/site-packages/tern/__main__.py", line 123, in do_main
    crun.execute_image(args)
  File "/root/ternenv/lib/python3.10/site-packages/tern/analyze/default/container/run.py", line 80, in execute_image
    cimage.default_analyze(full_image, args)
  File "/root/ternenv/lib/python3.10/site-packages/tern/analyze/default/container/image.py", line 70, in default_analyze
    passthrough.run_extension_layer(image_obj.layers[0], options.extend,
  File "/root/ternenv/lib/python3.10/site-packages/tern/analyze/passthrough.py", line 92, in run_extension_layer
    return mgr.driver.execute_layer(image_layer, redo)
  File "/root/ternenv/lib/python3.10/site-packages/tern/extensions/scancode/executor.py", line 219, in execute_layer
    file_list, package_list = collect_layer_data(image_layer)
  File "/root/ternenv/lib/python3.10/site-packages/tern/extensions/scancode/executor.py", line 162, in collect_layer_data
    files.append(get_scancode_file(f))
  File "/root/ternenv/lib/python3.10/site-packages/tern/extensions/scancode/executor.py", line 65, in get_scancode_file
    fd.copyrights = [c['value'] for c in file_dict['copyrights']]
  File "/root/ternenv/lib/python3.10/site-packages/tern/extensions/scancode/executor.py", line 65, in <listcomp>
    fd.copyrights = [c['value'] for c in file_dict['copyrights']]
KeyError: 'value'

# scancode --version; tern --version; python --version
ScanCode version: 31.2.4
ScanCode Output Format version: 2.0.0
SPDX License list version: 3.18
Tern version 2.11.0
   python version = 3.10.6 (main, Nov 14 2022, 16:10:14)
Python 3.10.6

@Jeeppler I have not forgot about this. The issue I am trying to workaround right now is simply installing scancode on my VM on my Mac M1 so I can debug and fix. I opened an issue for it with Scancode here. What I’ve been told is that they are re-releasing Scancode with fixes for some of the dependent libraries so I’ll have to wait for that…