scorecard: Binary-Artifacts false positives

As far as I can tell, the Binary-Artifacts check was supposed to catch executables people can run unintentionally after cloning repositories but in reality it seems to just look for extensions and, for example, flags projects using binary files for testing purposes. (The check would be even more noisy if it ignored extensions and searched for magic numbers by analogy with file because it would effectively penalize projects for keeping, for example, regression tests generated by fuzz targets (a lot of which look like executables or, maybe, even kind of executables if ELF files are in seed corpora). As an example, below is what the Binary-Artifacts says about systemd:

{
  "date": "2021-11-11",
  "repo": {
    "name": "github.com/systemd/systemd",
    "commit": "9cc615460830afdb51ad23e594906bbe60a3b25a"
  },
  "scorecard": {
    "version": "3.1.1-57-g8da30e6",
    "commit": "8da30e63afbc62e25c2c1252003aed9d34c3d04d"
  },
  "score": 0.0,
  "checks": [
    {
      "details": [
        "Warn: binary detected: test/dmidecode-dumps/HP-Z600.bin",
        "Warn: binary detected: test/dmidecode-dumps/Lenovo-ThinkPad-X280.bin",
        "Warn: binary detected: test/dmidecode-dumps/Lenovo-Thinkcentre-m720s.bin"
      ],
      "score": 0,
      "reason": "binaries present in source code",
      "name": "Binary-Artifacts",
      "documentation": {
        "url": "https://github.com/ossf/scorecard/blob/8da30e63afbc62e25c2c1252003aed9d34c3d04d/docs/checks.md#binary-artifacts",
        "short": "Determines if the project has generated executable (binary) artifacts in the source repository."
      }
    }
  ],
  "metadata": null
}

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 21 (20 by maintainers)

Commits related to this issue

Most upvoted comments

I don’t think scorecard should complain about tests so I think I’d keep the issue open

shall we close this issue?

Would you like to have the list hardcoded, or given as option to scorecard? An option allows users to run it with the convention for the language they use, if there is any; and with better knowledge of the repo.

I think by default scorecard should produce meaningful metrics without any additional options to let people run it against different repositories written in different languages. Options would probably help if it was necessary to relax or tighten the check on a case-by-case basic should the need arise.