coreutils: cksum: failing GNU tests

The latest version of GNU’s tests add tests/misc/cksum, tests/misc/cksum-a, tests/misc/cksum-c, and tests/misc/sm3sum, all of which currently fail. This will track the respective problems from each.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 16 (14 by maintainers)

Commits related to this issue

Most upvoted comments

Looks like test/misc/b2sum is also failing, both because hashsum doesn’t support -l/--length, and cksum doesn’t support -a blake2b.

I’m going to try implementing this.

If you all decide to do something like --bytes, it might be reasonable to add a similar option upstream on b3sum, so that there’s at least something we could tell people to do that’ll work (or give a clear error message) everywhere?

@sergeevabc b3sum normalizes backslashes to forward slashes so that b3sum --check on a Unix machine has a reasonably good chance of accepting a b3sum output from a Windows machine. This also has consequences for e.g. Unicode/OsStr handling. We have a doc about it: https://github.com/BLAKE3-team/BLAKE3/blob/master/b3sum/what_does_check_do.md

My understanding of forward slashes on Windows is that they’re always(?) valid in place of backslashes at the API level, even though some higher level applications choke on them (particularly cmd.exe). So even though there’s code in b3sum to convert backslashes to forward slashes, there is no conversion in the other direction; after maybe unescaping newlines, --check opens filepaths verbatim.

Another mess here:

$ coreutils.exe 
coreutils 0.0.30 (multi-call binary)

$ coreutils.exe b3sum coreutils.exe
0755ad0a3492592f566d90acb6df93c4087d3bedda4915912dba036d815d0605 *coreutils.exe

BUT

$ coreutils.exe b3sum C:\coreutils.exe
\0755ad0a3492592f566d90acb6df93c4087d3bedda4915912dba036d815d0605 *C:\\coreutils.exe
^ WTF???

$ b3sum.exe C:\coreutils.exe
0755ad0a3492592f566d90acb6df93c4087d3bedda4915912dba036d815d0605  C:/coreutils.exe

Current coreutils b3sum generates hash *filename, but the original implementation by @oconnor663 uses a second space instead of an asterisk and refuses to check hashes if such character is missing as follows

$ type B3SUMS
79b4c2c09d6f3ac0beee530cb69262f81ce303144620728e8c98044dd543ceb2 *out1.png
7febfd4fd14642e0f70c50dd096ec684f6d05afce5d606aeb157ec93a96890ed *out2.png
17a7f68466e2b437042a5b2ebad2b3abde27f2b83d0c57358a2cbfbb31cf05ed *out3.png

$ b3sum -c B3SUMS
b3sum: Invalid space
b3sum: WARNING: 3 computed checksums did NOT match

We need to either discuss it or bring this part of coreutils into line.

P.S. I already have a dozen BLAKE3 implementations, mostly downloaded from Github, and the output format is different between them. The original one also refuses to hash lists with CRLF line breaks, typical for Windows (such lists are generated, for example, by RapidCRC). Obviously, this UX mess creates a difficulty for replacing SHA256.

Click to broaden your horizons even further (and to see a SFW picture!)…

2024-0105-0309 crayfish swan pike

When, among partners, concord there is not. Successful issues scarce are got And the result is loss, disaster and repining.

A crayfish, swan and pike combining. Resolve to draw a cart and freight; In harness soon, their efforts ne’er abate. However much they work, the load to stir refuses. It seems to be perverse with selfwill vast endowed; The swan makes upward for a cloud. The crayfish falls behind, the pike the river uses; To judge of each one’s merit lies beyond my will; I know the cart remains there, still.

— Ivan Krylov, “Crayfish, Swan and Pike” (1814), translated from Russian by Charles Fillingham Coxwell.

Interesting! That does make it clearer as to why b3sum is different. Maybe a solution could be to make a split API. So b2sum (and the other ones GNU supports) would accept a --bits option and -l would work but be deprecated/hidden. b3sum would accept a --bytes option and using the wrong argument would result in an error. However, a downside to this is that we break compat with both GNU and b3sum, although we could still accept the length argument.

Yeah, the inconsistency blows. Here are some scattered thoughts about why it ended up the way it is:

  • Unfortunately, different BLAKE2 interfaces have been inconsistent for a long time. CLIs and written naming conventions tend to use bits, but Python’s hashlib.blake2b API takes the digest_size parameter in bytes. Similarly, C APIs usually take a size_t somewhere that counts bytes.

  • Measuring BLAKE2 output in bits makes sense from a security perspective. The security properties of BLAKE2b-512 and BLAKE2b-256 mirror those of SHA-512 and SHA-256. However, this relationship relies somewhat on BLAKE2b not allowing output sizes larger than its internal state size. For example, b2sum -l 1024 will just error out. The same doesn’t apply to BLAKE3, though, which has a built-in XOF that’ll give you as many output bytes as you like. “BLAKE3-512” (if we were to call it that) does not provide more security than the default 256-bit BLAKE3. I find this is a common source of confusion, so I discourage naming BLAKE3 instances in terms of bits, even though it can make sense on the small end.

  • Another way BLAKE2 is similar to SHA-2 is that BLAKE2b-512 and BLAKE2b-256 are independent (“domain separated”) hash functions. Changing the output length gives you a completely unrecognizable hash. This is a natural design for those functions, since they can mix the output length into state initialization one way or another. But for an XOF like BLAKE3, it’s pretty common for the output length to be “who knows, we’re gonna use this as a stream cipher”, so domain-separating outputs of different lengths isn’t always possible. Rather than having two modes for this, as BLAKE2X did, in BLAKE3 shorter outputs are always prefixes of longer ones. This is another reason why I discourage bit-counting names like “BLAKE3-512”, to avoid implying that variants with different names are independent of each other.

Which is all to say, that when there’s a choice between counting bits and counting bytes, BLAKE3 pretty strongly prefers to count bytes. It keeps CLIs and library APIs consistent with each other, and it avoids some common points of confusion. But sometimes there isn’t a choice, like our situation here, and then we suffer some inconsistency.

Ugh, this is annoying: b3sum was added in https://github.com/uutils/coreutils/pull/3108, and its corresponding upstream binary defines -l/--length in bytes, whereas GNU Coreutils b2sum defines -l/--length in bits.

There are three options here:

  • Define -l/--length as bytes to match upstream b3sum. Given that compatibility with GNU Coreutils is being prioritised (per this issue), I think that would not be considered acceptable as b2sum would then never pass GNU tests.
  • Define -l/--length as bits to match GNU Coreutils. I really dislike this option, for two reasons:
    • --length being bits means it needs to be a multiple of 8, so 7/8 of values that upstream b3sum accepts will be rejected here. That’s not so bad, it makes the difference discoverable.
    • In the cases where an upstream length is a multiple of 8, b3sum here would output 1/8 of the length, e.g. instead of a 16-byte hash the user gets a 16-bit hash (2 bytes). This is a TERRIBLE silent security issue.
  • Define -l/--length as bits when used with b2sum or cksum, and bytes when used with b3sum. This is awful for UX; in particular, it is undefined how hashsum should treat it. The least-bad option might be to just special-case the byte interpretation to b3sum, and not allow it to be used with -a blake3 on hashsum (otherwise hashsum would almost certainly end up conflicting with cksum if the latter adds BLAKE3 support).

We have a somewhat-similar issue in that hashsum itself defines a separate --bits flag for its SHA3 and SHAKE support, which is effectively duplicative of GNU Coreutils -l/--length. I’m not going to try solve that here.

Investigating the codebase a bit further, I see that both md5sum and b2sum are implemented via hashsum. This suggests we should make cksum also be implemented by hashsum. I’ve checked off above all the options currently implemented by hashsum.