coreutils: cksum: failing GNU tests
The latest version of GNU’s tests add tests/misc/cksum, tests/misc/cksum-a, tests/misc/cksum-c, and tests/misc/sm3sum, all of which currently fail. This will track the respective problems from each.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 16 (14 by maintainers)
Commits related to this issue
- hashsum: Refactor `uu_app` to isolate non-"GNU Coreutils" options Several binaries have been added to `hashsum` that have never been part of GNU Coreutils: - `sha3*sum` (uutils/coreutils#869) - `sha... — committed to str4d/coreutils by str4d 5 years ago
- hashsum: Refactor `uu_app` to isolate non-"GNU Coreutils" options Several binaries have been added to `hashsum` that have never been part of GNU Coreutils: - `sha3*sum` (uutils/coreutils#869) - `sha... — committed to str4d/coreutils by str4d 5 years ago
- hashsum: Refactor `uu_app` to isolate non-"GNU Coreutils" options Several binaries have been added to `hashsum` that have never been part of GNU Coreutils: - `sha3*sum` (uutils/coreutils#869) - `sha... — committed to str4d/coreutils by str4d 5 years ago
- hashsum: Refactor `uu_app` to isolate non-"GNU Coreutils" options Several binaries have been added to `hashsum` that have never been part of GNU Coreutils: - `sha3*sum` (uutils/coreutils#869) - `sha... — committed to str4d/coreutils by str4d 5 years ago
Looks like
test/misc/b2sumis also failing, both becausehashsumdoesn’t support-l/--length, andcksumdoesn’t support-a blake2b.I’m going to try implementing this.
If you all decide to do something like
--bytes, it might be reasonable to add a similar option upstream onb3sum, so that there’s at least something we could tell people to do that’ll work (or give a clear error message) everywhere?@sergeevabc
b3sumnormalizes backslashes to forward slashes so thatb3sum --checkon a Unix machine has a reasonably good chance of accepting ab3sumoutput from a Windows machine. This also has consequences for e.g. Unicode/OsStrhandling. We have a doc about it: https://github.com/BLAKE3-team/BLAKE3/blob/master/b3sum/what_does_check_do.mdMy understanding of forward slashes on Windows is that they’re always(?) valid in place of backslashes at the API level, even though some higher level applications choke on them (particularly
cmd.exe). So even though there’s code inb3sumto convert backslashes to forward slashes, there is no conversion in the other direction; after maybe unescaping newlines,--checkopens filepaths verbatim.Another mess here:
Current
coreutils b3sumgenerateshash *filename, but the original implementation by @oconnor663 uses a second space instead of an asterisk and refuses to check hashes if such character is missing as followsWe need to either discuss it or bring this part of
coreutilsinto line.P.S. I already have a dozen BLAKE3 implementations, mostly downloaded from Github, and the output format is different between them. The original one also refuses to hash lists with CRLF line breaks, typical for Windows (such lists are generated, for example, by RapidCRC). Obviously, this UX mess creates a difficulty for replacing SHA256.
Click to broaden your horizons even further (and to see a SFW picture!)…
When, among partners, concord there is not. Successful issues scarce are got And the result is loss, disaster and repining.
A crayfish, swan and pike combining. Resolve to draw a cart and freight; In harness soon, their efforts ne’er abate. However much they work, the load to stir refuses. It seems to be perverse with selfwill vast endowed; The swan makes upward for a cloud. The crayfish falls behind, the pike the river uses; To judge of each one’s merit lies beyond my will; I know the cart remains there, still.
— Ivan Krylov, “Crayfish, Swan and Pike” (1814), translated from Russian by Charles Fillingham Coxwell.
Interesting! That does make it clearer as to why b3sum is different. Maybe a solution could be to make a split API. So b2sum (and the other ones GNU supports) would accept a
--bitsoption and-lwould work but be deprecated/hidden. b3sum would accept a--bytesoption and using the wrong argument would result in an error. However, a downside to this is that we break compat with both GNU and b3sum, although we could still accept the length argument.Yeah, the inconsistency blows. Here are some scattered thoughts about why it ended up the way it is:
Unfortunately, different BLAKE2 interfaces have been inconsistent for a long time. CLIs and written naming conventions tend to use bits, but Python’s
hashlib.blake2bAPI takes thedigest_sizeparameter in bytes. Similarly, C APIs usually take asize_tsomewhere that counts bytes.Measuring BLAKE2 output in bits makes sense from a security perspective. The security properties of BLAKE2b-512 and BLAKE2b-256 mirror those of SHA-512 and SHA-256. However, this relationship relies somewhat on BLAKE2b not allowing output sizes larger than its internal state size. For example,
b2sum -l 1024will just error out. The same doesn’t apply to BLAKE3, though, which has a built-in XOF that’ll give you as many output bytes as you like. “BLAKE3-512” (if we were to call it that) does not provide more security than the default 256-bit BLAKE3. I find this is a common source of confusion, so I discourage naming BLAKE3 instances in terms of bits, even though it can make sense on the small end.Another way BLAKE2 is similar to SHA-2 is that BLAKE2b-512 and BLAKE2b-256 are independent (“domain separated”) hash functions. Changing the output length gives you a completely unrecognizable hash. This is a natural design for those functions, since they can mix the output length into state initialization one way or another. But for an XOF like BLAKE3, it’s pretty common for the output length to be “who knows, we’re gonna use this as a stream cipher”, so domain-separating outputs of different lengths isn’t always possible. Rather than having two modes for this, as BLAKE2X did, in BLAKE3 shorter outputs are always prefixes of longer ones. This is another reason why I discourage bit-counting names like “BLAKE3-512”, to avoid implying that variants with different names are independent of each other.
Which is all to say, that when there’s a choice between counting bits and counting bytes, BLAKE3 pretty strongly prefers to count bytes. It keeps CLIs and library APIs consistent with each other, and it avoids some common points of confusion. But sometimes there isn’t a choice, like our situation here, and then we suffer some inconsistency.
Ugh, this is annoying:
b3sumwas added in https://github.com/uutils/coreutils/pull/3108, and its corresponding upstream binary defines-l/--lengthin bytes, whereas GNU Coreutilsb2sumdefines-l/--lengthin bits.There are three options here:
-l/--lengthas bytes to match upstreamb3sum. Given that compatibility with GNU Coreutils is being prioritised (per this issue), I think that would not be considered acceptable asb2sumwould then never pass GNU tests.-l/--lengthas bits to match GNU Coreutils. I really dislike this option, for two reasons:--lengthbeing bits means it needs to be a multiple of 8, so 7/8 of values that upstreamb3sumaccepts will be rejected here. That’s not so bad, it makes the difference discoverable.b3sumhere would output 1/8 of the length, e.g. instead of a 16-byte hash the user gets a 16-bit hash (2 bytes). This is a TERRIBLE silent security issue.-l/--lengthas bits when used withb2sumorcksum, and bytes when used withb3sum. This is awful for UX; in particular, it is undefined howhashsumshould treat it. The least-bad option might be to just special-case the byte interpretation tob3sum, and not allow it to be used with-a blake3onhashsum(otherwisehashsumwould almost certainly end up conflicting withcksumif the latter adds BLAKE3 support).We have a somewhat-similar issue in that
hashsumitself defines a separate--bitsflag for its SHA3 and SHAKE support, which is effectively duplicative of GNU Coreutils-l/--length. I’m not going to try solve that here.Investigating the codebase a bit further, I see that both
md5sumandb2sumare implemented viahashsum. This suggests we should makecksumalso be implemented byhashsum. I’ve checked off above all the options currently implemented byhashsum.