botan: Tests segfault on i386

On Alpine Linux Edge, we noticed that the botan test suite segfaults in the whirlpool.cpp code on i386.

The relevant part of the backtrace looks as follows:

Threefish-512/OCB(32) ran 41 tests in 0.12 msec all ok
TripleDES/EAX ran 818 tests in 3.23 msec all ok
Twofish/EAX ran 1602 tests in 4.60 msec all ok

Thread 4 "botan-test" received signal SIGSEGV, Segmentation fault.
[Switching to LWP 1689]
0xf7a4b2b5 in Botan::rotr<32u, unsigned long long> (input=<optimized out>) at build/include/botan/internal/rotate.h:36
36	   return static_cast<T>((input >> ROT) | (input << (8 * sizeof(T) - ROT)));
(gdb) bt
#0  0xf7a4b2b5 in Botan::rotr<32u, unsigned long long> (input=<optimized out>) at build/include/botan/internal/rotate.h:36
#1  Botan::Whirlpool::compress_n (this=0xf72756c0, in=0xf7ec1080 "\200", blocks=1) at src/lib/hash/whirlpool/whirlpool.cpp:114
#2  0xf7a275da in Botan::MDx_HashFunction::final_result (this=0xf72756c0, output=0xf7ec1200 "")
    at src/lib/hash/mdx_hash/mdx_hash.cpp:102
#3  0x56755a64 in Botan::Buffered_Computation::final<std::vector<unsigned char, Botan::secure_allocator<unsigned char> > > (
    this=0xf72756c0) at build/include/botan/buf_comp.h:79
#4  Botan_Tests::(anonymous namespace)::Hash_Function_Tests::run_one_test (this=<optimized out>, algo=..., vars=...)
    at src/tests/test_hash.cpp:95

So it seems to segfault in Botan::rotr:

https://github.com/randombit/botan/blob/bfd60f09090620158d1e9fdaafb77b2bf97d1bd4/src/lib/utils/rotate.h#L27-L37

The invocation of Botan::rotr is rotr<32>(WHIRL_S[get_byte<4>(K4)]) from:

https://github.com/randombit/botan/blob/bfd60f09090620158d1e9fdaafb77b2bf97d1bd4/src/lib/hash/whirlpool/whirlpool.cpp#L114

Dockerfile to reproduce the segfault:

FROM i386/alpine:edge

RUN apk update && apk add --no-cache build-base boost-dev bzip2-dev \
	sqlite-dev xz-dev zlib-dev python3 setarch git
RUN git clone --depth=1 https://github.com/randombit/botan.git /tmp/botan

RUN cd /tmp/botan && setarch i386 python3 ./configure.py
RUN cd /tmp/botan && make -j35 && env LD_LIBRARY_PATH="/tmp/botan" ./botan-test

Make sure to adjust the make jobs accordingly.

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 18 (11 by maintainers)

Commits related to this issue

Most upvoted comments

since the Os does work, insofar as tests, (edited above) we set that for now. hopefully there’s a response and it’s not an ignored bug, in which case we get a real fix 😃

thank you!