openssl: Regression test (1.1.0 apps against 1.1.1-dev shared libraries) failures

Using #5620, I ran a quick regression test:

: ; OPENSSL_REGRESSION=$(cd ../_build; pwd) ./util/shlib_wrap.sh apps/openssl version
OpenSSL 1.1.0h-dev  xx XXX xxxx (Library: OpenSSL 1.1.1-pre3-dev  xx XXX xxxx)

: ; OPENSSL_REGRESSION=$(cd ../_build; pwd) make -j8 test

The result:

( cd test; \
  mkdir -p test-runs; \
  SRCTOP=../../1.1.0 \
  BLDTOP=../. \
  RESULT_D=test-runs \
  PERL="/usr/bin/perl" \
  EXE_EXT= \
  OPENSSL_ENGINES=`cd .././engines; pwd` \
  OPENSSL_DEBUG_MEMORY=on \
    /usr/bin/perl ../../1.1.0/test/run_tests.pl  )
../../1.1.0/test/recipes/01-test_abort.t ............ ok
../../1.1.0/test/recipes/01-test_sanity.t ........... ok
../../1.1.0/test/recipes/01-test_symbol_presence.t .. ok
../../1.1.0/test/recipes/02-test_ordinals.t ......... ok
../../1.1.0/test/recipes/03-test_exdata.t ........... ok
../../1.1.0/test/recipes/03-test_ui.t ............... ok
../../1.1.0/test/recipes/04-test_pem.t .............. ok
../../1.1.0/test/recipes/05-test_bf.t ............... ok
../../1.1.0/test/recipes/05-test_cast.t ............. ok
../../1.1.0/test/recipes/05-test_des.t .............. ok
../../1.1.0/test/recipes/05-test_hmac.t ............. ok
../../1.1.0/test/recipes/05-test_idea.t ............. ok
../../1.1.0/test/recipes/05-test_md2.t .............. skipped: md2 is not supported by this OpenSSL build
../../1.1.0/test/recipes/05-test_md4.t .............. ok
../../1.1.0/test/recipes/05-test_md5.t .............. ok
../../1.1.0/test/recipes/05-test_mdc2.t ............. ok
../../1.1.0/test/recipes/05-test_rand.t ............. ok
../../1.1.0/test/recipes/05-test_rc2.t .............. ok
../../1.1.0/test/recipes/05-test_rc4.t .............. ok
../../1.1.0/test/recipes/05-test_rc5.t .............. skipped: rc5 is not supported by this OpenSSL build
../../1.1.0/test/recipes/05-test_rmd.t .............. ok
../../1.1.0/test/recipes/05-test_sha1.t ............. ok
../../1.1.0/test/recipes/05-test_sha256.t ........... ok
../../1.1.0/test/recipes/05-test_sha512.t ........... ok
../../1.1.0/test/recipes/05-test_wp.t ............... ok
../../1.1.0/test/recipes/10-test_bn.t ............... ok
../../1.1.0/test/recipes/10-test_exp.t .............. ok
../../1.1.0/test/recipes/15-test_dh.t ............... ok
../../1.1.0/test/recipes/15-test_dsa.t .............. ok
../../1.1.0/test/recipes/15-test_ec.t ............... ok
../../1.1.0/test/recipes/15-test_ecdsa.t ............ ok
../../1.1.0/test/recipes/15-test_genrsa.t ........... Dubious, test returned 4 (wstat 1024, 0x400)
Failed 4/5 subtests 
../../1.1.0/test/recipes/15-test_rsa.t .............. ok
../../1.1.0/test/recipes/15-test_rsapss.t ........... ok
../../1.1.0/test/recipes/20-test_enc.t .............. ok
../../1.1.0/test/recipes/20-test_passwd.t ........... ok
../../1.1.0/test/recipes/25-test_crl.t .............. ok
../../1.1.0/test/recipes/25-test_d2i.t .............. ok
../../1.1.0/test/recipes/25-test_pkcs7.t ............ ok
../../1.1.0/test/recipes/25-test_req.t .............. ok
../../1.1.0/test/recipes/25-test_sid.t .............. ok
../../1.1.0/test/recipes/25-test_verify.t ........... ok
../../1.1.0/test/recipes/25-test_x509.t ............. ok
../../1.1.0/test/recipes/30-test_afalg.t ............ ok
../../1.1.0/test/recipes/30-test_engine.t ........... ok
../../1.1.0/test/recipes/30-test_evp.t .............. ok
../../1.1.0/test/recipes/30-test_evp_extra.t ........ ok
../../1.1.0/test/recipes/30-test_pbelu.t ............ ok
../../1.1.0/test/recipes/40-test_rehash.t ........... ok
../../1.1.0/test/recipes/60-test_x509_store.t ....... ok
../../1.1.0/test/recipes/70-test_asyncio.t .......... ok
../../1.1.0/test/recipes/70-test_bad_dtls.t ......... ok
../../1.1.0/test/recipes/70-test_clienthello.t ...... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests 
../../1.1.0/test/recipes/70-test_packet.t ........... ok
../../1.1.0/test/recipes/70-test_sslcbcpadding.t .... Dubious, test returned 5 (wstat 1280, 0x500)
Failed 5/5 subtests 
../../1.1.0/test/recipes/70-test_sslcertstatus.t .... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests 
../../1.1.0/test/recipes/70-test_sslextension.t ..... Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/3 subtests 
../../1.1.0/test/recipes/70-test_sslmessages.t ...... Dubious, test returned 5 (wstat 1280, 0x500)
Failed 5/5 subtests 
../../1.1.0/test/recipes/70-test_sslrecords.t ....... Dubious, test returned 5 (wstat 1280, 0x500)
Failed 5/11 subtests 
../../1.1.0/test/recipes/70-test_sslsessiontick.t ... Dubious, test returned 8 (wstat 2048, 0x800)
Failed 8/10 subtests 
../../1.1.0/test/recipes/70-test_sslskewith0p.t ..... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests 
../../1.1.0/test/recipes/70-test_sslvertol.t ........ Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/2 subtests 
../../1.1.0/test/recipes/70-test_tlsextms.t ......... Dubious, test returned 9 (wstat 2304, 0x900)
Failed 9/9 subtests 
../../1.1.0/test/recipes/70-test_verify_extra.t ..... ok
../../1.1.0/test/recipes/80-test_ca.t ............... ok
../../1.1.0/test/recipes/80-test_cipherlist.t ....... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests 
../../1.1.0/test/recipes/80-test_cms.t .............. ok
../../1.1.0/test/recipes/80-test_ct.t ............... ok
../../1.1.0/test/recipes/80-test_dane.t ............. ok
../../1.1.0/test/recipes/80-test_dtls.t ............. ok
../../1.1.0/test/recipes/80-test_dtlsv1listen.t ..... ok
../../1.1.0/test/recipes/80-test_ocsp.t ............. ok
../../1.1.0/test/recipes/80-test_pkcs12.t ........... ok
../../1.1.0/test/recipes/80-test_ssl_new.t .......... Dubious, test returned 5 (wstat 1280, 0x500)
Failed 5/19 subtests 
../../1.1.0/test/recipes/80-test_ssl_old.t .......... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/7 subtests 
../../1.1.0/test/recipes/80-test_ssl_test_ctx.t ..... ok
../../1.1.0/test/recipes/80-test_sslcorrupt.t ....... ok
../../1.1.0/test/recipes/80-test_tsa.t .............. ok
../../1.1.0/test/recipes/80-test_x509aux.t .......... ok
../../1.1.0/test/recipes/90-test_async.t ............ ok
../../1.1.0/test/recipes/90-test_bio_enc.t .......... ok
../../1.1.0/test/recipes/90-test_bioprint.t ......... ok
../../1.1.0/test/recipes/90-test_constant_time.t .... ok
../../1.1.0/test/recipes/90-test_fatalerr.t ......... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests 
../../1.1.0/test/recipes/90-test_fuzz.t ............. ok
../../1.1.0/test/recipes/90-test_gmdiff.t ........... ok
../../1.1.0/test/recipes/90-test_heartbeat.t ........ skipped: heartbeats is not supported by this OpenSSL build
../../1.1.0/test/recipes/90-test_ige.t .............. ok
../../1.1.0/test/recipes/90-test_memleak.t .......... ok
../../1.1.0/test/recipes/90-test_p5_crpt2.t ......... ok
../../1.1.0/test/recipes/90-test_secmem.t ........... ok
../../1.1.0/test/recipes/90-test_shlibload.t ........ ok
../../1.1.0/test/recipes/90-test_srp.t .............. ok
../../1.1.0/test/recipes/90-test_sslapi.t ........... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests 
../../1.1.0/test/recipes/90-test_threads.t .......... ok
../../1.1.0/test/recipes/90-test_v3name.t ........... ok

Test Summary Report
-------------------
../../1.1.0/test/recipes/15-test_genrsa.t         (Wstat: 1024 Tests: 5 Failed: 4)
  Failed tests:  2-5
  Non-zero exit status: 4
../../1.1.0/test/recipes/70-test_clienthello.t    (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
../../1.1.0/test/recipes/70-test_sslcbcpadding.t  (Wstat: 1280 Tests: 5 Failed: 5)
  Failed tests:  1-5
  Non-zero exit status: 5
../../1.1.0/test/recipes/70-test_sslcertstatus.t  (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
../../1.1.0/test/recipes/70-test_sslextension.t   (Wstat: 512 Tests: 3 Failed: 2)
  Failed tests:  2-3
  Non-zero exit status: 2
../../1.1.0/test/recipes/70-test_sslmessages.t    (Wstat: 1280 Tests: 5 Failed: 5)
  Failed tests:  1-5
  Non-zero exit status: 5
../../1.1.0/test/recipes/70-test_sslrecords.t     (Wstat: 1280 Tests: 11 Failed: 5)
  Failed tests:  1-3, 6, 10
  Non-zero exit status: 5
../../1.1.0/test/recipes/70-test_sslsessiontick.t (Wstat: 2048 Tests: 10 Failed: 8)
  Failed tests:  1-8
  Non-zero exit status: 8
../../1.1.0/test/recipes/70-test_sslskewith0p.t   (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
../../1.1.0/test/recipes/70-test_sslvertol.t      (Wstat: 512 Tests: 2 Failed: 2)
  Failed tests:  1-2
  Non-zero exit status: 2
../../1.1.0/test/recipes/70-test_tlsextms.t       (Wstat: 2304 Tests: 9 Failed: 9)
  Failed tests:  1-9
  Non-zero exit status: 9
../../1.1.0/test/recipes/80-test_cipherlist.t     (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
../../1.1.0/test/recipes/80-test_ssl_new.t        (Wstat: 1280 Tests: 19 Failed: 5)
  Failed tests:  1-2, 6, 8, 17
  Non-zero exit status: 5
../../1.1.0/test/recipes/80-test_ssl_old.t        (Wstat: 256 Tests: 7 Failed: 1)
  Failed test:  3
  Non-zero exit status: 1
../../1.1.0/test/recipes/90-test_fatalerr.t       (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
../../1.1.0/test/recipes/90-test_sslapi.t         (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
Files=96, Tests=556, 183 wallclock secs ( 1.95 usr  0.20 sys + 140.39 cusr 62.38 csys = 204.92 CPU)
Result: FAIL
make[1]: *** [Makefile:158: _tests] Error 1
make[1]: Leaving directory '/home/levitte/gitwrk/openssl.net/official/_build-1.1.0'
make: *** [Makefile:155: tests] Fel 2

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 16 (16 by maintainers)

Commits related to this issue

Support "-min_protocol" and "-max_protocol" in s_server and s_client If for nothing else, they are needed when doing a regression test Partially fixes #5661 — committed to levitte/openssl by levitte 6 years ago
In TLSProxy::Proxy, specify TLSv1.2 as maximum allowable protocol Partially fixes #5661 — committed to levitte/openssl by levitte 6 years ago
Support "-min_protocol" and "-max_protocol" in s_server and s_client If for nothing else, they are needed when doing a regression test Partially fixes #5661 Reviewed-by: Matt Caswell <matt@openssl.... — committed to openssl/openssl by levitte 6 years ago
In TLSProxy::Proxy, specify TLSv1.2 as maximum allowable protocol Partially fixes #5661 Reviewed-by: Matt Caswell <matt@openssl.org> (Merged from https://github.com/openssl/openssl/pull/5662) — committed to openssl/openssl by levitte 6 years ago

Most upvoted comments

That’s an interesting observation. In that case, would you say that it would be valid to slightly alter those tests to check for any alert rather than specific ones? At least the protocol version tests?

The tests are allowed to know about the OpenSSL internal implementation and therefore know to expect a specific alert in certain cases (in this way they are not like normal applications). This could be useful in certain circumstances, i.e. if a test is checking that a certain scenario fails, then it could be useful to additionally check that it is failing for the right reason and not because something else unrelated to the test happened to go wrong.

mattcaswell on Apr 16, 2018

As I understand it, the test fails because some alert has changed between 1.1.0 and 1.1.1. While it may seem nice, it’s also perfectly possible that some application actually pays attention to the exact alert and that it might therefore be significant. Again, I cannot tell if this means 1.1.1 or 1.1.0 should be fixed.

It’s not immediately obvious what the issue is here, but it reads to me like we were expecting an alert but got some other failure instead. In any case there are various alerts that may have changed between 1.1.0 and 1.1.1. I think applications shouldn’t rely on always getting a specific one.

mattcaswell on Apr 16, 2018

… if I understand the test correctly, this test injects a second client or server hello, and this should create a failure in TLSProxy, and because that’s what the test recipe is detecting, the recipe should be successful. Not so when running against 1.1.1 libraries, which I assume means that duplicate hellos were somehow tolerated

It injects a second client or server hello extension. In 1.1.1 we completely rewrote the extensions code. 1.1.1 will refuse duplicate extensions for extensions that it knows about and ignore others. In 1.1.0 we refuse all duplicate extensions regardless of whether we recognise the extension or not. This was a deliberate design decision on the basis that it made the code simpler and shouldn’t have any adverse impacts.

mattcaswell on Apr 16, 2018