openssl: Master fails 20-test_app.t

MacOS 10.15.5, Xcode-11.5, current master.

Test Summary Report
-------------------
20-test_app.t                    (Wstat: 256 Tests: 3 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
Files=203, Tests=2022, 567 wallclock secs ( 3.26 usr  0.61 sys + 243.44 cusr 65.14 csys = 312.45 CPU)
Result: FAIL
make[1]: *** [_tests] Error 1
make: *** [tests] Error 2

Output of perl configdata.pm --dump: config.dump.txt

$ make test TESTS="test_app" VF0=1 V=1
/Applications/Xcode.app/Contents/Developer/usr/bin/make depend && /Applications/Xcode.app/Contents/Developer/usr/bin/make _tests
( SRCTOP=. \
	  BLDTOP=. \
	  PERL="/opt/local/bin/perl5.26" \
	  EXE_EXT= \
	  /opt/local/bin/perl5.26 ./test/run_tests.pl test_app )
20-test_app.t .. 
# The results of this test will end up in test-runs/test_app
1..3
Usage: help
../../util/wrap.pl ../../apps/openssl => 1
not ok 1 - Run openssl app with no args

#   Failed test 'Run openssl app with no args'
#   at test/recipes/20-test_app.t line 18.
Standard commands
asn1parse         ca                ciphers           cmp               
cms               crl               crl2pkcs7         dgst              
dhparam           dsa               dsaparam          ec                
ecparam           enc               engine            errstr            
fipsinstall       gendsa            genpkey           genrsa            
help              info              kdf               list              
mac               nseq              ocsp              passwd            
pkcs12            pkcs7             pkcs8             pkey              
pkeyparam         pkeyutl           prime             provider          
rand              rehash            req               rsa               
rsautl            s_client          s_server          s_time            
sess_id           smime             speed             spkac             
srp               storeutl          ts                verify            
version           x509              

Message Digest commands (see the `dgst' command for more details)
blake2b512        blake2s256        gost              md2               
md4               md5               mdc2              rmd160            
sha1              sha224            sha256            sha3-224          
sha3-256          sha3-384          sha3-512          sha384            
sha512            sha512-224        sha512-256        shake128          
shake256          sm3               

Cipher commands (see the `enc' command for more details)
aes-128-cbc       aes-128-ecb       aes-192-cbc       aes-192-ecb       
aes-256-cbc       aes-256-ecb       aria-128-cbc      aria-128-cfb      
aria-128-cfb1     aria-128-cfb8     aria-128-ctr      aria-128-ecb      
aria-128-ofb      aria-192-cbc      aria-192-cfb      aria-192-cfb1     
aria-192-cfb8     aria-192-ctr      aria-192-ecb      aria-192-ofb      
aria-256-cbc      aria-256-cfb      aria-256-cfb1     aria-256-cfb8     
aria-256-ctr      aria-256-ecb      aria-256-ofb      base64            
bf                bf-cbc            bf-cfb            bf-ecb            
bf-ofb            camellia-128-cbc  camellia-128-ecb  camellia-192-cbc  
camellia-192-ecb  camellia-256-cbc  camellia-256-ecb  cast              
cast-cbc          cast5-cbc         cast5-cfb         cast5-ecb         
cast5-ofb         des               des-cbc           des-cfb           
des-ecb           des-ede           des-ede-cbc       des-ede-cfb       
des-ede-ofb       des-ede3          des-ede3-cbc      des-ede3-cfb      
des-ede3-ofb      des-ofb           des3              desx              
idea              idea-cbc          idea-cfb          idea-ecb          
idea-ofb          rc2               rc2-40-cbc        rc2-64-cbc        
rc2-cbc           rc2-cfb           rc2-ecb           rc2-ofb           
rc4               rc4-40            rc5               rc5-cbc           
rc5-cfb           rc5-ecb           rc5-ofb           seed              
seed-cbc          seed-cfb          seed-ecb          seed-ofb          
sm4-cbc           sm4-cfb           sm4-ctr           sm4-ecb           
sm4-ofb           zlib              

../../util/wrap.pl ../../apps/openssl help => 0
ok 2 - Run openssl app with help
Invalid command '-help'; type "help" for a list.
../../util/wrap.pl ../../apps/openssl -help => 1
ok 3 - Run openssl app with incorrect arg
# Looks like you failed 1 test of 3.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/3 subtests 

Test Summary Report
-------------------
20-test_app.t (Wstat: 256 Tests: 3 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
Files=1, Tests=3,  1 wallclock secs ( 0.01 usr  0.00 sys +  0.11 cusr  0.05 csys =  0.17 CPU)
Result: FAIL
make[1]: *** [_tests] Error 1
make: *** [tests] Error 2

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 19 (19 by maintainers)

Most upvoted comments

Fixed on master by 5b286641efef67aed5af026302b2176e3e368ae9.

it does show all output of each failed (sub-)test case,

Ah, so the provided output matches the expectations? In that case I’m OK.

Maybe we should go for less cryptic names 😄 instead of VFO and VFP - suggestions welcome.

$ make test TESTS="test_cmp_cli" VF0=1

You’ve been using VF0=1 instead VFO=1 which stands for HARNESS_VERBOSE_FAILURES_ONLY.