Ceedling: ceedling exit code is 0 on unit test failure

When running ceedling test:all with failing unit tests, the process returns a 0 status code. It should return 1 as the status code so CI tools can detect the failure.

I am using the following versions of the farmework:

Ceedling:: 0.27.0 CException:: 1.3.1.18 CMock:: 2.4.4.215 Unity:: 2.4.1.120

This issue was reported in #117 and was reported to be fixed in #130; however, I seem to still experience this problem.

I have setup a repository that reproduces this problem. It can be found at: https://github.com/nmisoft/ceedling_exit_code_bug

Interestingly using rake test:all will correctly return 1 as the status code.

I have reproduced this problem both on Windows and Linux.

Windows

OS Version: 10.0.14393 N/A Build 14393

Using ceedling

>ceedling test:all
---------------------
BUILD FAILURE SUMMARY
---------------------
Unit test failures.

ERROR: Aborted with Exit Code 1
>echo %ERRORLEVEL%
0

Using rake

>rake test:all
Test 'test_demo.c'
------------------
Running test_demo.out...

-------------------
FAILED TEST SUMMARY
-------------------
[test_demo.c]
  Test: test_demo_fail
  At line (16): "Expected TRUE Was FALSE"

--------------------
OVERALL TEST SUMMARY
--------------------
TESTED:  1
PASSED:  0
FAILED:  1
IGNORED: 0

---------------------
BUILD FAILURE SUMMARY
---------------------
Unit test failures.
>echo %ERRORLEVEL%
1

Linux

Linux 4.4.0-78-generic #99-Ubuntu SMP Thu Apr 27 15:29:09 UTC 2017 x86_64 GNU/Linux

Using ceedling

$ ceedling test:all


Test 'test_demo.c'
------------------
Running test_demo.out...

-------------------
FAILED TEST SUMMARY
-------------------
[test_demo.c]
  Test: test_demo_fail
  At line (16): "Expected TRUE Was FALSE"

--------------------
OVERALL TEST SUMMARY
--------------------
TESTED:  1
PASSED:  0
FAILED:  1
IGNORED: 0

---------------------
BUILD FAILURE SUMMARY
---------------------
Unit test failures.
$ echo $?
0

Using rake

$ rake test:all


Test 'test_demo.c'
------------------
Running test_demo.out...

-------------------
FAILED TEST SUMMARY
-------------------
[test_demo.c]
  Test: test_demo_fail
  At line (16): "Expected TRUE Was FALSE"

--------------------
OVERALL TEST SUMMARY
--------------------
TESTED:  1
PASSED:  0
FAILED:  1
IGNORED: 0

---------------------
BUILD FAILURE SUMMARY
---------------------
Unit test failures.
$ echo $?
1

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Reactions: 1
  • Comments: 26 (12 by maintainers)

Most upvoted comments

@mvandervoord I have tested this on my Linux Mint setup and Windows 10 setup and it works in both cases!

@mvandervoord it works fine on Ubuntu