Ceedling: A test function dying causes the loss of previous results
...
Linking test_TBuffer.out...
Running test_TBuffer.out...
sh: line 1: 58227 Segmentation fault: 11 ../../../build/test/out/test_TBuffer.out 2>&1
ERROR: Test executable "test_TBuffer.out" failed.
> Produced no output to $stdout.
> And exited with status: [139] (count of failed tests).
> This is often a symptom of a bad memory access in source or test code.
rake aborted!
/Users/mija/.gem/ruby/2.0.0/gems/ceedling-0.20.3/lib/ceedling/generator_helper.rb:36:in `test_results_error_handler'
/Users/mija/.gem/ruby/2.0.0/gems/ceedling-0.20.3/lib/ceedling/generator.rb:153:in `generate_test_results'
/Users/mija/.gem/ruby/2.0.0/gems/ceedling-0.20.3/lib/ceedling/rules_tests.rake:42:in `block in <top (required)>'
/Users/mija/.gem/ruby/2.0.0/gems/ceedling-0.20.3/lib/ceedling/task_invoker.rb:74:in `invoke_test_results'
/Users/mija/.gem/ruby/2.0.0/gems/ceedling-0.20.3/lib/ceedling/test_invoker.rb:68:in `block in setup_and_invoke'
/Users/mija/.gem/ruby/2.0.0/gems/ceedling-0.20.3/lib/ceedling/test_invoker.rb:32:in `setup_and_invoke'
/Users/mija/.gem/ruby/2.0.0/gems/ceedling-0.20.3/lib/ceedling/tasks_tests.rake:8:in `block (2 levels) in <top (required)>'
Tasks: TOP => ../../../build/test/results/test_TBuffer.pass
(See full trace by running task with --trace)
--------------------
OVERALL TEST SUMMARY
--------------------
No tests executed.
At the point of failure, 5 tests had already run (I know because the test function failing is the last one I added). No results are gathered, and no hint appears on what test function has failed. Furthermore, the Test Summary is wrong.
About this issue
- Original URL
- State: closed
- Created 11 years ago
- Comments: 27 (23 by maintainers)
An idea about how to fix it: I see that <ceedling gem>/lib/ceedling/generator_test_runner.rb generates the test runners (duh ;P), and in line 58 it does
If I understand correctly, Ceedling generates one test runner per test_*.c file. Why not generate instead one test runner per test function? The module_name could be changed to module_name+test_case[n], and the result parsing should be changed to only accept exit value 0 as success, and anything else as failure. This way, any test dying would rightly be caught as a failure, and next tests would continue running without any problem. Furthermore individual tests would be more reliable. I guess that the rest of Ceedling would need no changes (since when the test running stage is reached, it looks like it runs any executables it finds that look like a test runner in the build directory?)
Additionally, having test runners with the granularity of test functions would fix issue #71 as a side effect 😉
😉 You’ve both convinced me. I agree that it’s worth having Ceedling spelunk through the output to give more meaningful context here.