Ceedling: A test function dying causes the loss of previous results

...
Linking test_TBuffer.out...
Running test_TBuffer.out...
sh: line 1: 58227 Segmentation fault: 11  ../../../build/test/out/test_TBuffer.out 2>&1

ERROR: Test executable "test_TBuffer.out" failed.
> Produced no output to $stdout.
> And exited with status: [139] (count of failed tests).
> This is often a symptom of a bad memory access in source or test code.

rake aborted!

/Users/mija/.gem/ruby/2.0.0/gems/ceedling-0.20.3/lib/ceedling/generator_helper.rb:36:in `test_results_error_handler'
/Users/mija/.gem/ruby/2.0.0/gems/ceedling-0.20.3/lib/ceedling/generator.rb:153:in `generate_test_results'
/Users/mija/.gem/ruby/2.0.0/gems/ceedling-0.20.3/lib/ceedling/rules_tests.rake:42:in `block in <top (required)>'
/Users/mija/.gem/ruby/2.0.0/gems/ceedling-0.20.3/lib/ceedling/task_invoker.rb:74:in `invoke_test_results'
/Users/mija/.gem/ruby/2.0.0/gems/ceedling-0.20.3/lib/ceedling/test_invoker.rb:68:in `block in setup_and_invoke'
/Users/mija/.gem/ruby/2.0.0/gems/ceedling-0.20.3/lib/ceedling/test_invoker.rb:32:in `setup_and_invoke'
/Users/mija/.gem/ruby/2.0.0/gems/ceedling-0.20.3/lib/ceedling/tasks_tests.rake:8:in `block (2 levels) in <top (required)>'
Tasks: TOP => ../../../build/test/results/test_TBuffer.pass
(See full trace by running task with --trace)

--------------------
OVERALL TEST SUMMARY
--------------------

No tests executed.

At the point of failure, 5 tests had already run (I know because the test function failing is the last one I added). No results are gathered, and no hint appears on what test function has failed. Furthermore, the Test Summary is wrong.

About this issue

  • Original URL
  • State: closed
  • Created 11 years ago
  • Comments: 27 (23 by maintainers)

Most upvoted comments

An idea about how to fix it: I see that <ceedling gem>/lib/ceedling/generator_test_runner.rb generates the test runners (duh ;P), and in line 58 it does

    @test_runner_generator.generate( module_name,
                                     runner_filepath,
                                     test_cases,
                                     mock_list,
                                     test_file_includes)

If I understand correctly, Ceedling generates one test runner per test_*.c file. Why not generate instead one test runner per test function? The module_name could be changed to module_name+test_case[n], and the result parsing should be changed to only accept exit value 0 as success, and anything else as failure. This way, any test dying would rightly be caught as a failure, and next tests would continue running without any problem. Furthermore individual tests would be more reliable. I guess that the rest of Ceedling would need no changes (since when the test running stage is reached, it looks like it runs any executables it finds that look like a test runner in the build directory?)

Additionally, having test runners with the granularity of test functions would fix issue #71 as a side effect 😉

😉 You’ve both convinced me. I agree that it’s worth having Ceedling spelunk through the output to give more meaningful context here.