simplecov: SimpleCov showing incorrect coverage result when parallelize is enabled in Rails 6.0.0 beta3

The solution by @bbascarevic-tti is almost there. Here’s what I ended up with; it seems to be working as expected:

if ENV['COVERAGE']
  require 'simplecov'

  SimpleCov.start 'rails'
end

ENV['RAILS_ENV'] ||= 'test'
require_relative '../config/environment'
require 'rails/test_help'

module ActiveSupport
  class TestCase
    # Run tests in parallel with specified workers
    parallelize(workers: :number_of_processors)

    if ENV['COVERAGE']
      parallelize_setup do |worker|
        SimpleCov.command_name "#{SimpleCov.command_name}-#{worker}"
      end

      parallelize_teardown do |worker|
        SimpleCov.result
      end
    end

    # Setup all fixtures in test/fixtures/*.yml for all tests in alphabetical order.
    fixtures :all

    # Add more helper methods to be used by all tests here...
  end
end

There are a few changes from @bbascarevic-tti’s solution:

1. Do not set SimpleCov.pid to Process.pid in the child process. This causes SimpleCov to report a failure for every single child process because SimpleCov then thinks that the child process is the coordinator.
2. Do not specify SimpleCov.use_merging since it defaults to true.
3. Don’t bother wiping out the SimpleCov.at_exit hook - these are not run in the workers anyway.
4. Set a parallelize_teardown hook to call SimpleCov.result. This complex method generates the result and, if SimpleCov.use_merging is true (which it is by default), stores the result for later merging. This allows the results of all workers to be merged into a single result.

Here’s an example output:

↪ rm -rf coverage/ && DISABLE_SPRING=1 COVERAGE=1 bin/rails test
Run options: --seed 52156

# Running:

......................

Finished in 0.393015s, 55.9775 runs/s, 101.7772 assertions/s.
22 runs, 40 assertions, 0 failures, 0 errors, 0 skips
Coverage report generated for MiniTest, MiniTest-0, MiniTest-1, MiniTest-2, MiniTest-3 to /home/herold/project/coverage. 58 / 148 LOC (39.19%) covered.

I’ve run the wipe / re-run without Spring many times and receive a consistent result so I don’t believe there are any race conditions in the process model. When you mix in Spring, you get the lovely problems inherent in such a venture, so YMMV if you choose to try it with Spring.

Threads

Note that I’ve only been able to produce completely consistent results using process parallelization. When I tested out with: :threads, I noticed that the coverage numbers would non-deterministically jump between 58 / 148 and 49 / 148. There were also errors due to thread deadlocks in the database too though, so I think threads on MRI just aren’t fully baked (which is fine since you don’t get true parallelism anyway).

Next Steps

I haven’t been able to figure out a way to upstream this workflow into SimpleCov, so if someone has any bright ideas, please feel free to do that! The way parallel_tests sets the environment variables and gives you access to both the current worker ID and the total number of workers makes sense. Perhaps Rails could use similar functionality to pass the total number of workers into the parallelize_setup and parallelize_teardown hooks?

And as a small update, I did get the ruby together fund and hence an improved rails support especially its new test parallelization is towards the top of the list (after branch coverage and some friends though)

Yes this is a biggie. Sorry for taking so long to respond, I took a bit of a break from maintaining simplecov.

Making simplecov completely work with rails and its new parallelization will be a bigger undertaking I fear. PRs are welcome. I’ll see if I can allocate the time for it but I’m unsure.

Thank you for reporting 💚

Any interest in maybe documenting @michaelherold’s (excellent! working!) approach in the README, until/unless built-in support for Rails parallelization is added?

Now that Rails will only start parallelizing tests once you hit 50 of them, the fact coverage will suddenly go from correct to near-0% is really confusing and takes time to piece together by searching issues in the repo. (The optimization is totally valid, but when a working initial setup stops working suddenly it can be disorienting.)

As a workaround we can piggy-back on the already existing feature of result merging, we just need to trick SimpleCov that each parallel run is another Command. We set SimpleCov.use_merging to true in the root process, and then in each fork we change the command name so forks would not overwrite each others’ results.

# test_helper.rb

require 'simplecov'

SimpleCov.use_merging(true)  # Important!
SimpleCov.start('rails') do
  ... # Whatever you use here...
end

module ActiveSupport
  class TestCase
    parallelize

    parallelize_setup do |worker|
      SimpleCov.command_name("#{SimpleCov.command_name}-#{worker}")
      SimpleCov.pid = Process.pid
      SimpleCov.at_exit {} # This will suppress formatters running at the end of each fork
    end
    # ...
  end
end

@alkesh26 I don’t have a solution for you but have link to my app and reproducible steps:

1. Uncomment parallelize(workers: :number_of_processors)
2. Run tests rm -rf coverage/; RAILS_ENV=test ./bin/rails test; open coverage/index.html
3. Coverage of app/lib/watermelon/example.rb shows 0%
4. Comment out parallelize(workers: :number_of_processors)
5. Run tests rm -rf coverage/; RAILS_ENV=test ./bin/rails test; open coverage/index.html
6. Coverage of app/lib/watermelon/example.rb shows 70%

Omit loading spring from bin/rails and bin/rake from Rails 6 application. Loading spring server has his side effects

# bin/rails

#!/usr/bin/env ruby
# load File.expand_path('spring', __dir__)
...

I think the problem was either that I still had parallelize(workers: 1) in my test setup OR that I actually didn’t have any system tests yet. As soon as I removed parallelize completely and added a working spec, the test coverage is working fine for rails test:system test 🎉

This may be common knowledge, but make sure you’re also eager_loading your testing environment. I followed the above advice, but still had some accuracy issues until I discovered this. Just run with CI=true bin/rails test:all

# config/environments/test.rb
Rails.application.configure do
  # many more configs...
  
  # Use the CI EnvVar to control this
  config.eager_load = ENV["CI"].present?
  
  # some more configs...
end