simplecov: Wrong coverage models result with Shoulda-Matchers

I was trying to setup a rails api with RSpec, SimpleCov and Should-Matchers and I figured out that after running tests all models appears always 100% covered regardless of written tests, including models without tests.

If I comment the Should-Matchers setup, SimpleCov works correctly for models.

I added SimpleCov call to top of rails_spec.rb:

require 'simplecov'
SimpleCov.start 'rails'

And Shoulda-Matchers config to support/shoulda_matchers.rb

Shoulda::Matchers.configure do |config|
  config.integrate do |with|
    with.test_framework :rspec
    with.library :rails
  end
end

Anyone with the same problem?

Thanks!

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 16

Most upvoted comments

Remove or comment all model tests

Your classes are still required by the acceptance tests:

when using FactoryBot to create Pets and Species
when hitting the controllers

Looking into your models - they are pretty simple (i.e has no method definitions, conditions, etc.), and all lines are actually executed when required.

In the following example you can see two green lines:

when Ruby executes the class keyword, it creates a new lexical scope for the new Pet class
then belongs_to is invoked with the :specie argument

Code within module/classes is executed just as any other. You can test this by adding a simple puts to Pet

class Pet < ApplicationRecord
  puts 'Pet is now defined!'
  
  # more code
end

Then when you run rake the output is:

$ rake
/Users/nikolay/.rvm/rubies/ruby-2.4.0/bin/ruby -I/Users/nikolay/.rvm/gems/ruby-2.4.0/gems/rspec-core-3.7.0/lib:/Users/nikolay/.rvm/gems/ruby-2.4.0/gems/rspec-support-3.7.0/lib /Users/nikolay/.rvm/gems/ruby-2.4.0/gems/rspec-core-3.7.0/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
Pet is now defined!

Pets
  GET /pets
    Listing pets
  GET /pets/:id
    Get a pet
  POST /pets
    Create a pet

Here’s what happens if we define a new method within Pet:

def greet is executed and and the new method is defined (it’s added to the Pet method list; line is green)
the actual method body is not invoked (line is red)

Please let me know what do you think of this. Maybe I’m missing something?

nbekirov on Feb 13, 2018

A very simple example:

Remove or comment all model tests
Run Rspec and check models coverage

Result: Models (100.0% covered at 1.0 hits/line)

SofiaSousa on Feb 13, 2018

I struggle to find online a document that describe C0 in a satisfying (suitable for different audiences) way that can be linked in a FAQ.

Here’s the most useful ones I came across:

https://en.wikipedia.org/wiki/Code_coverage#Basic_coverage_criteria (C0 is not mentioned as such)
http://atlas-softquality.com/faqanswer3.html
https://grosser.it/2008/04/04/whats-my-coverage-c0-c1-c2-c3-path-coverage/

nbekirov on Feb 15, 2018

Ruby’s coverage module and SimpleCov which uses it only track C0 coverage. Any line that is evaluated when a file is loaded is considered ‘covered’, whether you not you executed it. https://github.com/colszowka/simplecov/issues/340

def foo
  puts "I am not covered unless you call 'foo'"
end

but

def foo; puts "I am covered when the file is required whether or not you call 'foo'"; end

bf4 on Feb 14, 2018

@nbekirov thanks for all the help figuring this one out 😃

PragTob on Feb 14, 2018

I’m having the same problem and thought I was covering everything 😕

erijonhson on Nov 18, 2017