minitest-reporters: Rails 5.1.3 display regression
Hello.
Checking Rails 5.1.3.rc3 against our main application, and I find display quite broken while executing tests. I formatted output from my terminal (adding LF and removing extra spaces to fit here in the issue), it seems that default reporter still run even with Rails trick from README (Minitest::Reporters.use! Minitest::Reporters::ProgressReporter.new, ENV, Minitest.backtrace_filter`
❯ rails test
Started with run options --seed 34013
Run options: --seed 34013-=---=---=---=---=---=---=-] 0% Time: 00:00:00, ETA: ??:??:??
# Running:
. 2756/2: [ ] 0% Time: 00:00:02, ETA: 00:56:0
. 2756/3: [ ] 0% Time: 00:00:02, ETA: 00:37:3
... 2756/6: [ ] 0% Time: 00:00:02, ETA: 00:18
. 2756/7: [ ] 0% Time: 00:00:02, ETA: 00:15:5
. 2756/8: [ ] 0% Time: 00:00:02, ETA: 00:14:0
.. 2756/10: [ ] 0% Time: 00:00:02, ETA: 00:12
:.. 2756/12: [ ] 0% Time: 00:00:02, ETA: 00:10
:. 2756/13: [ ] 0% Time: 00:00:02, ETA: 00:10:1
. 2756/14: [ ] 0% Time: 00:00:03, ETA: 00:09:5
. 2756/15: [ ] 0% Time: 00:00:03, ETA: 00:09:5
. 2756/16: [ ] 0% Time: 00:00:03, ETA: 00:09:4
. 2756/17: [ ] 0% Time: 00:00:03, ETA: 00:09:3
. 2756/18: [ ] 0% Time: 00:00:03, ETA: 00:09:0
... 2756/21: [ ] 0% Time: 00:00:03, ETA: 00:07
......^CInterrupted. Exiting...
2756/2756: [=====================================] 100% Time: 00:00:03, Time: 00:00:03
Finished in 3.71382s
26 tests, 81 assertions, 0 failures, 0 errors, 0 skips
Finished in 3.713659s, 7.0012 runs/s, 21.8114 assertions/s.
26 runs, 81 assertions, 0 failures, 0 errors, 0 skips
I have no idea how things work so I only report to keep track of it. It will eventually be resolved after Rails 5.1.3 final release, or may be it simply a minitest bug?
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 13
- Comments: 28 (5 by maintainers)
Commits related to this issue
- Remove minitest-reporters gem Problem: An intractable conflict between the gem and the Rails 5.x test reporter: double reporting which furthermore negates the benefit of the gem's progress bar. See h... — committed to MushroomObserver/mushroom-observer by JoeCohen 7 years ago
- Make sure our Minitest plugin initializes last. This ensures that our changes to Minitest.reporters are not trampled by other Minitest plugins, like the core Rails Minitest plugin that was introduced... — committed to samcday/minitest-reporters by samcday 7 years ago
- Make sure our Minitest plugin initializes last. This ensures that our changes to Minitest.reporters are not trampled by other Minitest plugins, like the core Rails Minitest plugin that was introduced... — committed to samcday/minitest-reporters by samcday 7 years ago
- Make sure our Minitest plugin initializes last. This ensures that our changes to Minitest.reporters are not trampled by other Minitest plugins, like the core Rails Minitest plugin that was introduced... — committed to samcday/minitest-reporters by samcday 7 years ago
I found a cleaner workaround than the one above. AFAICT the underlying problem is Rails had this commit to make sure the Rails minitest plugin would initialize first. I think at some point that commit was lost (the file that was changed in that commit no longer exists in Rails 5.1 series, they probably missed it when cherry-picking changes around or whatever). I’ve raised rails/rails#30491 to track the upstream issue.
You can replicate the same behaviour by adding the following to your
test_helper.rb, directly after requiring everything:I made a PR that should fix the double reporters output https://github.com/rails/rails/pull/31901
So heres the trouble.
Minitest plugins are loaded based on being in the path of minitest/*_plugin.rb.
Now that Rails uses a minitest plugin as of 5.1.3,
rails_plugin.rb, it gets loaded through minitest’s normal plugin loading (which I assume is alphabetical) along withminitest_reporters_plugin.rb.this gem’s plugin resets minitests reporters to be just its own
DelegateReporter, passing in the reporters that already exist into itself so it can manage reporters for you.Rails then comes along and deletes the default reporters (though they aren’t there anymore, they’re in delegate reporter now) and then adds its own. So now delegate reporter loops through the originals for output, and then rails reporters do their thing – double output.
I’m not sure what the best fix is in terms of fixing it from MiniTest::Reporters.
I’ve just tested this in Rails 5.1.6 and it seems working just fine. No double output.
Any way to fix this in combination with Rails 5.0.7.2 (currently newest Rails 5.0)? Problem occurs with both minitest-reporters 1.1.14 as well as minitest-reporters 1.3.8. @tekniklr did you manage to fix this in Rails 5.0?
🤔 Hum…
When disabling minutest-reporters from test_helper, the test suite appears to double output “Running”. So it’s probably just minutest related
@likeuwill Yes. It’s fixed on
5.1.6https://github.com/rails/rails/blob/5-1-stable/railties/CHANGELOG.md#rails-516-march-29-2018
This is expected since I have reverted changes which fixed the problem in .18 because of regressions they added 😦
Experiencing the same thing in the final release of 5.1.3.
Was able to confirm that adding this gem alone reproduces the same behavior as @bobmaerten’s screenshot. I don’t even need to add
Minitest::Reporters.use! Minitest::Reporters::SpecReporter.newCommenting out
gem 'minitest-reporterin the Gemfile +bundle install; rails testremoves the doubled output.