rspec-core: on long strings, rspec may not display the part of the string that does not match
I’m trying to track down an error in a flaky test. Unfortunately the string output presented by rspec does not display the problem.
The following test should be enough to demonstrate the problem:
describe 'a long string' do
it 'shows the right part of the diff' do
got = "a"*100 + "Z" + "a"*100
want = "a" * (1 + 100*2)
expect(got).to eq(want)
end
end
Yields the following output:
Failures:
1) a long string shows the right part of the diff
Failure/Error: expect(got).to eq(want)
expected: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
got: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
(compared using ==)
# ./longstring_spec.rb:5:in `block (2 levels) in <top (required)>'
The strings displayed by rspec are identical; the diff is in the truncated part. If the string is any shorter, the diff appears correctly.
Running on a Mac in iTerm2.
$ rspec --version
RSpec 3.7
- rspec-core 3.7.1
- rspec-expectations 3.7.0
- rspec-mocks 3.7.0
- rspec-support 3.7.1
About this issue
- Original URL
- State: open
- Created 6 years ago
- Reactions: 51
- Comments: 27 (17 by maintainers)
Ah, this is a dupe of https://github.com/rspec/rspec-expectations/issues/991.
The suggestion there is to set
RSpec::Support::ObjectFormatter.default_instance.max_formatted_output_lengthto a high enough value.Example
Say we have two strings of strings of length 11:
And the value of
RSpec.Support.ObjectFormatter.max_formatted_output_lengthwas set to4Current formatting
I’m not a big fan of that.
Proposed formatting
Leading & traling ellipsis. You’d see something like this…
Here’s where the formatting is actually done:
https://github.com/rspec/rspec-support/blob/master/lib/rspec/support/object_formatter.rb#L32
I will probably try to work on this if we are ok with the scope of the task.
What I think could be nice to display is:
But I have few issues:
Scanning through the discussion here, there is one simple solution that is missing that would help me 95% of the time when I encounter this issue: Provide an option that when enabled just prints out the complete actual and expected strings, without diffing or trimming (and maybe add a switch to disable it by default if running an entire suite and enable by default if running a single spec). Usually I encounter these issues in data where context within the string t is important - especially when there are similar data sections. The semi-intelligent prettifying helpers are more of a hinderance than a help most of the time in my experience. Yes, having several screens full of data can be annoying, but having insert puts statements or jumping into byebug to inspect the strings in-situ is more annoying 😃
FWIW, I think RSpec badly needs to move away from using diff-lcs for diffing, and implement its own differ. diff-lcs works OK, but we convert everything to strings to diff, and we only get a line-by-line diff. This has several problems:
A differ written for RSpec, that understands the semantics of RSpec matching, would produce far better output.
I know that’s a much larger task than just this issue, but it’s something I’ve been thinking about for a while, and was thinking could be a headline feature for RSpec 4.
If someone wants to experiment with this, it might be instructive to look at the differ that is included with Elixir’s ExUnit. It works quite well:
https://github.com/elixir-lang/elixir/blob/master/lib/ex_unit/lib/ex_unit/diff.ex
@benoittgt @kevinburkeomg Thanks for suggesting
max_formatted_output_length.Code to set
max_formatted_output_length(docs):Thanks to this StackOverflow answer by @pjmartorell.
@sriedel Did you try to use
RSpec::Support::ObjectFormatter.default_instance.max_formatted_output_length?I have a draft version of the “new” object formater but it needs more work. I hope to send a proper version soon.
ObjectFormaterdoes a lot I need to understand what I need to keep from the original object_formater. Also I need to DRY the code withObjectFormater.At the same time I’m doing a small class about computer science at Harvard (essentially to learn C) and I have a final project. I can give a try to “port”
ex_unit/diffto ruby (elixir test are instructive on how it works elixir-https://github.com/elixir-lang/elixir/blob/master/lib/ex_unit/test/ex_unit/diff_test.exs). I did some Elixir last year it could be a good project to end that class and validate my diploma.This issue is about printing different strings even when truncated, increasing the length is a workaround but not the ideal fix
The problem we’re dealing in this ticket is that string diff is not displayed in its entirety. It is certainly not user-friendly, and I, personally, keep manually adding
expectations.max_formatted_output_length = 1000tospec/spec_helper.rbin projects I’m working on.If we look at the
spec/spec_helper.rbinitializer, there is config for rspec-expectations there, even though rspec-core does not require use of rspec-expectations, and can be used without it:Would it be sufficiently user-friendly if the generated
spec/spec_helper.rbalready had themax_formatted_output_lengthoption, at least to the default value?We’ve actually added this option to the generated
spec_helper.rbalready in the not yet released RSpec 4 with this change:I might be biased due to professional deformation, but I think the current DSL for RSpec configuration is already user-friendly. Still open to suggestions.
A PR to improve the documentation for this option would be appreciated. Otherwise, is there any there anything actionable left here?
Sorry for the slow review here @benoittgt !
The
ObjectFormatteris currently used in several ways, to format objects for descriptions etc, and to format them for expected / actual. I think this diffing formatter is worth having as an additional object (although if we could refactor it to share some code with the current formatter that’d be nice), which is then used when we know what expected / actual are. (Seerspec-expectationmatchers for how this is used). This could obviously then just be added as an additional peice, with tests demonstrating usage of course, and then we can switch the matchers to use this.From the differs perspective everything is strings.
If we get it to show at least one difference thats a big improvement, I wouldn’t worry so much about multiples.
I think it should just happen
I’m fine with going with a simpler solution, as long as it improves things (as I expect it to!) and we can always iterate.