rspec-core: on long strings, rspec may not display the part of the string that does not match

I’m trying to track down an error in a flaky test. Unfortunately the string output presented by rspec does not display the problem.

The following test should be enough to demonstrate the problem:

describe 'a long string' do
  it 'shows the right part of the diff' do
    got = "a"*100 + "Z" + "a"*100
    want = "a" * (1 + 100*2)
    expect(got).to eq(want)
  end
end

Yields the following output:

Failures:

  1) a long string shows the right part of the diff
     Failure/Error: expect(got).to eq(want)

       expected: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
            got: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"

       (compared using ==)
     # ./longstring_spec.rb:5:in `block (2 levels) in <top (required)>'

The strings displayed by rspec are identical; the diff is in the truncated part. If the string is any shorter, the diff appears correctly.

Running on a Mac in iTerm2.

$ rspec --version
RSpec 3.7
  - rspec-core 3.7.1
  - rspec-expectations 3.7.0
  - rspec-mocks 3.7.0
  - rspec-support 3.7.1

About this issue

  • Original URL
  • State: open
  • Created 6 years ago
  • Reactions: 51
  • Comments: 27 (17 by maintainers)

Most upvoted comments

Ah, this is a dupe of https://github.com/rspec/rspec-expectations/issues/991.

The suggestion there is to set RSpec::Support::ObjectFormatter.default_instance.max_formatted_output_length to a high enough value.

Example

Say we have two strings of strings of length 11:

 str1 = "aaaaaaaaaaa"
 str2 = "aaaaaZaaaaa"

And the value of RSpec.Support.ObjectFormatter.max_formatted_output_length was set to 4

Current formatting

   expected: "aa...a"
        got: "aa...a"

I’m not a big fan of that.

Proposed formatting

Leading & traling ellipsis. You’d see something like this…

   expected: "...aaaZ..."
        got: "...aaaa..."

Here’s where the formatting is actually done:

https://github.com/rspec/rspec-support/blob/master/lib/rspec/support/object_formatter.rb#L32

I will probably try to work on this if we are ok with the scope of the task.

What I think could be nice to display is:

Failures:

  1) a long string shows the right part of the diff
     Failure/Error: expect(got).to eq(want)

       expected: "...aaaaaaaa..."
            got: "...aaaabaaa..."

       (compared using ==)
     # ./longstring_spec.rb:5:in `block (2 levels) in <top (required)>'

But I have few issues:

  • Is it limited to strings?
  • What happens when there are multiple differences?
  • Is it an option to set?

Scanning through the discussion here, there is one simple solution that is missing that would help me 95% of the time when I encounter this issue: Provide an option that when enabled just prints out the complete actual and expected strings, without diffing or trimming (and maybe add a switch to disable it by default if running an entire suite and enable by default if running a single spec). Usually I encounter these issues in data where context within the string t is important - especially when there are similar data sections. The semi-intelligent prettifying helpers are more of a hinderance than a help most of the time in my experience. Yes, having several screens full of data can be annoying, but having insert puts statements or jumping into byebug to inspect the strings in-situ is more annoying 😃

FWIW, I think RSpec badly needs to move away from using diff-lcs for diffing, and implement its own differ. diff-lcs works OK, but we convert everything to strings to diff, and we only get a line-by-line diff. This has several problems:

  • We aren’t able to get diffs of long single-line strings
  • Non-strings (e.g. arrays, hashes, etc) are only diffable if they get formatted as multi-line strings
  • The differ is only able to work off of string differences, but that doesn’t align with our matching semantics. For example, since we’ve supported composable matchers, it’s very common to have an actual value in a data structure and a matcher in the expected data structure. They have different string formatting (and thus show up as a difference) even if they match. Even putting aside composable matchers, it’s not true that equal objects always have the same string output, or that unequal objects have different string output.

A differ written for RSpec, that understands the semantics of RSpec matching, would produce far better output.

I know that’s a much larger task than just this issue, but it’s something I’ve been thinking about for a while, and was thinking could be a headline feature for RSpec 4.

If someone wants to experiment with this, it might be instructive to look at the differ that is included with Elixir’s ExUnit. It works quite well:

https://github.com/elixir-lang/elixir/blob/master/lib/ex_unit/lib/ex_unit/diff.ex

@benoittgt @kevinburkeomg Thanks for suggesting max_formatted_output_length.

Code to set max_formatted_output_length (docs):

RSpec.configure do |rspec|
  rspec.expect_with :rspec do |c|
    c.max_formatted_output_length = 200 # Set to nil to prevent RSpec from doing truncation
  end
end

Thanks to this StackOverflow answer by @pjmartorell.

@sriedel Did you try to use RSpec::Support::ObjectFormatter.default_instance.max_formatted_output_length ?

I have a draft version of the “new” object formater but it needs more work. I hope to send a proper version soon. ObjectFormater does a lot I need to understand what I need to keep from the original object_formater. Also I need to DRY the code with ObjectFormater.

At the same time I’m doing a small class about computer science at Harvard (essentially to learn C) and I have a final project. I can give a try to “port” ex_unit/diff to ruby (elixir test are instructive on how it works elixir-https://github.com/elixir-lang/elixir/blob/master/lib/ex_unit/test/ex_unit/diff_test.exs). I did some Elixir last year it could be a good project to end that class and validate my diploma.

This issue is about printing different strings even when truncated, increasing the length is a workaround but not the ideal fix

The problem we’re dealing in this ticket is that string diff is not displayed in its entirety. It is certainly not user-friendly, and I, personally, keep manually adding expectations.max_formatted_output_length = 1000 to spec/spec_helper.rb in projects I’m working on.

If we look at the spec/spec_helper.rb initializer, there is config for rspec-expectations there, even though rspec-core does not require use of rspec-expectations, and can be used without it:

  config.expect_with :rspec do |expectations|
    expectations.include_chain_clauses_in_custom_matcher_descriptions = true
  end

Would it be sufficiently user-friendly if the generated spec/spec_helper.rb already had the max_formatted_output_length option, at least to the default value?

We’ve actually added this option to the generated spec_helper.rb already in the not yet released RSpec 4 with this change:

RSpec.configure do |config|
  # rspec-expectations config goes here. You can use an alternate
  # assertion/expectation library such as wrong or minitest
  # assertions if you prefer.
  config.expect_with :rspec do |expectations|
    # The default output length is 200, you can increase it to make the
    # output more verbose, or decrease it to make it more concise.
    expectations.max_formatted_output_length = 200
  end

I might be biased due to professional deformation, but I think the current DSL for RSpec configuration is already user-friendly. Still open to suggestions.

A PR to improve the documentation for this option would be appreciated. Otherwise, is there any there anything actionable left here?

Sorry for the slow review here @benoittgt !

The ObjectFormatter is currently used in several ways, to format objects for descriptions etc, and to format them for expected / actual. I think this diffing formatter is worth having as an additional object (although if we could refactor it to share some code with the current formatter that’d be nice), which is then used when we know what expected / actual are. (See rspec-expectation matchers for how this is used). This could obviously then just be added as an additional peice, with tests demonstrating usage of course, and then we can switch the matchers to use this.

Is it limited to strings?

From the differs perspective everything is strings.

What happens when there are multiple differences?

If we get it to show at least one difference thats a big improvement, I wouldn’t worry so much about multiples.

Is it an option to set?

I think it should just happen

I’m fine with going with a simpler solution, as long as it improves things (as I expect it to!) and we can always iterate.