capybara: Chrome 134 breaks Selenium tests - intermittent failures with visit(visit_uri)

My test suite runs fine with Chrome 133 (Selenium 4.29.0-20250222).

Upgrading to Chrome 134 (Selenium 4.29.0-20250303) results in intermittent failures due to the session’s path not updating after a call to Capybara::Session#visit. In isolating the problem I did something like this:

...
visit '/foo/bar'
puts page.current_path # => '/' - i.e. path not updated
...

I’ve temporarily fixed the problem with a workaround which overrides Capybara::Session#visit like so in my test helper:

def visit(url)
    10.times do
      Capybara.current_session.visit url
      return if page.current_path == url

      sleep 0.1
    end
    abort 'Capybara::Session#visit broken after 1 second wait'
  end

Happy to provide a MRE if this is not a known problem.

Note this does not appear to be duplicate of #2793 as the problem is not observed with Chrome 133.

Ruby 3.4.2 Capybara 3.40.0 selenium-webdriver 4.29.1 (in Docker container) OS: LMDE faye (based on Debian bookworm)

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 25
  • Comments: 33 (7 by maintainers)

Commits related to this issue

Most upvoted comments

The ChromeDriver bug for the navigation race condition (click sometimes returns control before navigation has happened) is still open as of Chrome 135 release: https://issues.chromium.org/issues/402796660.

If you’re impacted by that one it might be worth voting on the ChromeDriver bug (there is a +1 button at the top of the tracker) for the ChromeDriver team to prioritize.

I was able to resolve our testing issues using the solution implemented in the PR referenced by @mike3985 above. We were already using the --no-sandbox option so the changes were implemented as follows:

     - name: Remove image-bundled Chrome
       run: sudo apt-get purge google-chrome-stable
     - name: Setup stable Chrome
       uses: browser-actions/setup-chrome@v1
       with:
         chrome-version: 128
         install-chromedriver: true
         install-dependencies: true

You can verify that the version changes were successful:

    - name: Print installed Chrome version
      run: chrome --product-version

You can verify that the headless browser driving your system specs is using the desired version by printing out the information from the browser capabilities hash:

    puts page.driver.browser.capabilities[:browser_name]
    puts page.driver.browser.capabilities[:browser_version]

The primary mechanism that Capybara uses to ensure the test process remains synchronized with the browser process is through the synchronize method here:

https://github.com/teamcapybara/capybara/blob/0480f90168a40780d1398c75031a255c1819dce8/lib/capybara/node/base.rb#L76-L103

AFAICS, there’s no mechanism that Selenium/Chromedriver can use to report back to Capybara that the page load is done so it just tries the next command and if it errors out in a certain way it will retry the command up until the max wait time. The list of exceptions that Capybara will retry for is in invalid_element_errors:

https://github.com/teamcapybara/capybara/blob/0480f90168a40780d1398c75031a255c1819dce8/lib/capybara/selenium/driver.rb#L297-L311

My working theory is that there’s a number of things happening here:

  1. Chrome/Chromedriver has changed how it returns control back after a navigation. This may be related to the removal of the old headless mode but supposedly that happened in Chrome 132. However, it could be that Chrome 134 has exacerbated things - we were commenting before its release that our system specs seemed more flakey than before.
  2. Selenium is now seeing errors that it didn’t see before which are raised as UnknownError - it uses the error string to map errors so anything it doesn’t know about will be an unknown error.
  3. The UnknownError exception isn’t in the list of exceptions in invalid_element_errors so what would’ve be retried before is now being raised.

This is the reasoning behind my suggestion to add UnknownError to the list of invalid_element_errors.

The suggestion to always assert on the response is a good one that’s always been true, however I’ve seen flakes where it was raising an unknown error exception on the assert because it hadn’t finished navigating and as a consequence I’ve been asserting the path as well since that doesn’t trigger the error. That’s fine when there’s a change of path but when you’re redirecting back to the same page it causes a problem so I’ve had to wrap those in a retry block of my own.

A combination of the above has at least made our test suite stable again but it’s not a long-term fix. Unfortunately don’t have time to dig deeper into this during this week but will look at the weekend if no-one has a full explanation.

@vknightbd I encountered something similar to what you described on some test suites.

I think newer Chrome versions are backgrounding pages more intensively in headless mode, and this can be worked around by setting a few options:

# Disable timers being throttled in background pages/tabs
browser_options.add_argument 'disable-background-timer-throttling'

# Normally, Chrome will treat a 'foreground' tab instead as backgrounded if the surrounding window is occluded (aka
# visually covered) by another window. This flag disables that.
browser_options.add_argument 'disable-backgrounding-occluded-windows'

# This disables non-foreground tabs from getting a lower process priority.
browser_options.add_argument 'disable-renderer-backgrounding'

Source: https://github.com/teamcapybara/capybara/issues/2796#issuecomment-2678172710

I suspect calling save_screenshot before a flaky assertion wakes up pages that headless Chrome is backgrounding. Also for me flaky behaviour disappeared when running non-headless.

The underlying cause seems to be errors like this:

Selenium::WebDriver::Error::UnknownError:
  unknown error: unhandled inspector error: {
    "code":-32000,"message":"Node with given id does not belong to the document"
  }

Depending on your code these can get swallowed up and failures manifest themselves in a variety of ways - for example I thought WebMock was failing with missing stubs but it was a request from the previous spec leaking into the next spec before stubs had been set. Similarly expecting on database records fails with not found errors or nil references.

Searching around I’ve seen some comments about the window handles list now contains the dev tools window and wonder whether the unhandled inspector error is a possible hint. In the meantime, it’s a hack, but adding Selenium::WebDriver::Error::UnknownError to the list of invalid element errors seems to help with our system specs

RSpec.configure do |config|
  config.before type: :system do |example|
    if page.driver.respond_to?(:invalid_element_errors)
      unless page.driver.invalid_element_errors.include?(Selenium::WebDriver::Error::UnknownError)
        page.driver.invalid_element_errors << Selenium::WebDriver::Error::UnknownError
      end
    end
  end
end

I’m experiencing the same issue.

I downgraded the following and all of my builds still fail, which makes me suspect there is a sub dependency that is causing the issue somewhere.

  • capybara
  • selenium-webdriver
  • Chrome
  • The Github workflow runner (tested ubuntu-22.04 and ubuntu-latest

I even tried to run a build from many weeks ago and that fails with the same errors.

Thanks to @hyodyllinentekniikka for pointing out capybara-lockstep, we have also just added it and it has seemingly fixed the issues we were having around this.

Thank you @eliotsykes !!! Adding browser_options.add_argument 'disable-backgrounding-occluded-windows' fixed my test issues that I had to use save_screenshot to hack around

Until I fully diagnosed the problem, I had success by downgrading all the way to Chrome 128, like this:

# application_system_test_case.rb
ENV["SE_CHROME_VERSION"] = "128"

Ultimately, I found that all the failures in my tests were due to legitimate race conditions. It seems that when using the latest version of Chrome, methods like click_button return control to the test faster, before the click has actually “happened”. I don’t think this is a bug, it’s just that timing is different.

I found https://github.com/teamcapybara/capybara/issues/2796 was helpful in solving our Chrome 134 related issues:

Specifically adding

browser_options.add_argument 'disable-backgrounding-occluded-windows'

What solved this for us was to circumvent the message center popups that warn you about leaked passwords.

For instance, this would fail:

ipad = create(:device, name: "ipad1", password: "123")
visit device_path(ipad)
fill_in "Password", with: "123"
click_on "Sign in"
expect(page).to have_content("Welcome")

Changing the dummy password to SüperSafeAndTotallyN07L34ked!20250523 fixed it.

Apparently the above mentioned problems have been resolved in Chrome 135.

Also, maybe using the Chrome switches under https://github.com/GoogleChrome/chrome-launcher/blob/main/docs/chrome-flags-for-tools.md, especially the ones about rendering background/occluded windows may help. We’re using the switches that you can see from here: https://github.com/Lombiq/UI-Testing-Toolbox/blob/23f4312d65169a8c84cc16765765d541c0058fc7/Lombiq.Tests.UI/Services/WebDriverFactory.cs#L153 And they seem to be useful; our tests since implementing all of these workarounds/config changes see very little flakyness.

(Not sure this is really a Capybara issue, but since people might find this thread while looking for information, might as well comment here.)

We’re experiencing the same issue. As mentioned earlier, downgrading Chromium/Chromedriver didn’t solve the issue, and we didn’t change anything in pull requests that were previously passing, so I guess there’s been a different change in GitHub Actions (which we use to run the tests) 🤔.

We’ve removed the few database calls we had, like the mentioned .count or .find_by, after starting the browser with a visit in the tests. In some cases, we’ve seen that there were ambiguous expectations (content we checked after doing an action was already present before that action). However, we’re still getting a dozen apparently random failures on every test run, and in many cases we couldn’t find anything wrong with either the failing test or the tests that were executed before the one which failed.

We’re seeing this too. In our case we’ve only seen it w/ clicks that submit a form. It also seems to be a race condition as to whether the click will return immediately or wait for navigation. The browser still waits for the page to load, so if the next action is another action that uses the browser (another click_on, has_content? ..., etc) then things will still work as the next action will still have to wait for the page to load from the previous click. However if the next action does something outside of the browser (like check the DB, Account.count in the example above), and click_on returned immediately then the action might happen before the next page load.

We can work around the issue by checking for a result in the browser after the form submission, before checking for side effects, but it’s still surprising to have inconsistent behavior.

We can make it very obvious by adding something like sleep 5 to the route handler, then we can see the click_on sometimes return after 5s and other times return immediately.

We’re using Selenium and Chromedriver and these are the logs of the click w/ a 5s delay:

selenium

2025-03-14 15:18:52 DEBUG Selenium [:command] -> POST session/faa1e3e078d1ec72a9223ae77b4cf4c5/element/f.1320DB0BC7087CA385C0B8D07C31977E.d.641C2A4E3A8AED4C38CDA2CE45316E7C.e.59/click 
2025-03-14 15:18:57 DEBUG Selenium [:header]    <<<  {"content-length"=>["14"], "content-type"=>["application/json; charset=utf-8"], "cache-control"=>["no-cache"]} 
2025-03-14 15:18:57 DEBUG Selenium [:command] <- {"value":null} 

chromedriver

[03-14-2025 15:18:52.307431][INFO]: [faa1e3e078d1ec72a9223ae77b4cf4c5] COMMAND ClickElement {
   "id": "f.1320DB0BC7087CA385C0B8D07C31977E.d.641C2A4E3A8AED4C38CDA2CE45316E7C.e.59"
}
[03-14-2025 15:18:52.307472][INFO]: Waiting for pending navigations...
[03-14-2025 15:18:52.307881][INFO]: Done waiting for pending navigations. Status: ok
[03-14-2025 15:18:52.341655][INFO]: Waiting for pending navigations...
[03-14-2025 15:18:57.504210][INFO]: Done waiting for pending navigations. Status: ok
[03-14-2025 15:18:57.504354][INFO]: [faa1e3e078d1ec72a9223ae77b4cf4c5] RESPONSE ClickElement

…and these are the logs when it returns immediately:

selenium

2025-03-14 15:17:28 DEBUG Selenium [:command] -> POST session/9e2f02a196c30375806e6b7b2b547382/element/f.6CEA5FEDD794621E4A99DCAF55FB1F49.d.905E6E26A4B9B078C5A6ED2D5EDF03B7.e.59/click 
2025-03-14 15:17:29 DEBUG Selenium [:header]    <<<  {"content-length"=>["14"], "content-type"=>["application/json; charset=utf-8"], "cache-control"=>["no-cache"]} 
2025-03-14 15:17:29 DEBUG Selenium [:command] <- {"value":null} 

chromium

[03-14-2025 15:17:28.968841][INFO]: [9e2f02a196c30375806e6b7b2b547382] COMMAND ClickElement {
   "id": "f.6CEA5FEDD794621E4A99DCAF55FB1F49.d.905E6E26A4B9B078C5A6ED2D5EDF03B7.e.59"
}
[03-14-2025 15:17:28.968896][INFO]: Waiting for pending navigations...
[03-14-2025 15:17:28.969347][INFO]: Done waiting for pending navigations. Status: ok
[03-14-2025 15:17:29.000385][INFO]: Waiting for pending navigations...
[03-14-2025 15:17:29.000834][INFO]: Done waiting for pending navigations. Status: ok
[03-14-2025 15:17:29.000899][INFO]: [9e2f02a196c30375806e6b7b2b547382] RESPONSE ClickElement

…based on those my guess is that the race is in Chrome or Chromedriver and Capybara/Selenium may not be responsible for waiting for navigation after the click.

EDIT: Possibly related bug report for Chrome/Chromedriver: https://issues.chromium.org/issues/402796660

It’s not only visit, we’ve been finding that every interaction that involves a page load will fail due to the browser not waiting for the next page load.

For example, this will fail:

assert_difference -> { Account.count } do
  click_button "Sign up"
end