capybara: Chrome 134 breaks Selenium tests - intermittent failures with visit(visit_uri)
My test suite runs fine with Chrome 133 (Selenium 4.29.0-20250222).
Upgrading to Chrome 134 (Selenium 4.29.0-20250303) results in intermittent failures due to the session’s path not updating after a call to Capybara::Session#visit. In isolating the problem I did something like this:
...
visit '/foo/bar'
puts page.current_path # => '/' - i.e. path not updated
...
I’ve temporarily fixed the problem with a workaround which overrides Capybara::Session#visit like so in my test helper:
def visit(url)
10.times do
Capybara.current_session.visit url
return if page.current_path == url
sleep 0.1
end
abort 'Capybara::Session#visit broken after 1 second wait'
end
Happy to provide a MRE if this is not a known problem.
Note this does not appear to be duplicate of #2793 as the problem is not observed with Chrome 133.
Ruby 3.4.2 Capybara 3.40.0 selenium-webdriver 4.29.1 (in Docker container) OS: LMDE faye (based on Debian bookworm)
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 25
- Comments: 33 (7 by maintainers)
Commits related to this issue
- try workaround from https://github.com/teamcapybara/capybara/issues/2800#issuecomment-2727634699 — committed to pachulo/nextcloud-snap by pachulo 2 years ago
- tests: workaround intermitent failing of integration tests The problem seems to be with the newest chromium 134. More context here: https://github.com/teamcapybara/capybara/issues/2800#issuecomment-2... — committed to pachulo/nextcloud-snap by pachulo 2 years ago
- Inline GovukTest config with --no-sandbox always The govuk_test gem only sets `--no-sandbox` if the `GOVUK_TEST_CHROME_NO_SANDBOX` environment variable is set. (And I haven't found any evidence of th... — committed to alphagov/signon by mike3985 2 years ago
- Add Chrome option --disable-backgrounding-occluded-windows Following a tip-off in the Capybara Selenium/Chrome issue thread https://github.com/teamcapybara/capybara/issues/2800#issuecomment-272750425... — committed to alphagov/signon by mike3985 2 years ago
- Use old Chrome in CI tests, with --no-sandbox flag Our JS-enabled MiniTest tests are failing fairly consistently on CI. We believe that this started when the most recent Github Actions ubuntu-latest ... — committed to alphagov/signon by mike3985 2 years ago
- Use old Chrome in CI tests, with --no-sandbox flag Our JS-enabled MiniTest tests are failing fairly consistently on CI. We believe that this started when the most recent Github Actions ubuntu-latest ... — committed to alphagov/signon by mike3985 2 years ago
- tests: workaround intermitent failing of integration tests The problem seems to be with the newest chromium 134. More context here: https://github.com/teamcapybara/capybara/issues/2800#issuecomment-2... — committed to pachulo/nextcloud-snap by pachulo 2 years ago
- Fix ci (#3765) * fix ci ## Context On March 12, the CI tests started failing. Pinning the chrome driver fixes this problem while we investigate. @aaronskiba offered https://github.com/teamcapybara... — committed to ualbertalib/jupiter by pgwillia 2 years ago
- Change Docker image to use older version of Chromium We are seeing intermittent failures when end to end tests are run locally. These failures are related to Selenium and chromedriver 134 [[1]]. They... — committed to alphagov/forms-e2e-tests by lfdebrux 2 years ago
- Try adding some chrome options to avoid recent capybara/selenium/chrome race conditions Such as `Node with given id does not belong to the document` These suggestions from: https://github.com/teamca... — committed to sciencehistory/scihist_digicoll by jrochkind 2 years ago
- Try adding some chrome options to avoid recent capybara/selenium/chrome race conditions Such as `Node with given id does not belong to the document` These suggestions from: https://github.com/teamca... — committed to sciencehistory/scihist_digicoll by jrochkind 2 years ago
- Try adding some chrome options to avoid recent capybara/selenium/chrome race conditions Such as `Node with given id does not belong to the document` These suggestions from: https://github.com/teamca... — committed to sciencehistory/scihist_digicoll by jrochkind 2 years ago
- Implement workaround for https://github.com/teamcapybara/capybara/issues/2800 Work-around implementation stolen from https://github.com/sciencehistory/scihist_digicoll/pull/2935 — committed to DARIAEngineering/dcaf_case_management by xmunoz 2 years ago
- Add Chrome option '--disable-backgrounding-occluded-windows' This should help fix some really ugly flaky test runs. See https://github.com/teamcapybara/capybara/issues/2800 for context. — committed to mamhoff/solidus by mamhoff 2 years ago
- Add Chrome option '--disable-backgrounding-occluded-windows' This should help fix some really ugly flaky test runs. See https://github.com/teamcapybara/capybara/issues/2800 for context. (cherry pic... — committed to solidusio/solidus by mamhoff 2 years ago
- Pin Chrome version used in integration tests There is an ongoing issue with recent versions of Chrome (see some discussion on this Capybara issue: https://github.com/teamcapybara/capybara/issues/28... — committed to alphagov/smart-answers by mtaylorgds 2 years ago
- Pin Chrome version used in integration tests There is an ongoing issue with recent versions of Chrome (see some discussion on this Capybara issue: https://github.com/teamcapybara/capybara/issues/28... — committed to alphagov/smart-answers by mtaylorgds 2 years ago
- Pin Chrome version used in integration tests There is an ongoing issue with recent versions of Chrome (see some discussion on this Capybara issue: https://github.com/teamcapybara/capybara/issues/28... — committed to alphagov/smart-answers by mtaylorgds 2 years ago
- Fix Capybara config for chromedriver 134+ https://github.com/teamcapybara/capybara/issues/2800 — committed to spree/spree by damianlegawiec 2 years ago
- Update capybara config for ChromeDriver 134+ (#12590) * Fix Capybara config for chromedriver 134+ https://github.com/teamcapybara/capybara/issues/2800 * Update capybara_config.rb — committed to spree/spree by damianlegawiec 2 years ago
The ChromeDriver bug for the navigation race condition (click sometimes returns control before navigation has happened) is still open as of Chrome 135 release: https://issues.chromium.org/issues/402796660.
If you’re impacted by that one it might be worth voting on the ChromeDriver bug (there is a +1 button at the top of the tracker) for the ChromeDriver team to prioritize.
I was able to resolve our testing issues using the solution implemented in the PR referenced by @mike3985 above. We were already using the
--no-sandboxoption so the changes were implemented as follows:You can verify that the version changes were successful:
You can verify that the headless browser driving your system specs is using the desired version by printing out the information from the browser capabilities hash:
The primary mechanism that Capybara uses to ensure the test process remains synchronized with the browser process is through the
synchronizemethod here:https://github.com/teamcapybara/capybara/blob/0480f90168a40780d1398c75031a255c1819dce8/lib/capybara/node/base.rb#L76-L103
AFAICS, there’s no mechanism that Selenium/Chromedriver can use to report back to Capybara that the page load is done so it just tries the next command and if it errors out in a certain way it will retry the command up until the max wait time. The list of exceptions that Capybara will retry for is in
invalid_element_errors:https://github.com/teamcapybara/capybara/blob/0480f90168a40780d1398c75031a255c1819dce8/lib/capybara/selenium/driver.rb#L297-L311
My working theory is that there’s a number of things happening here:
UnknownError- it uses the error string to map errors so anything it doesn’t know about will be an unknown error.UnknownErrorexception isn’t in the list of exceptions ininvalid_element_errorsso what would’ve be retried before is now being raised.This is the reasoning behind my suggestion to add
UnknownErrorto the list ofinvalid_element_errors.The suggestion to always assert on the response is a good one that’s always been true, however I’ve seen flakes where it was raising an unknown error exception on the assert because it hadn’t finished navigating and as a consequence I’ve been asserting the path as well since that doesn’t trigger the error. That’s fine when there’s a change of path but when you’re redirecting back to the same page it causes a problem so I’ve had to wrap those in a retry block of my own.
A combination of the above has at least made our test suite stable again but it’s not a long-term fix. Unfortunately don’t have time to dig deeper into this during this week but will look at the weekend if no-one has a full explanation.
@vknightbd I encountered something similar to what you described on some test suites.
I think newer Chrome versions are backgrounding pages more intensively in headless mode, and this can be worked around by setting a few options:
Source: https://github.com/teamcapybara/capybara/issues/2796#issuecomment-2678172710
I suspect calling
save_screenshotbefore a flaky assertion wakes up pages that headless Chrome is backgrounding. Also for me flaky behaviour disappeared when running non-headless.The underlying cause seems to be errors like this:
Depending on your code these can get swallowed up and failures manifest themselves in a variety of ways - for example I thought WebMock was failing with missing stubs but it was a request from the previous spec leaking into the next spec before stubs had been set. Similarly expecting on database records fails with not found errors or nil references.
Searching around I’ve seen some comments about the window handles list now contains the dev tools window and wonder whether the
unhandled inspector erroris a possible hint. In the meantime, it’s a hack, but addingSelenium::WebDriver::Error::UnknownErrorto the list of invalid element errors seems to help with our system specsI’m experiencing the same issue.
I downgraded the following and all of my builds still fail, which makes me suspect there is a sub dependency that is causing the issue somewhere.
ubuntu-22.04andubuntu-latestI even tried to run a build from many weeks ago and that fails with the same errors.
Thanks to @hyodyllinentekniikka for pointing out capybara-lockstep, we have also just added it and it has seemingly fixed the issues we were having around this.
Thank you @eliotsykes !!! Adding
browser_options.add_argument 'disable-backgrounding-occluded-windows'fixed my test issues that I had to usesave_screenshotto hack aroundUntil I fully diagnosed the problem, I had success by downgrading all the way to Chrome 128, like this:
Ultimately, I found that all the failures in my tests were due to legitimate race conditions. It seems that when using the latest version of Chrome, methods like
click_buttonreturn control to the test faster, before the click has actually “happened”. I don’t think this is a bug, it’s just that timing is different.I found https://github.com/teamcapybara/capybara/issues/2796 was helpful in solving our Chrome 134 related issues:
Specifically adding
What solved this for us was to circumvent the message center popups that warn you about leaked passwords.
For instance, this would fail:
Changing the dummy password to
SüperSafeAndTotallyN07L34ked!20250523fixed it.Apparently the above mentioned problems have been resolved in Chrome 135.
Also, maybe using the Chrome switches under https://github.com/GoogleChrome/chrome-launcher/blob/main/docs/chrome-flags-for-tools.md, especially the ones about rendering background/occluded windows may help. We’re using the switches that you can see from here: https://github.com/Lombiq/UI-Testing-Toolbox/blob/23f4312d65169a8c84cc16765765d541c0058fc7/Lombiq.Tests.UI/Services/WebDriverFactory.cs#L153 And they seem to be useful; our tests since implementing all of these workarounds/config changes see very little flakyness.
(Not sure this is really a Capybara issue, but since people might find this thread while looking for information, might as well comment here.)
We’re experiencing the same issue. As mentioned earlier, downgrading Chromium/Chromedriver didn’t solve the issue, and we didn’t change anything in pull requests that were previously passing, so I guess there’s been a different change in GitHub Actions (which we use to run the tests) 🤔.
We’ve removed the few database calls we had, like the mentioned
.countor.find_by, after starting the browser with avisitin the tests. In some cases, we’ve seen that there were ambiguous expectations (content we checked after doing an action was already present before that action). However, we’re still getting a dozen apparently random failures on every test run, and in many cases we couldn’t find anything wrong with either the failing test or the tests that were executed before the one which failed.We’re seeing this too. In our case we’ve only seen it w/ clicks that submit a form. It also seems to be a race condition as to whether the click will return immediately or wait for navigation. The browser still waits for the page to load, so if the next action is another action that uses the browser (another
click_on,has_content? ..., etc) then things will still work as the next action will still have to wait for the page to load from the previous click. However if the next action does something outside of the browser (like check the DB,Account.countin the example above), andclick_onreturned immediately then the action might happen before the next page load.We can work around the issue by checking for a result in the browser after the form submission, before checking for side effects, but it’s still surprising to have inconsistent behavior.
We can make it very obvious by adding something like
sleep 5to the route handler, then we can see theclick_onsometimes return after 5s and other times return immediately.We’re using Selenium and Chromedriver and these are the logs of the click w/ a 5s delay:
selenium
chromedriver
…and these are the logs when it returns immediately:
selenium
chromium
…based on those my guess is that the race is in Chrome or Chromedriver and Capybara/Selenium may not be responsible for waiting for navigation after the click.
EDIT: Possibly related bug report for Chrome/Chromedriver: https://issues.chromium.org/issues/402796660
It’s not only
visit, we’ve been finding that every interaction that involves a page load will fail due to the browser not waiting for the next page load.For example, this will fail: