selenium: [๐Ÿ› Bug]: Selenium 4.32.0 ruby bindings fail to start firefox in parallel testing

Description

Selenium 4.32.0 attempts to address issues with Firefox 136 and later failing when more that one instance is started (see https://github.com/mozilla/geckodriver/issues/2218 for details) by allocating a unique bidi port for each instance in #15458.

That works fine if all the instances are started in a single thread but fails if two threads are starting instances simultaneously as PortProber works by allocating and then immediately closing a port leaving another thread free to do the same thing in the gap before firefox starts and binds it.

Reproducible Code

Not available

Debugging Logs

I added puts command.inspect to the build_process method in ServiceManager in order to see what processes were being started and then ran my test suite single threaded:

% PARALLEL_WORKERS=1 bundle exec rails test:system

Run options: --seed 22623

# Running:

["/usr/bin/geckodriver", "--port=4444", "--websocket-port", "9223"]
.["/usr/bin/geckodriver", "--port=4445", "--websocket-port", "9224"]
............................................................["/usr/bin/geckodriver", "--port=4446", "--websocket-port", "9225"]
...................................................................................................................

Finished in 110.831997s, 1.5880 runs/s, 7.3084 assertions/s.
176 runs, 810 assertions, 0 failures, 0 errors, 0 skips
Coverage report generated for Minitest to /home/tom/src/osm/rails/coverage.
Line Coverage: 46.57% (3587 / 7702)
Lcov style coverage report generated for Minitest to coverage/lcov.info

and then I repeated the same with multiple threads:

% bundle exec rails test:system
Running 176 tests in parallel using 24 processes
Run options: --seed 57887

# Running:

["/usr/bin/geckodriver", "--port=4444", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4445", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4446", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4447", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4448", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4449", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4450", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4451", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4452", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4453", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4454", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4455", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4456", "--websocket-port", "9225"]
.["/usr/bin/geckodriver", "--port=4457", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4458", "--websocket-port", "9225"]
.["/usr/bin/geckodriver", "--port=4459", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4460", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4461", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4462", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4463", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4464", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4465", "--websocket-port", "9225"]
.["/usr/bin/geckodriver", "--port=4466", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4467", "--websocket-port", "9225"]
.....E

Error:
IndexTest#test_should_not_add_overlay_when_not_closing_right_menu_popup:
Selenium::WebDriver::Error::NoSuchWindowError: Browsing context has been discarded
    test/system/index_test.rb:26:in 'block in <class:IndexTest>'

As you can see in the second case multiple instances were started using the same bidi port and tests then started failing en mass as most of the firefox instances had failed to start.


โ„น๏ธ Last known working version: 4.31.0 with firefox < 136 or a monkey patch to add `--websocket-port=0`

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (16 by maintainers)

Most upvoted comments

I gave up trying to get anything to run locally and Iโ€™ve opened #15727 and weโ€™ll see what GHA makes of the testsโ€ฆ

Using port 0 would work for desktop but could cause issues for Android given that geckodriver uses adb to forward/reverse ports and for HTTP based sockets it requires the same port number for the device and host as explained in https://bugzilla.mozilla.org/show_bug.cgi?id=1754016.

We do not have a solution for that yet.