selenium: [🐛 Bug]: Memory leak running hub, due to HttpClient-x-SelectorManager threads never closed

What happened?

Hello

On our hub (version 4.15.0, JDK 11.0.20), running 13 nodes, memory increase continuously as tests run. Attaching a VisualVM shows that it seems to be caused by an increasing number of HttpClient-x-SelectorManager threads that are not closed (seems similar to https://github.com/SeleniumHQ/selenium/issues/12206 or https://github.com/SeleniumHQ/selenium/issues/11336)

Operations are normal, except when max memory is reached. In this case, hub does not respond anymore

In the graphic above, you can see that 300 more threads have been created in an hour

Is there a way (to help) to track down the creator of the staled threads ? Thread dump doesn’t help much


"HttpClient-111-SelectorManager" - Thread t@756
   java.lang.Thread.State: RUNNABLE
        at java.base@11.0.20.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at java.base@11.0.20.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:357)
        at java.base@11.0.20.1/sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:182)
        at java.base@11.0.20.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        - locked <2eebf6a6> (a sun.nio.ch.Util$2)
        - locked <4a436b2f> (a sun.nio.ch.WindowsSelectorImpl)
        at java.base@11.0.20.1/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
        at platform/java.net.http@11.0.20.1/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:867)

   Locked ownable synchronizers:
        - None

Thanks

How can we reproduce the issue?

Start a hub
Start some nodes
Execute many tests. I think the more tests are run, the easier it's to point the issue

Relevant log output

no relevant logs available on hub

Operating System

Windows / Linux

Selenium version

4.11.0

What are the browser(s) and version(s) where you see this issue?

any browser

What are the browser driver(s) and version(s) where you see this issue?

any browser

Are you using Selenium Grid?

4.15.0

About this issue

Original URL
State: open
Created 7 months ago
Reactions: 2
Comments: 28 (27 by maintainers)

Commits related to this issue

work around https://github.com/SeleniumHQ/selenium/issues/13246 — committed to bhecquet/seleniumRobot by bhecquet 6 months ago
workaround https://github.com/SeleniumHQ/selenium/issues/13246 — committed to bhecquet/seleniumRobot by bhecquet 6 months ago

Most upvoted comments

I was talking about the “HttpClient-xx-SelectorManager” threads on hub

bhecquet on Dec 19, 2023

This should be the one for the CdpEndpointFinder to get the websocket url of the DevTools, before the driver is created.

joerg1985 on Dec 15, 2023

Thank you! But that is an error situation which involves enabling BiDi and using CDP. I agree that needs to be solved too. But I understand the issue here is mainly that threads are leaking when tests are running without any errors or I will know better once the example is shared.

pujagani on Dec 15, 2023

I can check this in a day or two. I am happy to help, though you are the expert in this area. I might ask questions as and when I need the guidance.

pujagani on Dec 7, 2023

Okay one thing that should be checked, we have chain like this:

Client <---- WS ----> Grid <---- WS ----> Node <----- WS ------> Driver

Now the Driver does raise an error on the Websocket

Client <---- WS ----> Grid <---- WS ----> Node * disconnected * Driver

I don’t think the other connections are also disconnected in this case. To have an error from the driver site is alarming easy, this code will provoke it on my system:

var options = new ChromeOptions();        
options.setCapability("webSocketUrl", true); // <- bidi must be enabled to run into errors of the devtools connection
var driver = new ChromeDriver(options);

driver.close(); // <- runs into onError of the devtools connection, not the bidi connection
driver.quit();

@pujagani would you like to check this or should i give it a try at a undefined point in the future? I think the fix for this would be to send a close message to the maybeHandler inside the WebSocketUpgradeHandler.exceptionCaught call, but this is more a educated guess.

joerg1985 on Dec 6, 2023