selenium: [🐛 Bug]: Cannot set read timeout value in Local Drivers

What happened?

Despite setting high timeouts (“{implicit: 10000000, pageLoad: 10000000, script: 10000000}”), calling executeAsyncScript will occasionally time out after exactly 3 minutes. This is using 4.5.0 of Selenium and Java 17 with Firefox 105 on Ubuntu 18. It’s also using selenium-http-jdk-client.

How can we reproduce the issue?

Download https://github.com/TolstoyDotCom/more-speech and follow the instructions in the "2.1 Notes" section. You might need to use different handles and different numbers of pages to see the issue.

When you click 'Check timeline', the app gets some tweets from the user's profile page then visits each tweet's page.

On an individual tweet page, com.tolstoy.censorship.twitter.checker.app.webdriver.WebDriverFactoryNT first calls executeAsyncScript to load the tweets using the scripts in the stockscripts directory (jquery + suededenim + tweet_retriever).

Then, com.tolstoy.censorship.twitter.checker.app.browserproxy.BrowserProxyHAR calls executeAsyncScript using (jquery + suededenim + har_retriever).

Then, WebDriverFactoryNT calls executeAsyncScript using (jquery + suededenim + json_parser).

So, three calls to executeAsyncScript per page and if one of them takes longer than 3 minutes (likely if there are lots of tweets), it will probably time out.

Relevant log output

[INFO ] 2022-Oct-21 11:45:18 AM c.t.c.t.c.a.h.SearchRunTimelineBuilder - Loading reply page [...]
[...]
[ERROR] 2022-Oct-21 11:48:18 AM c.t.c.t.c.a.h.SearchRunTimelineBuilder - error logging in or building searchRun org.openqa.selenium.TimeoutException: java.util.concurrent.TimeoutException
Build info: version: '4.5.0', revision: 'fe167b119a'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-125-generic', java.version: '17.0.4'
Driver info: org.openqa.selenium.firefox.FirefoxDriver
Command: [c57d3478-3d94-41ae-a991-f4792364f495, get {url=https://twitter.com/[...]}]
Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 105.0, moz:accessibilityChecks: false, moz:buildID: 20220915150737, moz:debuggerAddress: 127.0.0.1:36466, moz:geckodriverVersion: 0.32.0, moz:headless: false, moz:platformVersion: 5.4.0-125-generic, moz:processID: 6398, moz:profile: /tmp/rust_mozprofileAhZ2z2, moz:shutdownTimeout: 60000, moz:useNonSpecCompliantPointerOrigin: false, moz:webdriverClick: true, moz:windowless: false, pageLoadStrategy: normal, platformName: LINUX, proxy: Proxy(), se:cdp: ws://127.0.0.1:36466/devtoo..., se:cdpVersion: 85.0, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 10000000, pageLoad: 10000000, script: 10000000}, unhandledPromptBehavior: dismiss}
Session ID: c57d3478-3d94-41ae-a991-f4792364f495
	at org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:65)
	at org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)
	at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
	at org.openqa.selenium.remote.http.netty.NettyHttpHandler.execute(NettyHttpHandler.java:49)
	at org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)
	at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
	at org.openqa.selenium.remote.http.netty.NettyClient.execute(NettyClient.java:99)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:181)
	at org.openqa.selenium.remote.service.DriverCommandExecutor.invokeExecute(DriverCommandExecutor.java:167)
	at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:142)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:547)
	at org.openqa.selenium.remote.RemoteWebDriver.get(RemoteWebDriver.java:310)
	at com.tolstoy.censorship.twitter.checker.app.helpers.SearchRunTimelineBuilder.getIndividualPage(SearchRunTimelineBuilder.java:353)
	at com.tolstoy.censorship.twitter.checker.app.helpers.SearchRunTimelineBuilder.getIndividualPages(SearchRunTimelineBuilder.java:323)
	at com.tolstoy.censorship.twitter.checker.app.helpers.SearchRunTimelineBuilder.buildSearchRunTimelineInternal(SearchRunTimelineBuilder.java:282)
	at com.tolstoy.censorship.twitter.checker.app.helpers.SearchRunTimelineBuilder.buildSearchRunTimeline(SearchRunTimelineBuilder.java:185)
	at com.tolstoy.censorship.twitter.checker.app.AppGUI$TimelineWorker.doInBackground(AppGUI.java:210)
	at com.tolstoy.censorship.twitter.checker.app.AppGUI$TimelineWorker.doInBackground(AppGUI.java:187)
	at java.desktop/javax.swing.SwingWorker$1.call(SwingWorker.java:304)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.desktop/javax.swing.SwingWorker.run(SwingWorker.java:343)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.util.concurrent.TimeoutException
	at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
	at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:206)
	at org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:59)
	... 23 more

Operating System

Ubuntu 18

Selenium version

4.5.0 Java

What are the browser(s) and version(s) where you see this issue?

Firefox 105

What are the browser driver(s) and version(s) where you see this issue?

webdrivermanager 5.0.3

Are you using Selenium Grid?

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 33 (26 by maintainers)

Commits related to this issue

re #11158 - Added overriden constructors to have configurable connect and read timeouts. — committed to code-with-abdullah/selenium by code-with-abdullah a year ago
Allow timeout config to be set for local drivers Fixes #11158 — committed to krmahadevan/selenium by krmahadevan a year ago
Allow timeout config to be set for local drivers Fixes #11158 — committed to krmahadevan/selenium by krmahadevan a year ago
Allow specifying timeouts Partially Fixes #11158 The client side read and connection timeouts are Currently hard coded and there’s no way to change them. While we are working on creating a proper ... — committed to krmahadevan/selenium by krmahadevan a year ago
Allow timeout config to be set for local drivers Fixes #11158 — committed to krmahadevan/selenium by krmahadevan a year ago
Allow timeout config to be set for local drivers Fixes #11158 — committed to krmahadevan/selenium by krmahadevan a year ago
Allow timeout config to be set for local drivers Fixes #11158 — committed to krmahadevan/selenium by krmahadevan a year ago
Allow timeout config to be set for local drivers Fixes #11158 — committed to krmahadevan/selenium by krmahadevan a year ago
[java] have overloaded constructors to have configurable connect and read timeouts while instantiating WebDriver object (#11532) * re #11158 - Added overriden constructors to have configurable connec... — committed to SeleniumHQ/selenium by code-with-abdullah a year ago

Most upvoted comments

@krmahadevan I left comments in the PR. I think overall it looks good, however it’d be nice to help the person who submitted the initial PR with the idea of having an extra constructor.

Absolutely @diemol . That’s the way to encourage everyone to be part of Selenium. My PR will be in Draft only and I will get rid of it, once the original PR submitted by one of our users gets merged.

krmahadevan on Jan 16, 2023

Ok, please ignore. That was a rush! That wont work. There’s a lot to it than what meets the eye 😃

krmahadevan on Jan 12, 2023

@titusfortner @diemol - Can you please check if this is something that can be done for fixing this ?

https://github.com/SeleniumHQ/selenium/pull/11537

This PR is not ready to be merged because I have just disabled a test that checks if a client config was specified for a local driver and if yes, then throw an exception and bail out.

It still needs a test that checks if the client config when specified for a local driver is honoured.

The flip side of this PR is basically that, when a user needs to specify a client config to a local driver, they would need to work with WebDriver interface and not with an actual driver flavor because the underlying driver after creation is augmented post creation.

A code sample would look like below

   @Test
  public void hello() {
    WebDriver driver = null;
    try {
      ClientConfig cfg = ClientConfig.defaultConfig().readTimeout(Duration.ofMinutes(1))
        .connectionTimeout(Duration.ofMinutes(1));
      driver = RemoteWebDriver.builder()
        .config(cfg)
        .oneOf(new ChromeOptions())
        .build();
    driver.get("https://www.google.com");
    } catch (Exception e) {
      if (driver != null) {
        driver.quit();
      }
    }
  }

@code-with-abdullah - FYI

krmahadevan on Jan 12, 2023

@krmahadevan Appreciate the detailed suggestions. It can probably be a discussion topic for tomorrow’s TLC meeting to identify the best approach.

pujagani on Jan 4, 2023

I think it is OK it is not a ChromeDriver instance, as I have seen the need to set a timeout locally to be rare. I agree with the second point, passing a default value is not ideal. Would be nice to change that and add docs about setting a timeout for the remote and local use case.

diemol on Jan 3, 2023