selenium: [šŸ› Bug]: Distributor Failing to Register Nodes due to Exception in EventListener

What happened?

Deployment Details: Type: Fully Distributed Node - 4.19.0 Distributor - 4.21.0 Router - 4.21.0 Smap - 4.21.0 Squeue - 4.21.0

Selelenium 4 Grid was running ~smoothly (for several days) and I then started receiving reports of SessionNotCreatedExceptions. I then looked in NewRelic and saw that the Queue was rather large and there were no running tests. Node Logs indicated that the registration event was being sent over and over.

16:22:46.554 INFO [NodeServer.execute] - Started Selenium node 4.19.0 (revision 5f9cec8963): http://10.28.177.94:5555 16:22:46.584 INFO [NodeServer$1.lambda$start$1] - Sending registration event… 16:22:56.593 INFO [NodeServer$1.lambda$start$1] - Sending registration event… 16:23:06.600 INFO [NodeServer$1.lambda$start$1] - Sending registration event… 16:23:16.604 INFO [NodeServer$1.lambda$start$1] - Sending registration event… 16:23:26.608 INFO [NodeServer$1.lambda$start$1] - Sending registration event… 16:23:36.614 INFO [NodeServer$1.lambda$start$1] - Sending registration event… 16:23:46.619 INFO [NodeServer$1.lambda$start$1] - Sending registration event… 16:23:56.623 INFO [NodeServer$1.lambda$start$1] - Sending registration event… 16:24:06.630 INFO [NodeServer$1.lambda$start$1] - Sending registration event… 16:24:16.634 INFO [NodeServer$1.lambda$start$1] - Sending registration event… 16:24:26.638 INFO [NodeServer$1.lambda$start$1] - Sending registration event… 16:24:36.642 INFO [NodeServer$1.lambda$start$1] - Sending registration event… 16:24:46.556 INFO [NodeServer$1.lambda$start$1] - Sending registration event…

Distributor Logs indicated that the Nodes did not have the sessiontimeout set

How can we reproduce the issue?

I am not entirely sure how to reproduce as the distributor is complaining that a configuration setting is not set (but is clearly located both in the config.toml file and as an argument in the java process running on the node container).

Relevant log output

Log from Distributor (4.21):

17:37:45.983 WARN [UnboundZmqEventBus$PollingRunnable.lambda$notifyListeners$2] - Caught exception from listener: org.openqa.selenium.events.EventListener@65ab8c00
java.lang.IllegalArgumentException: Session timeout must be set
	at org.openqa.selenium.internal.Require.positive(Require.java:99)
	at org.openqa.selenium.grid.node.Node.<init>(Node.java:129)
	at org.openqa.selenium.grid.node.remote.RemoteNode.<init>(RemoteNode.java:91)
	at org.openqa.selenium.grid.distributor.local.LocalDistributor.register(LocalDistributor.java:317)
	at org.openqa.selenium.grid.distributor.local.LocalDistributor.lambda$new$4(LocalDistributor.java:211)
	at org.openqa.selenium.events.EventListener.accept(EventListener.java:42)
	at org.openqa.selenium.events.EventListener.accept(EventListener.java:24)
	at org.openqa.selenium.events.zeromq.UnboundZmqEventBus$PollingRunnable.lambda$notifyListeners$2(UnboundZmqEventBus.java:319)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

Log from Node:

docker exec node ps aux | grep java
seluser    172  0.5  3.9 3597236 155940 ?      Sl   16:22   0:23 java -Dwebdriver.chrome.driver=/usr/bin/chromedriver -Dwebdriver.edge.driver=/usr/bin/msedgedriver -Dwebdriver.gecko.driver=/usr/bin/geckodriver -jar /opt/selenium/selenium-server.jar node --bind-host false --config /opt/selenium/config.toml --log-level CONFIG --session-timeout 600 --heartbeat-period 30
[root@selenium4-node-linux-10-28-177-94 inin]# docker exec node cat /opt/selenium/config.toml
[events]
publish = "tcp://selenium4-distributor-internal-ip.devdomain.com:4442"  
subscribe = "tcp://selenium4-distributor-internal-ip.devdomain.com:4443"

[server]
host = "10.28.177.94"
[node]
session-timeout = 600
override-max-sessions = false
detect-drivers = false
drain-after-session-count = 1
enable-managed-downloads = true
max-sessions = 1

[[node.driver-configuration]]
display-name = "chrome"
stereotype = '{"browserName": "chrome", "browserVersion": "124.0", "platformName": "Linux", "goog:chromeOptions": {"binary": "/usr/bin/google-chrome"}}'
max-sessions = 1
    
[[node.driver-configuration]]
display-name = "firefox"
stereotype = '{"browserName": "firefox", "browserVersion": "126.0", "platformName": "Linux"}'
max-sessions = 1
    
[[node.driver-configuration]]
display-name = "edge"
stereotype = '{"browserName": "MicrosoftEdge", "browserVersion": "125.0", "platformName": "Linux"}'
max-sessions = 1

Operating System

Amazon Linux 2

Selenium version

Java 4.21.0

What are the browser(s) and version(s) where you see this issue?

n/a

What are the browser driver(s) and version(s) where you see this issue?

n/a

Are you using Selenium Grid?

Yes - 4.0

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 20 (6 by maintainers)

Most upvoted comments

My apologies guys; I was testing this out in our dev environment, and there was an old node that was continuously trying to connect to the distributor (and that node was on 4.09). After stopping this node and scaling our Linux Nodes ASG back up, I am no longer seeing the timeout traces. As an aside, I did attempt to add the --session-request-timeout argument to the 4.09 node and still received the message from the distributor. Also, I still think it’d be very helpful for traces that are about a specific node, to have some identifying information in the trace, so that you know which node is causing the problem.

There are no old nodes connected to the distributor (I can check grid status, but my nodes are in an ASG and when the old ASG is terminated, all old nodes are terminated). The nodes that were built with version 4.21.0-20240517 no longer have the repeated ā€œSending registration eventā€¦ā€ traces, but I am still seeing the ā€œjava.lang.IllegalArgumentException: Session timeout must be setā€ from the distributor. I also use a centralized logging solution (Sumo) and typically have ~150 nodes running tens of thousands of tests daily. Node containers are terminated after a single session, and thus the new node will register and we see this trace on the distributor. If I were to deploy this change to our production grid, we’d be seeing tens of thousands of these stack traces every day. I will scale down to one node and run some tests to see if any behavior changes occur.