selenium: [š Bug]: Distributor Failing to Register Nodes due to Exception in EventListener
What happened?
Deployment Details: Type: Fully Distributed Node - 4.19.0 Distributor - 4.21.0 Router - 4.21.0 Smap - 4.21.0 Squeue - 4.21.0
Selelenium 4 Grid was running ~smoothly (for several days) and I then started receiving reports of SessionNotCreatedExceptions. I then looked in NewRelic and saw that the Queue was rather large and there were no running tests. Node Logs indicated that the registration event was being sent over and over.
16:22:46.554 INFO [NodeServer.execute] - Started Selenium node 4.19.0 (revision 5f9cec8963): http://10.28.177.94:5555 16:22:46.584 INFO [NodeServer$1.lambda$start$1] - Sending registration event⦠16:22:56.593 INFO [NodeServer$1.lambda$start$1] - Sending registration event⦠16:23:06.600 INFO [NodeServer$1.lambda$start$1] - Sending registration event⦠16:23:16.604 INFO [NodeServer$1.lambda$start$1] - Sending registration event⦠16:23:26.608 INFO [NodeServer$1.lambda$start$1] - Sending registration event⦠16:23:36.614 INFO [NodeServer$1.lambda$start$1] - Sending registration event⦠16:23:46.619 INFO [NodeServer$1.lambda$start$1] - Sending registration event⦠16:23:56.623 INFO [NodeServer$1.lambda$start$1] - Sending registration event⦠16:24:06.630 INFO [NodeServer$1.lambda$start$1] - Sending registration event⦠16:24:16.634 INFO [NodeServer$1.lambda$start$1] - Sending registration event⦠16:24:26.638 INFO [NodeServer$1.lambda$start$1] - Sending registration event⦠16:24:36.642 INFO [NodeServer$1.lambda$start$1] - Sending registration event⦠16:24:46.556 INFO [NodeServer$1.lambda$start$1] - Sending registration eventā¦
Distributor Logs indicated that the Nodes did not have the sessiontimeout set
How can we reproduce the issue?
I am not entirely sure how to reproduce as the distributor is complaining that a configuration setting is not set (but is clearly located both in the config.toml file and as an argument in the java process running on the node container).
Relevant log output
Log from Distributor (4.21):
17:37:45.983 WARN [UnboundZmqEventBus$PollingRunnable.lambda$notifyListeners$2] - Caught exception from listener: org.openqa.selenium.events.EventListener@65ab8c00
java.lang.IllegalArgumentException: Session timeout must be set
at org.openqa.selenium.internal.Require.positive(Require.java:99)
at org.openqa.selenium.grid.node.Node.<init>(Node.java:129)
at org.openqa.selenium.grid.node.remote.RemoteNode.<init>(RemoteNode.java:91)
at org.openqa.selenium.grid.distributor.local.LocalDistributor.register(LocalDistributor.java:317)
at org.openqa.selenium.grid.distributor.local.LocalDistributor.lambda$new$4(LocalDistributor.java:211)
at org.openqa.selenium.events.EventListener.accept(EventListener.java:42)
at org.openqa.selenium.events.EventListener.accept(EventListener.java:24)
at org.openqa.selenium.events.zeromq.UnboundZmqEventBus$PollingRunnable.lambda$notifyListeners$2(UnboundZmqEventBus.java:319)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Log from Node:
docker exec node ps aux | grep java
seluser 172 0.5 3.9 3597236 155940 ? Sl 16:22 0:23 java -Dwebdriver.chrome.driver=/usr/bin/chromedriver -Dwebdriver.edge.driver=/usr/bin/msedgedriver -Dwebdriver.gecko.driver=/usr/bin/geckodriver -jar /opt/selenium/selenium-server.jar node --bind-host false --config /opt/selenium/config.toml --log-level CONFIG --session-timeout 600 --heartbeat-period 30
[root@selenium4-node-linux-10-28-177-94 inin]# docker exec node cat /opt/selenium/config.toml
[events]
publish = "tcp://selenium4-distributor-internal-ip.devdomain.com:4442"
subscribe = "tcp://selenium4-distributor-internal-ip.devdomain.com:4443"
[server]
host = "10.28.177.94"
[node]
session-timeout = 600
override-max-sessions = false
detect-drivers = false
drain-after-session-count = 1
enable-managed-downloads = true
max-sessions = 1
[[node.driver-configuration]]
display-name = "chrome"
stereotype = '{"browserName": "chrome", "browserVersion": "124.0", "platformName": "Linux", "goog:chromeOptions": {"binary": "/usr/bin/google-chrome"}}'
max-sessions = 1
[[node.driver-configuration]]
display-name = "firefox"
stereotype = '{"browserName": "firefox", "browserVersion": "126.0", "platformName": "Linux"}'
max-sessions = 1
[[node.driver-configuration]]
display-name = "edge"
stereotype = '{"browserName": "MicrosoftEdge", "browserVersion": "125.0", "platformName": "Linux"}'
max-sessions = 1
Operating System
Amazon Linux 2
Selenium version
Java 4.21.0
What are the browser(s) and version(s) where you see this issue?
n/a
What are the browser driver(s) and version(s) where you see this issue?
n/a
Are you using Selenium Grid?
Yes - 4.0
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 20 (6 by maintainers)
My apologies guys; I was testing this out in our dev environment, and there was an old node that was continuously trying to connect to the distributor (and that node was on 4.09). After stopping this node and scaling our Linux Nodes ASG back up, I am no longer seeing the timeout traces. As an aside, I did attempt to add the --session-request-timeout argument to the 4.09 node and still received the message from the distributor. Also, I still think itād be very helpful for traces that are about a specific node, to have some identifying information in the trace, so that you know which node is causing the problem.
There are no old nodes connected to the distributor (I can check grid status, but my nodes are in an ASG and when the old ASG is terminated, all old nodes are terminated). The nodes that were built with version 4.21.0-20240517 no longer have the repeated āSending registration eventā¦ā traces, but I am still seeing the ājava.lang.IllegalArgumentException: Session timeout must be setā from the distributor. I also use a centralized logging solution (Sumo) and typically have ~150 nodes running tens of thousands of tests daily. Node containers are terminated after a single session, and thus the new node will register and we see this trace on the distributor. If I were to deploy this change to our production grid, weād be seeing tens of thousands of these stack traces every day. I will scale down to one node and run some tests to see if any behavior changes occur.