selenium: [🐛 Bug]: Possible memory leak involving hub and chromium-based nodes
What happened?
We run Selenium Grid Hub and Nodes via Docker on our server. Frequently (about once every 2-3 days) we see that the hub displays it’s status as “unhealthy”, and our server RAM usage spikes close to 100%. Additionally, the chrome/edge node containers show as using way more storage than the rest of the containers, typically on the scale of gigabytes.
Restarting the containers does not resolve the issue, but stopping/killing/starting the containers does. We leave our containers running for days/weeks at a time, and this has never been an issue prior to about 3 months ago.
How can we reproduce the issue?
We have not been able to consistently reproduce the issue. However, it seems to somewhat consistently manifest when we induce a heavy test load on the grid.
Relevant log output
12:23:35.702 INFO [LocalDistributor.newSession] - Unable to find a free slot for request 86dbd876-1306-49a3-922d-edb8e36b7e1e.
Capabilities {browserName: chrome, goog:chromeOptions: {args: [user-agent=Chrome <app-name>/2.12.6]}, goog:loggingPrefs: {browser: INFO, performance: INFO}, pageLoadStrategy: eager, platformName: linux, proxy: {httpProxy: fwd_proxy_all:3128, proxyType: manual, sslProxy: fwd_proxy_all:3128}}
12:23:35.702 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "11ec0e7f003a46e8793ec9784fffa8e5","eventTime": 1694348615685052940,"eventName": "Session request received by the Distributor","attributes": {"logger": "org.openqa.selenium.grid.distributor.local.LocalDistributor","request.payload": "[Capabilities {browserName: chrome, goog:chromeOptions: {args: [user-agent=Chrome <app-name>\u002f2.12.6]}, goog:loggingPrefs: {browser: INFO, performance: INFO}, pageLoadStrategy: eager, platformName: linux, proxy: {httpProxy: fwd_proxy_all:3128, proxyType: manual, sslProxy: fwd_proxy_all:3128}}]"}}
12:23:35.762 WARN [SeleniumSpanExporter$1.lambda$export$1] - Will retry session 86dbd876-1306-49a3-922d-edb8e36b7e1e
12:23:35.762 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "11ec0e7f003a46e8793ec9784fffa8e5","eventTime": 1694348615701163474,"eventName": "exception","attributes": {"exception.message": "Will retry session 86dbd876-1306-49a3-922d-edb8e36b7e1e","logger": "org.openqa.selenium.grid.distributor.local.LocalDistributor","request.payload": "[Capabilities {browserName: chrome, goog:chromeOptions: {args: [user-agent=Chrome <app-name>\u002f2.12.6]}, goog:loggingPrefs: {browser: INFO, performance: INFO}, pageLoadStrategy: eager, platformName: linux, proxy: {httpProxy: fwd_proxy_all:3128, proxyType: manual, sslProxy: fwd_proxy_all:3128}}]"}}
12:23:35.705 INFO [LocalDistributor.newSession] - Unable to find a free slot for request 62952ed8-2e68-44a0-b3a8-635c054537f4.
Capabilities {browserName: chrome, goog:chromeOptions: {args: [user-agent=Chrome <app-name>/2.12.6]}, goog:loggingPrefs: {browser: INFO, performance: INFO}, pageLoadStrategy: eager, platformName: linux, proxy: {httpProxy: fwd_proxy_all:3128, proxyType: manual, sslProxy: fwd_proxy_all:3128}}
12:23:35.765 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "4687cba99b98a49fa32963a6cf4c26c5","eventTime": 1694348615689081642,"eventName": "Session request received by the Distributor","attributes": {"logger": "org.openqa.selenium.grid.distributor.local.LocalDistributor","request.payload": "[Capabilities {browserName: chrome, goog:chromeOptions: {args: [user-agent=Chrome <app-name>\u002f2.12.6]}, goog:loggingPrefs: {browser: INFO, performance: INFO}, pageLoadStrategy: eager, platformName: linux, proxy: {httpProxy: fwd_proxy_all:3128, proxyType: manual, sslProxy: fwd_proxy_all:3128}}]"}}
12:23:35.765 WARN [SeleniumSpanExporter$1.lambda$export$1] - Will retry session 62952ed8-2e68-44a0-b3a8-635c054537f4
12:23:35.765 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "4687cba99b98a49fa32963a6cf4c26c5","eventTime": 1694348615765110736,"eventName": "exception","attributes": {"exception.message": "Will retry session 62952ed8-2e68-44a0-b3a8-635c054537f4","logger": "org.openqa.selenium.grid.distributor.local.LocalDistributor","request.payload": "[Capabilities {browserName: chrome, goog:chromeOptions: {args: [user-agent=Chrome <app-name>\u002f2.12.6]}, goog:loggingPrefs: {browser: INFO, performance: INFO}, pageLoadStrategy: eager, platformName: linux, proxy: {httpProxy: fwd_proxy_all:3128, proxyType: manual, sslProxy: fwd_proxy_all:3128}}]"}}
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000636c00000, 578813952, 0) failed; error='Not enough space' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 578813952 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid12.log
2023-09-10 12:23:40,485 INFO exited: selenium-grid-hub (exit status 1; not expected)
$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
db067c502979 nginx:stable-alpine "/docker-entrypoint.…" 46 hours ago Up 46 hours (healthy) 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:4444->4444/tcp, :::4444->4444/tcp, 0.0.0.0:6984->6984/tcp, :::6984->6984/tcp, 0.0.0.0:9443->9443/tcp, :::9443->9443/tcp app_rev_proxy_1
059b3f6b035e company-name/app-ui:v2.18.0 "/docker-entrypoint.…" 46 hours ago Up 46 hours (healthy) 80/tcp app_ui_1
d4b361b703a0 company-name/app-client:v2.18.0 "/bin/sh -c 'sh scri…" 46 hours ago Up 44 hours app_client_1
52216db66b8f company-name/app-api:v2.18.0 "/bin/sh -c 'node .'" 46 hours ago Up 46 hours (healthy) 80/tcp app_api_4
001884e2dfba company-name/app-api:v2.18.0 "/bin/sh -c 'node .'" 46 hours ago Up 44 hours (healthy) 80/tcp app_api_2
ee6c0bb93580 company-name/app-api:v2.18.0 "/bin/sh -c 'node .'" 46 hours ago Up 46 hours (healthy) 80/tcp app_api_5
b25980a637de muenchhausen/docker-squidguard:1.0.0 "/sbin/entrypoint.sh…" 46 hours ago Up 46 hours 80/tcp, 3128/tcp app_fwd_proxy_bc_1
cbd45d55c76c muenchhausen/docker-squidguard:1.0.0 "/sbin/entrypoint.sh…" 46 hours ago Up 46 hours 80/tcp, 3128/tcp app_fwd_proxy_all_1
a257e83921eb selenium/node-chrome:4 "/opt/bin/entry_poin…" 46 hours ago Up 46 hours 5900/tcp app_grid_node_chrome_4
e135f3dc62ca selenium/node-edge:4 "/opt/bin/entry_poin…" 46 hours ago Up 46 hours 5900/tcp app_grid_node_edge_4
915dbd47a4d7 selenium/node-chrome:4 "/opt/bin/entry_poin…" 46 hours ago Up 46 hours 5900/tcp app_grid_node_chrome_1
d61aea9f50a7 selenium/node-chrome:4 "/opt/bin/entry_poin…" 46 hours ago Up 46 hours 5900/tcp app_grid_node_chrome_3
92a9e028348f selenium/node-chrome:4 "/opt/bin/entry_poin…" 46 hours ago Up 46 hours 5900/tcp app_grid_node_chrome_2
fabc037b6bdf selenium/node-edge:4 "/opt/bin/entry_poin…" 46 hours ago Up 46 hours 5900/tcp app_grid_node_edge_3
8ffdfdb7a8d4 selenium/node-firefox:4 "/opt/bin/entry_poin…" 46 hours ago Up 46 hours 5900/tcp app_grid_node_firefox_2
10b00881c14f selenium/node-firefox:4 "/opt/bin/entry_poin…" 46 hours ago Up 46 hours 5900/tcp app_grid_node_firefox_1
7579b56d6045 selenium/node-firefox:4 "/opt/bin/entry_poin…" 46 hours ago Up 46 hours 5900/tcp app_grid_node_firefox_4
17627abda037 selenium/node-firefox:4 "/opt/bin/entry_poin…" 46 hours ago Up 46 hours 5900/tcp app_grid_node_firefox_3
6fb81fa2631a selenium/node-edge:4 "/opt/bin/entry_poin…" 46 hours ago Up 46 hours 5900/tcp app_grid_node_edge_2
969eafe147bf selenium/node-firefox:4 "/opt/bin/entry_poin…" 46 hours ago Up 46 hours 5900/tcp app_grid_node_firefox_5
f9f52eacffdf selenium/node-chrome:4 "/opt/bin/entry_poin…" 46 hours ago Up 46 hours 5900/tcp app_grid_node_chrome_5
4d369a7b53e5 selenium/node-edge:4 "/opt/bin/entry_poin…" 46 hours ago Up 46 hours 5900/tcp app_grid_node_edge_5
9a2660fcf943 selenium/node-edge:4 "/opt/bin/entry_poin…" 46 hours ago Up 46 hours 5900/tcp app_grid_node_edge_1
788039fb34af company-name/app-api:v2.18.0 "/bin/sh -c 'node .'" 46 hours ago Up 46 hours (healthy) 80/tcp app_api_3
8ab0be6b5e44 company-name/app-api:v2.18.0 "/bin/sh -c 'node .'" 46 hours ago Up 46 hours (healthy) 80/tcp app_api_1
92caccf94b39 couchdb:3 "tini -- /docker-ent…" 46 hours ago Up 46 hours (healthy) 4369/tcp, 5984/tcp, 9100/tcp app_db_node_5
69f1e4f99062 couchdb:3 "tini -- /docker-ent…" 46 hours ago Up 46 hours (healthy) 4369/tcp, 5984/tcp, 9100/tcp app_db_node_2
51bd245f8d5e couchdb:3 "tini -- /docker-ent…" 46 hours ago Up 46 hours (healthy) 4369/tcp, 5984/tcp, 9100/tcp app_db_node_3
0c1ef24fca1d couchdb:3 "tini -- /docker-ent…" 46 hours ago Up 46 hours (healthy) 4369/tcp, 5984/tcp, 9100/tcp app_db_node_1
1d486df50fa8 couchdb:3 "tini -- /docker-ent…" 46 hours ago Up 46 hours (healthy) 4369/tcp, 5984/tcp, 9100/tcp app_db_node_4
193ef6793db1 selenium/hub:4 "/opt/bin/entry_poin…" 46 hours ago Up 46 hours (unhealthy) 4442-4444/tcp app_grid_hub_1
$ docker container stats --no-stream
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
4c025db6df81 <app-name>_rev_proxy_1 0.00% 15.52MiB / 31.42GiB 0.05% 359MB / 297MB 930MB / 90.1kB 9
eb2deb4f24aa <app-name>_ui_1 0.00% 10.96MiB / 31.42GiB 0.03% 146MB / 256MB 693MB / 24.6kB 9
fea00da2b15d <app-name>_client_1 0.01% 84.16MiB / 31.42GiB 0.26% 44.4MB / 23.4MB 160MB / 4.1kB 24
d2e9de74ba96 <app-name>_api_2 0.00% 55.4MiB / 31.42GiB 0.17% 417MB / 103MB 7.37GB / 193kB 11
edfc4eaad300 <app-name>_api_4 0.00% 50.32MiB / 31.42GiB 0.16% 348MB / 95.6MB 6.4GB / 147kB 11
9bcbb0e8066d <app-name>_api_1 0.00% 50.59MiB / 31.42GiB 0.16% 419MB / 95MB 8.8GB / 152kB 11
716af736ce15 <app-name>_api_3 0.00% 96.84MiB / 31.42GiB 0.30% 335MB / 97.5MB 8.7GB / 90.1kB 11
f8c88b7dbbaa <app-name>_api_5 0.00% 54.6MiB / 31.42GiB 0.17% 362MB / 96.4MB 7.1GB / 213kB 11
4a68855cd5ed <app-name>_grid_node_chrome_1 4.83% 12.18GiB / 31.42GiB 38.75% 26.7GB / 2.93GB 69.9GB / 24.3GB 9798
f92234577fd7 <app-name>_grid_node_edge_1 5.34% 10.41GiB / 31.42GiB 33.13% 7.49GB / 1.35GB 63.9GB / 9.84GB 7381
eaf451b62c5a <app-name>_grid_node_firefox_1 0.93% 765.1MiB / 31.42GiB 2.38% 8.04GB / 942MB 15.9GB / 45.6GB 70
d6503f6034d7 <app-name>_fwd_proxy_all_1 0.07% 144.4MiB / 31.42GiB 0.45% 24.3GB / 24.5GB 2.43GB / 274kB 72
aea02ca0e5de <app-name>_fwd_proxy_bc_1 0.12% 76.2MiB / 31.42GiB 0.24% 324MB / 253MB 1.86GB / 41kB 72
b67b0f4d10a8 <app-name>_db_node_2 0.71% 168.9MiB / 31.42GiB 0.52% 1.06GB / 1.23GB 3.53GB / 182MB 62
54ba614ec686 <app-name>_db_node_5 0.86% 184.1MiB / 31.42GiB 0.57% 1.27GB / 1.32GB 4.62GB / 204MB 89
fe64eb861d3f <app-name>_db_node_3 4.96% 226.8MiB / 31.42GiB 0.70% 1.47GB / 1.44GB 4.45GB / 374MB 89
fc6038ea3fa7 <app-name>_db_node_1 0.64% 306MiB / 31.42GiB 0.95% 1.1GB / 1.24GB 4.27GB / 526MB 62
1b0bb24cd80a <app-name>_db_node_4 5.09% 169.7MiB / 31.42GiB 0.53% 1.28GB / 1.24GB 5.65GB / 203MB 89
8a9208b0921a <app-name>_grid_hub_1 0.01% 27MiB / 31.42GiB 0.08% 2.69GB / 2.6GB 6.67GB / 111kB 2
Operating System
Ubuntu 16.04
Selenium version
4.12 Java
What are the browser(s) and version(s) where you see this issue?
Chrome/Edge/Firefox docker nodes 4.12
What are the browser driver(s) and version(s) where you see this issue?
Chrome/Edge/Firefox docker nodes 4.12
Are you using Selenium Grid?
Yes, the docker version of 4.12
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 16 (8 by maintainers)
We’ve only had this one report and we haven’t been given something we can reproduce on this one in several months. I’m going to close this pending more information. If this issue locks, please open a new one using the latest version of Selenium. Thanks.