selenium: [🐛 Bug]: Possible memory leak involving hub and chromium-based nodes

What happened?

We run Selenium Grid Hub and Nodes via Docker on our server. Frequently (about once every 2-3 days) we see that the hub displays it’s status as “unhealthy”, and our server RAM usage spikes close to 100%. Additionally, the chrome/edge node containers show as using way more storage than the rest of the containers, typically on the scale of gigabytes.

Restarting the containers does not resolve the issue, but stopping/killing/starting the containers does. We leave our containers running for days/weeks at a time, and this has never been an issue prior to about 3 months ago.

How can we reproduce the issue?

We have not been able to consistently reproduce the issue. However, it seems to somewhat consistently manifest when we induce a heavy test load on the grid.

Relevant log output

12:23:35.702 INFO [LocalDistributor.newSession] - Unable to find a free slot for request 86dbd876-1306-49a3-922d-edb8e36b7e1e.
 Capabilities {browserName: chrome, goog:chromeOptions: {args: [user-agent=Chrome <app-name>/2.12.6]}, goog:loggingPrefs: {browser: INFO, performance: INFO}, pageLoadStrategy: eager, platformName: linux, proxy: {httpProxy: fwd_proxy_all:3128, proxyType: manual, sslProxy: fwd_proxy_all:3128}}
12:23:35.702 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "11ec0e7f003a46e8793ec9784fffa8e5","eventTime": 1694348615685052940,"eventName": "Session request received by the Distributor","attributes": {"logger": "org.openqa.selenium.grid.distributor.local.LocalDistributor","request.payload": "[Capabilities {browserName: chrome, goog:chromeOptions: {args: [user-agent=Chrome <app-name>\u002f2.12.6]}, goog:loggingPrefs: {browser: INFO, performance: INFO}, pageLoadStrategy: eager, platformName: linux, proxy: {httpProxy: fwd_proxy_all:3128, proxyType: manual, sslProxy: fwd_proxy_all:3128}}]"}}

12:23:35.762 WARN [SeleniumSpanExporter$1.lambda$export$1] - Will retry session 86dbd876-1306-49a3-922d-edb8e36b7e1e
12:23:35.762 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "11ec0e7f003a46e8793ec9784fffa8e5","eventTime": 1694348615701163474,"eventName": "exception","attributes": {"exception.message": "Will retry session 86dbd876-1306-49a3-922d-edb8e36b7e1e","logger": "org.openqa.selenium.grid.distributor.local.LocalDistributor","request.payload": "[Capabilities {browserName: chrome, goog:chromeOptions: {args: [user-agent=Chrome <app-name>\u002f2.12.6]}, goog:loggingPrefs: {browser: INFO, performance: INFO}, pageLoadStrategy: eager, platformName: linux, proxy: {httpProxy: fwd_proxy_all:3128, proxyType: manual, sslProxy: fwd_proxy_all:3128}}]"}}

12:23:35.705 INFO [LocalDistributor.newSession] - Unable to find a free slot for request 62952ed8-2e68-44a0-b3a8-635c054537f4.
 Capabilities {browserName: chrome, goog:chromeOptions: {args: [user-agent=Chrome <app-name>/2.12.6]}, goog:loggingPrefs: {browser: INFO, performance: INFO}, pageLoadStrategy: eager, platformName: linux, proxy: {httpProxy: fwd_proxy_all:3128, proxyType: manual, sslProxy: fwd_proxy_all:3128}}
12:23:35.765 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "4687cba99b98a49fa32963a6cf4c26c5","eventTime": 1694348615689081642,"eventName": "Session request received by the Distributor","attributes": {"logger": "org.openqa.selenium.grid.distributor.local.LocalDistributor","request.payload": "[Capabilities {browserName: chrome, goog:chromeOptions: {args: [user-agent=Chrome <app-name>\u002f2.12.6]}, goog:loggingPrefs: {browser: INFO, performance: INFO}, pageLoadStrategy: eager, platformName: linux, proxy: {httpProxy: fwd_proxy_all:3128, proxyType: manual, sslProxy: fwd_proxy_all:3128}}]"}}

12:23:35.765 WARN [SeleniumSpanExporter$1.lambda$export$1] - Will retry session 62952ed8-2e68-44a0-b3a8-635c054537f4
12:23:35.765 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "4687cba99b98a49fa32963a6cf4c26c5","eventTime": 1694348615765110736,"eventName": "exception","attributes": {"exception.message": "Will retry session 62952ed8-2e68-44a0-b3a8-635c054537f4","logger": "org.openqa.selenium.grid.distributor.local.LocalDistributor","request.payload": "[Capabilities {browserName: chrome, goog:chromeOptions: {args: [user-agent=Chrome <app-name>\u002f2.12.6]}, goog:loggingPrefs: {browser: INFO, performance: INFO}, pageLoadStrategy: eager, platformName: linux, proxy: {httpProxy: fwd_proxy_all:3128, proxyType: manual, sslProxy: fwd_proxy_all:3128}}]"}}

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000636c00000, 578813952, 0) failed; error='Not enough space' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 578813952 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid12.log
2023-09-10 12:23:40,485 INFO exited: selenium-grid-hub (exit status 1; not expected)

$ docker container ls
CONTAINER ID   IMAGE                                  COMMAND                  CREATED        STATUS                    PORTS                                                                                                                                                                                                       NAMES
db067c502979   nginx:stable-alpine                    "/docker-entrypoint.…"   46 hours ago   Up 46 hours (healthy)     0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:4444->4444/tcp, :::4444->4444/tcp, 0.0.0.0:6984->6984/tcp, :::6984->6984/tcp, 0.0.0.0:9443->9443/tcp, :::9443->9443/tcp   app_rev_proxy_1
059b3f6b035e   company-name/app-ui:v2.18.0              "/docker-entrypoint.…"   46 hours ago   Up 46 hours (healthy)     80/tcp                                                                                                                                                                                                      app_ui_1
d4b361b703a0   company-name/app-client:v2.18.0          "/bin/sh -c 'sh scri…"   46 hours ago   Up 44 hours                                                                                                                                                                                                                           app_client_1
52216db66b8f   company-name/app-api:v2.18.0             "/bin/sh -c 'node .'"    46 hours ago   Up 46 hours (healthy)     80/tcp                                                                                                                                                                                                      app_api_4
001884e2dfba   company-name/app-api:v2.18.0             "/bin/sh -c 'node .'"    46 hours ago   Up 44 hours (healthy)     80/tcp                                                                                                                                                                                                      app_api_2
ee6c0bb93580   company-name/app-api:v2.18.0             "/bin/sh -c 'node .'"    46 hours ago   Up 46 hours (healthy)     80/tcp                                                                                                                                                                                                      app_api_5
b25980a637de   muenchhausen/docker-squidguard:1.0.0   "/sbin/entrypoint.sh…"   46 hours ago   Up 46 hours               80/tcp, 3128/tcp                                                                                                                                                                                            app_fwd_proxy_bc_1
cbd45d55c76c   muenchhausen/docker-squidguard:1.0.0   "/sbin/entrypoint.sh…"   46 hours ago   Up 46 hours               80/tcp, 3128/tcp                                                                                                                                                                                            app_fwd_proxy_all_1
a257e83921eb   selenium/node-chrome:4                 "/opt/bin/entry_poin…"   46 hours ago   Up 46 hours               5900/tcp                                                                                                                                                                                                    app_grid_node_chrome_4
e135f3dc62ca   selenium/node-edge:4                   "/opt/bin/entry_poin…"   46 hours ago   Up 46 hours               5900/tcp                                                                                                                                                                                                    app_grid_node_edge_4
915dbd47a4d7   selenium/node-chrome:4                 "/opt/bin/entry_poin…"   46 hours ago   Up 46 hours               5900/tcp                                                                                                                                                                                                    app_grid_node_chrome_1
d61aea9f50a7   selenium/node-chrome:4                 "/opt/bin/entry_poin…"   46 hours ago   Up 46 hours               5900/tcp                                                                                                                                                                                                    app_grid_node_chrome_3
92a9e028348f   selenium/node-chrome:4                 "/opt/bin/entry_poin…"   46 hours ago   Up 46 hours               5900/tcp                                                                                                                                                                                                    app_grid_node_chrome_2
fabc037b6bdf   selenium/node-edge:4                   "/opt/bin/entry_poin…"   46 hours ago   Up 46 hours               5900/tcp                                                                                                                                                                                                    app_grid_node_edge_3
8ffdfdb7a8d4   selenium/node-firefox:4                "/opt/bin/entry_poin…"   46 hours ago   Up 46 hours               5900/tcp                                                                                                                                                                                                    app_grid_node_firefox_2
10b00881c14f   selenium/node-firefox:4                "/opt/bin/entry_poin…"   46 hours ago   Up 46 hours               5900/tcp                                                                                                                                                                                                    app_grid_node_firefox_1
7579b56d6045   selenium/node-firefox:4                "/opt/bin/entry_poin…"   46 hours ago   Up 46 hours               5900/tcp                                                                                                                                                                                                    app_grid_node_firefox_4
17627abda037   selenium/node-firefox:4                "/opt/bin/entry_poin…"   46 hours ago   Up 46 hours               5900/tcp                                                                                                                                                                                                    app_grid_node_firefox_3
6fb81fa2631a   selenium/node-edge:4                   "/opt/bin/entry_poin…"   46 hours ago   Up 46 hours               5900/tcp                                                                                                                                                                                                    app_grid_node_edge_2
969eafe147bf   selenium/node-firefox:4                "/opt/bin/entry_poin…"   46 hours ago   Up 46 hours               5900/tcp                                                                                                                                                                                                    app_grid_node_firefox_5
f9f52eacffdf   selenium/node-chrome:4                 "/opt/bin/entry_poin…"   46 hours ago   Up 46 hours               5900/tcp                                                                                                                                                                                                    app_grid_node_chrome_5
4d369a7b53e5   selenium/node-edge:4                   "/opt/bin/entry_poin…"   46 hours ago   Up 46 hours               5900/tcp                                                                                                                                                                                                    app_grid_node_edge_5
9a2660fcf943   selenium/node-edge:4                   "/opt/bin/entry_poin…"   46 hours ago   Up 46 hours               5900/tcp                                                                                                                                                                                                    app_grid_node_edge_1
788039fb34af   company-name/app-api:v2.18.0             "/bin/sh -c 'node .'"    46 hours ago   Up 46 hours (healthy)     80/tcp                                                                                                                                                                                                      app_api_3
8ab0be6b5e44   company-name/app-api:v2.18.0             "/bin/sh -c 'node .'"    46 hours ago   Up 46 hours (healthy)     80/tcp                                                                                                                                                                                                      app_api_1
92caccf94b39   couchdb:3                              "tini -- /docker-ent…"   46 hours ago   Up 46 hours (healthy)     4369/tcp, 5984/tcp, 9100/tcp                                                                                                                                                                                app_db_node_5
69f1e4f99062   couchdb:3                              "tini -- /docker-ent…"   46 hours ago   Up 46 hours (healthy)     4369/tcp, 5984/tcp, 9100/tcp                                                                                                                                                                                app_db_node_2
51bd245f8d5e   couchdb:3                              "tini -- /docker-ent…"   46 hours ago   Up 46 hours (healthy)     4369/tcp, 5984/tcp, 9100/tcp                                                                                                                                                                                app_db_node_3
0c1ef24fca1d   couchdb:3                              "tini -- /docker-ent…"   46 hours ago   Up 46 hours (healthy)     4369/tcp, 5984/tcp, 9100/tcp                                                                                                                                                                                app_db_node_1
1d486df50fa8   couchdb:3                              "tini -- /docker-ent…"   46 hours ago   Up 46 hours (healthy)     4369/tcp, 5984/tcp, 9100/tcp                                                                                                                                                                                app_db_node_4
193ef6793db1   selenium/hub:4                         "/opt/bin/entry_poin…"   46 hours ago   Up 46 hours (unhealthy)   4442-4444/tcp                                                                                                                                                                                               app_grid_hub_1

$ docker container stats --no-stream
CONTAINER ID   NAME                        CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
4c025db6df81   <app-name>_rev_proxy_1           0.00%     15.52MiB / 31.42GiB   0.05%     359MB / 297MB     930MB / 90.1kB    9
eb2deb4f24aa   <app-name>_ui_1                  0.00%     10.96MiB / 31.42GiB   0.03%     146MB / 256MB     693MB / 24.6kB    9
fea00da2b15d   <app-name>_client_1              0.01%     84.16MiB / 31.42GiB   0.26%     44.4MB / 23.4MB   160MB / 4.1kB     24
d2e9de74ba96   <app-name>_api_2                 0.00%     55.4MiB / 31.42GiB    0.17%     417MB / 103MB     7.37GB / 193kB    11
edfc4eaad300   <app-name>_api_4                 0.00%     50.32MiB / 31.42GiB   0.16%     348MB / 95.6MB    6.4GB / 147kB     11
9bcbb0e8066d   <app-name>_api_1                 0.00%     50.59MiB / 31.42GiB   0.16%     419MB / 95MB      8.8GB / 152kB     11
716af736ce15   <app-name>_api_3                 0.00%     96.84MiB / 31.42GiB   0.30%     335MB / 97.5MB    8.7GB / 90.1kB    11
f8c88b7dbbaa   <app-name>_api_5                 0.00%     54.6MiB / 31.42GiB    0.17%     362MB / 96.4MB    7.1GB / 213kB     11
4a68855cd5ed   <app-name>_grid_node_chrome_1    4.83%     12.18GiB / 31.42GiB   38.75%    26.7GB / 2.93GB   69.9GB / 24.3GB   9798
f92234577fd7   <app-name>_grid_node_edge_1      5.34%     10.41GiB / 31.42GiB   33.13%    7.49GB / 1.35GB   63.9GB / 9.84GB   7381
eaf451b62c5a   <app-name>_grid_node_firefox_1   0.93%     765.1MiB / 31.42GiB   2.38%     8.04GB / 942MB    15.9GB / 45.6GB   70
d6503f6034d7   <app-name>_fwd_proxy_all_1       0.07%     144.4MiB / 31.42GiB   0.45%     24.3GB / 24.5GB   2.43GB / 274kB    72
aea02ca0e5de   <app-name>_fwd_proxy_bc_1        0.12%     76.2MiB / 31.42GiB    0.24%     324MB / 253MB     1.86GB / 41kB     72
b67b0f4d10a8   <app-name>_db_node_2             0.71%     168.9MiB / 31.42GiB   0.52%     1.06GB / 1.23GB   3.53GB / 182MB    62
54ba614ec686   <app-name>_db_node_5             0.86%     184.1MiB / 31.42GiB   0.57%     1.27GB / 1.32GB   4.62GB / 204MB    89
fe64eb861d3f   <app-name>_db_node_3             4.96%     226.8MiB / 31.42GiB   0.70%     1.47GB / 1.44GB   4.45GB / 374MB    89
fc6038ea3fa7   <app-name>_db_node_1             0.64%     306MiB / 31.42GiB     0.95%     1.1GB / 1.24GB    4.27GB / 526MB    62
1b0bb24cd80a   <app-name>_db_node_4             5.09%     169.7MiB / 31.42GiB   0.53%     1.28GB / 1.24GB   5.65GB / 203MB    89
8a9208b0921a   <app-name>_grid_hub_1            0.01%     27MiB / 31.42GiB      0.08%     2.69GB / 2.6GB    6.67GB / 111kB    2

Operating System

Ubuntu 16.04

Selenium version

4.12 Java

What are the browser(s) and version(s) where you see this issue?

Chrome/Edge/Firefox docker nodes 4.12

What are the browser driver(s) and version(s) where you see this issue?

Chrome/Edge/Firefox docker nodes 4.12

Are you using Selenium Grid?

Yes, the docker version of 4.12

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 16 (8 by maintainers)

Most upvoted comments

We’ve only had this one report and we haven’t been given something we can reproduce on this one in several months. I’m going to close this pending more information. If this issue locks, please open a new one using the latest version of Selenium. Thanks.