cypress: Tests causing CI agents to run out of memory on cypress 8.2 & above

Hey, my company has this issue too and I think I found the cause and am close to having a PR ready to fix this.

Setup

In my environment, I found this out of memory issue only happens in Chrome, it doesn’t seem to occur in FireFox. Hopefully someone might be able to confirm this for me, as this would make me hopeful my fix solves this issue.

According to Task Manager, Cypress’s server was growing to multiple GB in size, so I added some code to make Cypress dump its server’s heap memory every 90 seconds. In the contents of that, I found a memory leak and I hope it’s the one related to this issue.

Investigation

In the memory dump, I found Cypress was storing hundreds (maybe thousands) copies of the same base64 encoded string in an array named pendingBrowserPreRequests.

I looked at my company’s app and found a css file that has the base64 font file in question hardcoded as the src field.

Side note: this issue could be remedied on the application side if base64 font files are removed from css files and changed to urls.

Root Cause

Cypress subscribes to the CDP event Network.requestWillBeSent and adds network requests that will be sent to an array. The elements in the array are removed when the corresponding response for the request comes back.

With these fonts inside the URL field of css stylesheets, this triggers Chrome to send a Network.requestWillBeSent event to Cypress with the URL of the request being the entire base64 encoded font file (lol). These requests are not actually going to be sent from the browser as it will parse these values instead, so Cypress never knows when to release these from its array since it never sees a response. It could be argued that this is a Chrome issue, but the fix should be relatively simple on the Cypress side.

I think I can have a PR out tomorrow to fix.

cc: @jennifer-shehane @brian-mann

@BlueWinds PR is out!

Come-on we need this to be fixed ASAP there are multiple features from Cypress v10 that would simplify so much things and I want to use them but Im unable to due this bug… @cellog @jennifer-shehane

Just an update, I rolled my project back to Cypress 8.1 for now until this can be resolved. All tests run fine on that version.

We have been stuck on 8.1 due to this issue. Can someone please prioritze this issue?

@DJSdev - Jennifer’s currently on leave, so she won’t be around to address this, but I’m certainly interested! Please ping me on your PR when you have it ready, I’ll make sure it gets the team’s attention ASAP.

Our team also runs on linux Github Actions runners which give only 7GB of RAM. We are also stuck on 8.1.

Tried to update to actual versions

"cypress": "9.3.1",
"cypress-cucumber-preprocessor": "4.3.1",
"typescript": "4.5.5"

Pushed changes to the test project. Still have the same issue

@BlueWinds, tried with version 9.0.0. No changes. Run fails on 295 test file. Here is the test project where the problem is reproduced

@jennifer-shehane I’ve got the same issue with 8.3.1 on Azure Linux agents:

2021-09-10T08:46:44.512Z cypress:cli checking environment variables
2021-09-10T08:46:44.513Z cypress:cli checking if executable exists /home/vsts/.cache/Cypress/8.3.1/Cypress/Cypress
2021-09-10T08:46:44.514Z cypress:cli Binary is executable? : true
2021-09-10T08:46:44.515Z cypress:cli binaryDir is  /home/vsts/.cache/Cypress/8.3.1/Cypress
2021-09-10T08:46:44.515Z cypress:cli Reading binary package.json from: /home/vsts/.cache/Cypress/8.3.1/Cypress/resources/app/package.json
2021-09-10T08:46:44.517Z cypress:cli Found binary version 8.3.1 installed in: /home/vsts/.cache/Cypress/8.3.1/Cypress
2021-09-10T08:46:44.519Z cypress:cli could not read binary_state.json file at "/home/vsts/.cache/Cypress/8.3.1/binary_state.json"
2021-09-10T08:46:44.520Z cypress:cli {}
2021-09-10T08:46:44.520Z cypress:cli is Verified ? undefined
2021-09-10T08:46:44.520Z cypress:cli running binary verification check 8.3.1
It looks like this is your first time using Cypress: 8.3.1

 cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'plugin'      │      2       │      '3757, 14814'       │   43.01    │     47.46      │  2574.7  │   1243.49    │   2654.23   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '3354'          │     5      │      4.31      │  221.75  │    249.44    │    265.1    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │ '3356, 3515, 3357, 3676' │   102.5    │     25.85      │  193.93  │    201.46    │   204.38    │
  cypress:server:util:process_profiler │    3    │    'Electron'     │      1       │         '14802'          │    0.05    │     14.75      │  118.04  │    242.54    │   712.32    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │      '14832, 14833'      │     0      │       0        │   3.48   │     3.45     │    3.56     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      10      │           '-'            │   150.56   │     91.66      │ 3111.91  │    1919.5    │   3262.54   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +11s
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'plugin'      │      2       │      '3757, 14814'       │   92.19    │      48.1      │ 2772.17  │   1265.33    │   2772.17   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '3354'          │    0.95    │      4.26      │  222.39  │    249.05    │    265.1    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │ '3356, 3515, 3357, 3676' │    0.48    │     25.49      │  193.93  │    201.36    │   204.38    │
  cypress:server:util:process_profiler │    3    │    'Electron'     │      1       │         '14802'          │    0.67    │     14.55      │  115.13  │    240.66    │   712.32    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │      '14987, 14988'      │     0      │       0        │   3.41   │     3.45     │    3.56     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      10      │           '-'            │   94.29    │      91.7      │ 3307.03  │   1939.05    │   3307.03   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s

<--- Last few GCs --->

[3757:0x3f6200000000]   741154 ms: Scavenge 1998.0 (2061.2) -> 1996.5 (2063.2) MB, 14.8 / 0.0 ms  (average mu = 0.797, current mu = 0.427) allocation failure 
[3757:0x3f6200000000]   741216 ms: Scavenge 2000.4 (2063.2) -> 1999.0 (2067.2) MB, 29.4 / 0.0 ms  (average mu = 0.797, current mu = 0.427) allocation failure 
[3757:0x3f6200000000]   742915 ms: Mark-sweep 2004.3 (2067.5) -> 2001.3 (2074.0) MB, 1670.7 / 0.0 ms  (average mu = 0.650, current mu = 0.179) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory