cypress: [Cypress Webkit Ci Run] Test stuck during Ci run

Current behavior

We are currently experimenting with cross browser testing within the CiCd pipeline. Our main goal is to run Cypress with Webkit and Chrome and run it in multiple viewports.

We run Cypress on multiple machines within Gitlab, adding the --parallel to connect to the Cypress Dashboard. We found out that with every run the Webkit tests got stuck and eventually timed out.

Example of one of the files: Screenshot 2022-11-29 at 15 29 19

Example of what it looks like within Gitlab: Screenshot 2022-11-29 at 15 26 52

Example of timeout log within gitlab: Screenshot 2022-11-29 at 15 27 03

We enabled the debugger, and when analysing it, we see that the cypress:server:util:process_profiler keeps repeating itself in the end

Desired behavior

All tests run and finish correctly.

Test code to reproduce

Screenshot 2022-11-29 at 15 29 19

MicrosoftTeams-image (30)

Cypress Version

10.11.0

Node version

v16.14.2

Operating System

cypress-image/browsers:latest-cypress-10.11.0 docker image

Debug Logs

cypress:server:browsers:webkit-automation received requestfinished { responseReceived: { requestId: '453', status: 200, headers: { 'content-type': 'image/png', date: 'Fri, 11 Nov 2022 09:25:31 GMT', 'access-control-allow-origin': '*', 'access-control-allow-credentials': 'true', 'x-content-type-options': 'nosniff', 'last-modified': 'Fri, 11 Nov 2022 09:25:31 GMT', etag: 'W/"ff8cc89-ea33c-5ed2e7897d023"', 'accept-ranges': 'bytes', 'cache-control': 'max-age=2629746, public', expires: 'Fri, 11 Nov 2022 09:30:30 GMT', 'x-xss-protection': '1; mode=block', via: '1.1 dfd84a17eaa88d79994b6524cab4931e.cloudfront.net (CloudFront)', age: '1639861', connection: 'keep-alive', 'keep-alive': 'timeout=5', 'transfer-encoding': 'chunked', 'strict-transport-security': 'max-age=15724800; includeSubDomains, max-age=63072000; includeSubdomains; preload', 'permissions-policy': 'geolocation=();midi=();notifications=();push=();sync-xhr=();microphone=();camera=();magnetometer=();gyroscope=();speaker=(self);vibrate=();fullscreen=(self);payment=();', 'x-cache': 'Hit from cloudfront', 'x-amz-cf-pop': 'DUB56-P1', 'x-amz-cf-id': 'EUKPC3vBU19zR_nsT4U8eoutO65EerEzUcAC-jC3NPL9lTDZ0NXajQ==' } } } +0ms
  cypress:server:browsers:webkit-automation received requestfinished { responseReceived: { requestId: '459', status: 200, headers: { 'access-control-allow-origin': 'https://www.ziggo.nl', date: 'Wed, 30 Nov 2022 08:56:32 GMT', pragma: 'no-cache', expires: 'Fri, 01 Jan 1990 00:00:00 GMT', 'cache-control': 'no-cache, no-store, must-revalidate', 'last-modified': 'Sun, 17 May 1998 03:00:00 GMT', 'x-content-type-options': 'nosniff', 'access-control-allow-credentials': 'true', 'content-type': 'image/gif', 'cross-origin-resource-policy': 'cross-origin', connection: 'keep-alive', 'keep-alive': 'timeout=5', 'transfer-encoding': 'chunked', server: 'Golfe2', 'alt-svc': 'h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="46, 43"' } } } +11ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51776 } +4s
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51824 } +933ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51836 } +82ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51830 } +26ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51846 } +0ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51844 } +18ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51838 } +48ms
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51826 } +16ms
  cypress:server:browsers:webkit-automation received request { browserPreRequest: { requestId: '463', method: 'POST', url: 'https://region1.google-analytics.com/g/collect?v=2&tid=G-TH972FDWFF&gtm=2oebs0&_p=776742999&cid=872464754.1669798568&ul=en-us&sr=1440x1024&_eu=AAg&_s=9&sid=1669798568&sct=1&seg=1&dl=https%3A%2F%2Fwww.ziggo.nl%2Finternet%2Finternet-only%3Futm_medium%3Dtesting%26utm_source%3Dtesting%26optimizely_disable%3Dtrue&dt=Alleen%20Internet%20Thuis%20%E2%80%93%20Internet%20Only%20Pakketten%20Vergelijken%20%7C%20Ziggo&en=click_on_button&ep.interaction_event=click.button&ep.element_text=ga%20verder&ep.element_link=none&ep.element_classes=z-basket__footer-btn%20btn%20btn-primary&_et=2', headers: { accept: '*/*', referer: 'https://www.ziggo.nl/', origin: 'https://www.ziggo.nl', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15', 'cache-control': 'max-age=0' }, resourceType: 'other', originalResourceType: 'beacon' } } +5s
  cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51850 } +255ms
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'WebKit'      │      4       │ '1075, 1080, 1094, 1098' │   83.81    │     68.01      │ 1097.35  │    887.91    │   1202.64   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '299'           │    1.89    │     14.79      │   295    │    248.49    │     295     │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │   '311, 471, 312, 518'   │     0      │       0        │  199.98  │    188.23    │   199.98    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │          '497'           │     0      │      0.54      │  137.66  │    182.66    │   198.71    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │       '1497, 1498'       │     0      │       0        │   3.48   │     3.47     │    3.55     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      12      │           '-'            │    85.7    │     75.73      │ 1733.46  │   1391.79    │   1886.53   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'WebKit'      │      4       │ '1075, 1080, 1094, 1098' │   81.64    │     69.52      │ 1072.17  │    908.38    │   1202.64   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '299'           │    0.3     │     13.34      │  295.07  │    253.14    │   295.07    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │   '311, 471, 312, 518'   │     0      │       0        │  199.98  │    189.4     │   199.98    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │          '497'           │     0      │      0.48      │  137.66  │    177.66    │   198.71    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │       '1520, 1521'       │     0      │       0        │   3.55   │     3.48     │    3.55     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      12      │           '-'            │   81.94    │     76.35      │ 1708.42  │   1423.45    │   1886.53   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'WebKit'      │      4       │ '1075, 1080, 1094, 1098' │   86.33    │     71.21      │ 1063.43  │    923.89    │   1202.64   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '299'           │    0.3     │     12.16      │  295.23  │    256.97    │   295.23    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │   '311, 471, 312, 518'   │     0      │       0        │  199.98  │    190.36    │   199.98    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │          '497'           │     0      │      0.43      │  137.66  │    173.66    │   198.71    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │       '1544, 1545'       │     0      │       0        │   3.35   │     3.47     │    3.55     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      12      │           '-'            │   86.63    │     77.28      │ 1699.65  │   1448.56    │   1886.53   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s
  cypress:server:socket-base socket-disconnecting ping timeout +37s
  cypress:server:socket-base socket-disconnect ping timeout +0ms
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'WebKit'      │      4       │ '1075, 1080, 1094, 1098' │   84.24    │     72.39      │ 1123.76  │    942.06    │   1202.64   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '299'           │    1.55    │     11.27      │  268.29  │    257.91    │   295.23    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │   '311, 471, 312, 518'   │     0      │       0        │  199.98  │    191.16    │   199.98    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │          '497'           │     0      │      0.39      │  137.66  │    170.38    │   198.71    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │       '1569, 1570'       │     0      │       0        │   3.33   │     3.46     │    3.55     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      12      │           '-'            │   85.79    │     77.99      │ 1733.01  │   1472.27    │   1886.53   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'WebKit'      │      4       │ '1075, 1080, 1094, 1098' │   85.64    │     73.49      │ 1108.19  │    955.9     │   1202.64   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '299'           │    3.54    │     10.68      │  230.77  │    255.83    │   295.23    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │   '311, 471, 312, 518'   │     0      │       0        │  199.98  │    191.84    │   199.98    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │          '497'           │     0      │      0.36      │  137.66  │    167.66    │   198.71    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │       '1592, 1593'       │     0      │       0        │   3.47   │     3.46     │    3.55     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      12      │           '-'            │   89.18    │     78.85      │ 1680.05  │   1488.25    │   1886.53   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s

Other

No response

About this issue

Original URL
State: open
Created 2 years ago
Comments: 31 (8 by maintainers)

Most upvoted comments

Hi all, We have the same issue as this topic. We used:

FROM cypress/browsers:node16.18.0-chrome107-ff106-edge
RUN npx playwright install-deps webkit
RUN npm install --save-dev playwright-webkit

Cypress:        12.8.1                                                                         
Browser:        WebKit 16 (headless)                                                           
Node Version:   v16.18.0 (/usr/local/bin/node)

When we run test: DEBUG=cypress:server:* npx cypress run --browser webkit

We tried many proposed solutions, but they didn’t work:

WEBKIT_DISABLE_COMPOSITING_MODE=1 
CYPRESS_NO_COMMAND_LOG=1 
and use --shm-size 2G

ggrek1988 on Apr 27, 2023

Summing up what we know so far:

The issue still seems to occur even after reverting 10.11.0’s video refactor: [Cypress Webkit Ci Run] Test stuck during Ci run #24896 (comment)

The issue could possibly, maybe, also be occurring on Chrome, though this could also be a red herring/different issue: [Cypress Webkit Ci Run] Test stuck during Ci run #24896 (comment)

The issue seems to only crop up on “complex” test runs - although we also run “complex” tests in WebKit. So it seems more likely that this is a specific test case that is now broken in WebKit in 10.11.0.

Issue seems to occur whether or not we are recording to Cypress Cloud.

@TomdeHaan To further narrow down this issue, the best thing would be if a reproduction could be shared with us, containing all required test code and app code to run these tests. If it’s at all possible, sending a reproduction to support@cypress.io or sharing in this thread would be a huge step towards solving this.

If it’s not possible to share a repro, here’s some more avenues you could explore. I’m hoping to discover the specific testing steps that are causing a hang after 10.11.0:

Is it failing on the same test file every time? For example, in the OP, it seems to hang on tv-internet.cy.ts.

Can you try to bisect to a specific test that causes the failure? Ex. skip/delete half of the tests, run CI, see if it fails, and repeat until a problematic test has been isolated.

We made contact with Brittani within Cypress regarding the sharing of our code. She will probably make contact with you about this.

TomdeHaan on Mar 9, 2023

@chrisbreiding / @emilyrohrbough, there are a few points made within this thread that point to this not being a Cloud issue (David not finding anything in the logs, Tom indicating failed runs regardless of whether he was recording to Cypress Cloud, and Tom indicating that v10.10.0 allowed for successful completion of their jobs). Unless either of you has something additional that you feel we should investigate on the Cloud side, we’re going to mark the investigation on our side complete. Before just disappearing from the conversation, though, I wanted to follow up with you to ensure this is understood (and agreed upon) to be on the App side again. I’m commenting here for visibility/transparency, but if you want to discuss more quickly than GH Issue back-and-forth comments, hit me up in Slack.

rockhold on Jan 13, 2023

After reading the message from @sergiubologa we did a run with Cypress version 10.10.0. And we can confirm that we had a successful run. Meaning, that webkit and chrome both run, finished and send a report to cypress cloud.

We will do some more runs today, to further confirm this.

TomdeHaan on Jan 13, 2023

@TomdeHaan How many times did you run without recording to the dashboard? Was each run successful? When recording to the dashboard, does it always hang or does it ever pass?

Also are you using a custom preprocessor? I am wondering what testFilter is in the example.

Recap of the deep dive into the logs… I am noticing your second test is not sending the mocha pass event or the mocha test end event. It seems the test is either 1-finishing and we aren’t correctly wrapping up the test (seems unlikely) or 2- the browser tab is hanging or crashing and test runner and the test is hanging waiting on the test results to be returned.

Has anyone on the team been able to reproduce this locally when running the the docker image you use in CI? When able to repro, have you been able to run in run mode with the headed browser to try & observe what might be happening?

@emilyrohrbough because of holiday vacations we can’t check it currently. But we will come back on this very shortly.

TomdeHaan on Jan 2, 2023