CodeceptJS: Headless Chrome slows tests by 10x
What are you trying to achieve?
I am trying to run my tests in headless Chrome using the --headless
argument. I expect the tests to run in approximately the same amount of time as the usual runs, as running a clean webdriverio script runs faster headless than in a normal browser window.
What do you get instead?
The tests run about ten times as slow, using 44 seconds instead of 4 seconds.
$ time $(npm bin)/codeceptjs run --verbose > headless-output.txt
real 0m44.973s
user 0m0.992s
sys 0m0.068s
$ nvim codecept.json # removing --headless option
developer at developer-XPS-13-9360 in ~/dev/dx-nuflo/tests/acceptance (acceptance-testing)
$ time $(npm bin)/codeceptjs run --verbose > output.txt
real 0m4.879s
user 0m0.768s
sys 0m0.088s
verbose output from headless run
CodeceptJS v0.6.3
Using test root "/home/developer/dev/dx-nuflo/tests/acceptance"
Top level-routes --
[1] Starting recording promises
Emitted | suite.before ([object Object])
[1] Queued | hook WebDriverIO._beforeSuite()
a page to add a new encounter should be rendered
> [Session] Starting singleton browser session
[2] Starting recording promises
Emitted | test.before
[2] Queued | hook WebDriverIO._before()
Emitted | test.start ([object Object])
Emitted | step.before (I am on page "/encounter/new")
[2] Queued | amOnPage: "/encounter/new"
Emitted | step.after (I am on page "/encounter/new")
[2] Queued | return step result
Emitted | step.before (I see "LOGG INN")
[2] Queued | see: "LOGG INN"
Emitted | step.after (I see "LOGG INN")
[2] Queued | return step result
[2] Queued | fire test.passed
[2] Queued | finish test
Emitted | step.start (I am on page "/encounter/new")
• I am on page "/encounter/new"
Emitted | step.start (I see "LOGG INN")
• I see "LOGG INN"
Emitted | test.passed ([object Object])
✓ OK in 22317ms
Emitted | test.after
[2] Queued | hook WebDriverIO._after()
> [Session] cleaning cookies and localStorage
fooScen
[3] Starting recording promises
Emitted | test.before
[3] Queued | hook WebDriverIO._before()
Emitted | test.start ([object Object])
Emitted | step.before (I am on page "/encounter/new")
[3] Queued | amOnPage: "/encounter/new"
Emitted | step.after (I am on page "/encounter/new")
[3] Queued | return step result
Emitted | step.before (I see "FOOLOGG INN")
[3] Queued | see: "FOOLOGG INN"
Emitted | step.after (I see "FOOLOGG INN")
[3] Queued | return step result
[3] Queued | fire test.passed
[3] Queued | finish test
Emitted | step.start (I am on page "/encounter/new")
• I am on page "/encounter/new"
Emitted | step.start (I see "FOOLOGG INN")
• I see "FOOLOGG INN"
[3] Error | Error
[3] Starting <teardown> session
Emitted | test.failed ([object Object])
[3] <teardown> Queued | hook WebDriverIO._failed()
[3] <teardown> Queued | () => done(err)
[3] <teardown> Stopping recording promises
> Screenshot has been saved to /home/developer/dev/dx-nuflo/tests/acceptance/output/fooScen.failed.png
✖ FAILED in 11599ms
Emitted | test.after
[3] <teardown> Queued | hook WebDriverIO._after()
> [Session] cleaning cookies and localStorage
[4] Starting recording promises
Emitted | suite.after ([object Object])
[4] Queued | hook WebDriverIO._afterSuite()
-- FAILURES:
1) Top level-routes: fooScen:
expected web page to include "FOOLOGG INN"
+ expected - actual
-Innlogging
-Brukernavn
-Passord
-LOGG INN
+FOOLOGG INN
Scenario Steps:
- I.see("FOOLOGG INN") at Test.Scenario (top-level-routes_test.js:18:5)
- I.amOnPage("/encounter/new") at Test.Scenario (top-level-routes_test.js:14:5)
Error
at Object.<anonymous> (/home/developer/dev/dx-nuflo/node_modules/codeceptjs/lib/helper/WebDriverIO.js:1129:51)
at Object.exec (/home/developer/dev/dx-nuflo/node_modules/webdriverio/build/lib/helpers/safeExecute.js:28:24)
at Object.resolve (/home/developer/dev/dx-nuflo/node_modules/webdriverio/build/lib/webdriverio.js:189:29)
at /home/developer/dev/dx-nuflo/node_modules/webdriverio/build/lib/webdriverio.js:538:32
at _fulfilled (/home/developer/dev/dx-nuflo/node_modules/q/q.js:854:54)
at self.promiseDispatch.done (/home/developer/dev/dx-nuflo/node_modules/q/q.js:883:30)
FAIL | 1 passed, 1 failed // 34s
Emitted | global.result ([object Object])
real 0m44.973s
user 0m0.992s
sys 0m0.068s
verbose output from normal (non-headless) run
CodeceptJS v0.6.3
Using test root "/home/developer/dev/dx-nuflo/tests/acceptance"
Top level-routes --
[1] Starting recording promises
Emitted | suite.before ([object Object])
[1] Queued | hook WebDriverIO._beforeSuite()
a page to add a new encounter should be rendered
> [Session] Starting singleton browser session
[2] Starting recording promises
Emitted | test.before
[2] Queued | hook WebDriverIO._before()
Emitted | test.start ([object Object])
Emitted | step.before (I am on page "/encounter/new")
[2] Queued | amOnPage: "/encounter/new"
Emitted | step.after (I am on page "/encounter/new")
[2] Queued | return step result
Emitted | step.before (I see "LOGG INN")
[2] Queued | see: "LOGG INN"
Emitted | step.after (I see "LOGG INN")
[2] Queued | return step result
[2] Queued | fire test.passed
[2] Queued | finish test
Emitted | step.start (I am on page "/encounter/new")
• I am on page "/encounter/new"
Emitted | step.start (I see "LOGG INN")
• I see "LOGG INN"
Emitted | test.passed ([object Object])
✓ OK in 2748ms
Emitted | test.after
[2] Queued | hook WebDriverIO._after()
> [Session] cleaning cookies and localStorage
fooScen
[3] Starting recording promises
Emitted | test.before
[3] Queued | hook WebDriverIO._before()
Emitted | test.start ([object Object])
Emitted | step.before (I am on page "/encounter/new")
[3] Queued | amOnPage: "/encounter/new"
Emitted | step.after (I am on page "/encounter/new")
[3] Queued | return step result
Emitted | step.before (I see "FOOLOGG INN")
[3] Queued | see: "FOOLOGG INN"
Emitted | step.after (I see "FOOLOGG INN")
[3] Queued | return step result
[3] Queued | fire test.passed
[3] Queued | finish test
Emitted | step.start (I am on page "/encounter/new")
• I am on page "/encounter/new"
Emitted | step.start (I see "FOOLOGG INN")
• I see "FOOLOGG INN"
[3] Error | Error
[3] Starting <teardown> session
Emitted | test.failed ([object Object])
[3] <teardown> Queued | hook WebDriverIO._failed()
[3] <teardown> Queued | () => done(err)
[3] <teardown> Stopping recording promises
> Screenshot has been saved to /home/developer/dev/dx-nuflo/tests/acceptance/output/fooScen.failed.png
✖ FAILED in 1299ms
Emitted | test.after
[3] <teardown> Queued | hook WebDriverIO._after()
> [Session] cleaning cookies and localStorage
[4] Starting recording promises
Emitted | suite.after ([object Object])
[4] Queued | hook WebDriverIO._afterSuite()
-- FAILURES:
1) Top level-routes: fooScen:
expected web page to include "FOOLOGG INN"
+ expected - actual
-Innlogging
-Brukernavn
-Passord
-LOGG INN
+FOOLOGG INN
Scenario Steps:
- I.see("FOOLOGG INN") at Test.Scenario (top-level-routes_test.js:18:5)
- I.amOnPage("/encounter/new") at Test.Scenario (top-level-routes_test.js:14:5)
Error
at Object.<anonymous> (/home/developer/dev/dx-nuflo/node_modules/codeceptjs/lib/helper/WebDriverIO.js:1129:51)
at Object.exec (/home/developer/dev/dx-nuflo/node_modules/webdriverio/build/lib/helpers/safeExecute.js:28:24)
at Object.resolve (/home/developer/dev/dx-nuflo/node_modules/webdriverio/build/lib/webdriverio.js:189:29)
at /home/developer/dev/dx-nuflo/node_modules/webdriverio/build/lib/webdriverio.js:538:32
at _fulfilled (/home/developer/dev/dx-nuflo/node_modules/q/q.js:854:54)
at self.promiseDispatch.done (/home/developer/dev/dx-nuflo/node_modules/q/q.js:883:30)
FAIL | 1 passed, 1 failed // 4s
Emitted | global.result ([object Object])
real 0m4.879s
user 0m0.768s
sys 0m0.088s
Test source code
Feature('Top level-routes');
Scenario('a page to add a new encounter should be rendered', (I) => {
I.amOnPage('/encounter/new');
I.see('LOGG INN');
});
// just to see how a failing test looks
Scenario('fooScen', (I) => {
I.amOnPage('/encounter/new');
I.see('FOOLOGG INN');
});
Details
- CodeceptJS version: 0.6.3
- NodeJS Version: 8.1.2
- Operating System: Ubuntu 16.04 LTS
- WebDriverIO: 4.8.0
- Configuration file:
This is my configuration:
{
"tests": "./*_test.js",
"timeout": 5000,
"output": "./output",
"helpers": {
"WebDriverIO" : {
"browser": "chrome",
"desiredCapabilities": {
"browserName": "chrome",
"chromeOptions": {
"args": [
"headless",
"disable-gpu"
]
}
},
"windowSize": "320x568",
"restart": false,
"timeouts": {
"script": 10000,
"page load": 3000,
"implicit" : 5000
},
"url": "http://localhost:23450"
}
},
"include": {
"I": "./steps_file.js"
},
"bootstrap": false,
"mocha": {},
"name": "acceptance tests",
"windowSize": "320x568"
}
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 8
- Comments: 28 (3 by maintainers)
Commits related to this issue
- Improve headless Chrome performance And decrease single test timeout, optimistically. See https://github.com/Codeception/CodeceptJS/issues/561#issuecomment-373666779 — committed to Opetushallitus/omatsivut by timorantalaiho 6 years ago
- Improve headless Chrome performance And decrease single test timeout, optimistically. See https://github.com/Codeception/CodeceptJS/issues/561#issuecomment-373666779 — committed to Opetushallitus/omatsivut by timorantalaiho 6 years ago
- Add possible fix to chrome stucked Find this fix at https://github.com/codeceptjs/CodeceptJS/issues/561 — committed to il12/livekit-recorder by il12 2 years ago
Try these two ChromeOptions:
I dunno in wich forum it was but this did the trick for me
Only thing that worked for me is adding UserAgent to the options:
options.add_argument("user-agent=Chrome/110.0.5481.77")
Ok so after looking into it a bit more it seems that avoiding using headless chrome is the solution for me. Instead I use the pyvirtualdisplay library and set its visibilty to false. This sped up my performance 10x. Here is the code:
with output:
instead of:
in my case, add the following argument solve my problem
chrome_options.add_argument('blink-settings=imagesEnabled=false')
I saw some people get this to work and others do not have any success with it:
I did not have success at first ether, then I looked at my code and realized that I have called setChromeOptions() more than one time and wondered if options are getting over-written. I moved things around to make only one call to
setChromeOptions()
and it worked.You can ignore the lines dealing with hiding images, windows size etc. if you choose.
I’m seeing a similar issue using Puppeteer and Chromium (no WebDriver), the proxy arguments do not seem to make a difference in my case. This may be an issue related to Chromium.
macOS: 10.13.4 Node: 8.10.0
headless: true
results in7747.209ms
headless: false
results in4775.771ms
@krrahulmanikanta It workes fine for me . But I am in Windows7 System.setProperty(“path to driver); ChromeOptions chromeOptions = new ChromeOptions(); chromeOptions.addArguments(”–headless"); chromeOptions.addArguments(“window-sized1200,600”); chromeOptions.addArguments(“–proxy-server=‘direct://’”); chromeOptions.addArguments(“–proxy-bypass-list=*”); ChromeDriver driver = new ChromeDriver(chromeOptions);