vscode-extension-tester: [Bug] `npm run ui-test` starts a shared VSCode instance, not running test files

Describe the bug

  • I’ve setup vscode-extension-tester[v5.8] in my repo
  • npm run ui-test
  • VSCode opens inside a framed window
  • Nothing happens

Console output

➤ npm run ui-test

> env-switcher@0.4.1 ui-test
> extest setup-and-run './out/ui-test/*.test.js' --code_version 1.71.0 --mocha_config './src/ui-test/setup/.mocharc.json' --extensions_dir '.vscode-test/extensions' --storage '.vscode-test/resources' --code_settings ./src/ui-test/setup/settings.json


                WARNING: You are using the outdated VSCode version '1.71.0'. The latest stable version is '1.80.2'.
            
Downloading VSCode: 1.71.0 / stable
VS Code exists in local cache, skipping download
ChromeDriver 102.0.5005.61 exists in local cache, skipping download
Executing prepublish script 'npm run vscode:prepublish'...
 
> env-switcher@0.4.1 vscode:prepublish
> npm run compile


> env-switcher@0.4.1 compile
> tsc -p ./

 DONE  Packaged: /[redacted]/vscode-env-switcher/env-switcher-0.4.1.vsix (28 files, 387.21KB)
Installing extensions...
(node:5079) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use `code --trace-deprecation ...` to show where the warning was created)
Extension 'env-switcher-0.4.1.vsix' was successfully installed.
Loading mocha configuration from ./src/ui-test/setup/.mocharc.json
LOADED TESTS


Detected user defined code settings
Writing code settings to /[redacted]/vscode-env-switcher/.vscode-test/resources/settings/User/settings.json
Launching browser...
Selenium Manager binary found at /[redacted]/vscode-env-switcher/node_modules/selenium-webdriver/bin/linux/selenium-manager

Note the LOADED TESTS, it’s a console.log I’ve added to the testing file, which is based on statusBar-test.ts in example repo:

import { expect } from 'chai';
import { StatusBar, VSBrowser } from 'vscode-extension-tester';

console.log(`LOADED TESTS`);

describe('Example status bar tests', () => {
  let statusBar: StatusBar;

  before(async () => {
    console.log(`STARTING TESTS`);

    // ...
  });

  it('Generic items', async () => {
    console.log(`STARTING TESTS`);
    // ...
  });
});

When running npm run ui-test, none of the other console logs are logged. The window is interactable, and I’ve opened a folder in it as that’s what the before was supposed to do. When closing the window and rerunning it again with npm run ui-test it will open the same folder and all changes to settings will persist. image

I’ve tried clearing anything related to out/test-resources/test-extensions/node_modules/npm cache clean --force. I’ve tried with the latest VSCode version (1.80.2) and I’ve tried downgrading to the vscode-extension-tester@5.7.1 so it will be the same as in the sample project, separately and together. I’ve tried to align it as much as I can to the sample project, including the resources and output directories. When running tests in the sample project, VSCode window will be opened without a frame and the logs are different:

Detected user defined code settings
Writing code settings to /[redacted]/vscode-extension-tester/sample-projects/helloworld-sample/test-resources/settings/User/settings.json
Launching browser...
Browser ready in 4552 ms
Launching tests...

Expected behavior

  • To run the tests
  • To be able to debug the tests

Environment information

  • OS:
    • Windows - WSL2 with WSLg[Ubuntu 20.04]
    • Linux
    • macOS
  • VSCode version: host[1.75.1], test[1.71.0 and 1.80.2]
  • ExTester version: 5.8 and 5.7.1
  • Node (npm) version: 16.13.0

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 15 (5 by maintainers)

Most upvoted comments

@EcksDy there is not manual mapping of vscode <=> chromedriver

There is mechanism which finds appropriate chromium/chromedriver version based on manifest provided by MIcrosoft for each release of VS Code.

We are supporting last 5 versions from another reason, which is more about complexity of vscode-extension-tester and resource we have to maintain this tool. So the support for last 5 releases means we are aiming to have vscode-extension-tester APi stable and working always with 5 last stable VS Code releases, we do not have resources to ensure better backwards compatibility at this moment.

Hopefully I have answered your question 🙂

Hello, this issue is an side effect of dependency updates in currect vscode-extension-tester release. I will provide fix and next release ASAP.

As workaround you can locally rever changes in package-lock.json file which are made when installing latest vscode-extension-tester. The changes where selenium-webdriver has newer version then 4.10.x needs to be reverted.

Regarding this issue, one of the identified steps was to rollback to selenium-webdriver version 4.10.0; I must say that doing it so proved easier than I thought, and while waiting for the new release with those fixes I suggest the following approach in your package.json;

	"overrides": {
		"vscode-extension-tester": {
			"selenium-webdriver": "4.10.0"
		}
	}

this change will allow you to override the version of the selenium-webdriver found as a dependency in the vscode-extension-tester package.json. Let me know your thoughts!

More info on overrides here

@sergiuilie works for me! thanks