puppeteer-extra: [Bug] My software stucks on browser.close with plugin-stealth 2.7.x version

This should be fixed now through #530 (thanks @dev-hyperweb 👍 )

Successfully published:
 - puppeteer-extra-plugin-stealth@2.7.9
 - puppeteer-extra-plugin-user-data-dir@2.2.13
 - puppeteer-extra-plugin-user-preferences@2.2.13

cc @LinkTree3 @Dam998

This happens for me as well with the default run settings listed in the documentation. browser.close locks up my terminal process. If I remove the puppeteer-extra requirement, and just go with the the vanilla puppeteer, everything is fine. There’s definitely a core issue with this package.

Literally just running the quickstart settings doesn’t work:

// puppeteer-extra is a drop-in replacement for puppeteer,
// it augments the installed puppeteer with plugin functionality.
// Any number of plugins can be added through `puppeteer.use()`
const puppeteer = require('puppeteer-extra')

// Add stealth plugin and use defaults (all tricks to hide puppeteer usage)
const StealthPlugin = require('puppeteer-extra-plugin-stealth')
puppeteer.use(StealthPlugin())

// Add adblocker plugin to block all ads and trackers (saves bandwidth)
const AdblockerPlugin = require('puppeteer-extra-plugin-adblocker')
puppeteer.use(AdblockerPlugin({ blockTrackers: true }))

// That's it, the rest is puppeteer usage as normal 😊
puppeteer.launch({ headless: true }).then(async browser => {
  const page = await browser.newPage()
  await page.setViewport({ width: 800, height: 600 })

  console.log(`Testing adblocker plugin..`)
  await page.goto('https://www.vanityfair.com')
  await page.waitForTimeout(1000)
  await page.screenshot({ path: 'adblocker.png', fullPage: true })

  console.log(`Testing the stealth plugin..`)
  await page.goto('https://bot.sannysoft.com')
  await page.waitForTimeout(5000)
  await page.screenshot({ path: 'stealth.png', fullPage: true })

  console.log(`All done, check the screenshots. ✨`)
  await browser.close()
})

@LinkTree3 awesome, we’re gonna go with this fix then

I reproduced it with this simple script:

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
const locateChrome = require('locate-chrome');
const delay = require('delay');

(async () => {
    console.log('start')
    for (var i = 0; i < 20; i++) {
        (async () => {
            console.log('start')
            try {
                var chromeOptions = {
                    headless: false,
                    ignoreDefaultArgs: ['--disable-extensions'],
                    ignoreHTTPSErrors: true,
                    args: [
                        '--disable-web-security',
                        '--disable-dev-shm-usage',
                        '--no-sandbox',
                        '--disable-setuid-sandbox',
                        '--disable-gpu',
                        '--ignore-certificate-errors',
                        '--enable-features=NetworkService'
                    ],
                };

                let chromePath = await locateChrome()
                chromeOptions.executablePath = chromePath;

                puppeteer.use(StealthPlugin())
                const browser = await puppeteer.launch(chromeOptions);
                const page = await browser.newPage();
                
                await page.goto('https://github.com/berstend/puppeteer-extra/issues/421', { 'waitUntil': 'domcontentloaded', timeout: 60000 });
                await delay(13000);

                var pages = await browser.pages()
                for (let j = 0; j < pages.length; j++) {
                    await pages[j].close();
                }
                await browser.close();
            }
            catch (err) {
                console.log(err)
            }
        })();
        await delay(10000);
    }
})();

this script just open more browsers and then close them continusly with 2.6.7 version everything works good, with 2.7.x after the first browser close, the script freezes

and with 2.7.x if you try to do CTRL + C in terminal it not wll close, with 2.6.7 yes

@LinkTree3 no worries 😃 could you try if changing onClose to onDisconnected (with the original fse.rmdirSync code) does make a difference in your case? https://github.com/berstend/puppeteer-extra/pull/530/files

Yes this one seems to work! I tested it a couple times to make sure 😃

I also added a console log right before and after the fse.rmdirSync call and they both fired after the puppeteer browser closed.

@LinkTree3 no worries 😃 could you try if changing onClose to onDisconnected (with the original fse.rmdirSync code) does make a difference in your case? https://github.com/berstend/puppeteer-extra/pull/530/files

YUP This did the trick! 😃

Sweet! Thanks for testing 👍 So we know that (for whatever reason) removing the temporary profile folder synchronously can cause issues on windows, whereas doing it asynchronously works fine.

I’m gonna prepare a fix + new version within the next days, as time permits.

Thank you so much! looking forward to it 😃

YUP This did the trick! 😃

Sweet! Thanks for testing 👍 So we know that (for whatever reason) removing the temporary profile folder synchronously can cause issues on windows, whereas doing it asynchronously works fine.

I’m gonna prepare a fix + new version within the next days, as time permits.

YUP This did the trick! 😃

rmSync

This did not work.

Ok, one last test 😃

fs.rm(this._userDataDir, { recursive: true, force: true }, console.log)

@LinkTree3 Cool, thanks for testing 😃

Would you be able to test if this change makes a difference (commenting the onClose handler back in)? https://github.com/berstend/puppeteer-extra/pull/502/files

(basically replacing rmdirSync with rmSync)

more specifically the onClose handler will result in this sync call: https://github.com/berstend/puppeteer-extra/blob/0049d6010311505f27e7f3be804bb198e2c09aa2/packages/puppeteer-extra-plugin-user-data-dir/index.js#L67-L76

Alright, to summarize what we know so far:

• The reason this happens in version 2.7 and up is that we started using the puppeteer-extra-plugin-user-data-dir plugin in the user-agent-override evasion (unfortunately that’s necessary to properly spoof locales)
• As to why the user-data-dir plugin causes this behavior is not entirely clear yet, but my gut tells it must be this code: https://github.com/berstend/puppeteer-extra/blob/0049d6010311505f27e7f3be804bb198e2c09aa2/packages/puppeteer-extra-plugin-user-data-dir/index.js#L111-L116

Can someone experiencing this issue monkey patch their ./node_modules/puppeteer-extra-plugin-user-data-dir/index.js file and comment out this onClose handler to see if this fixes it?

Hello, just wanted to report the same problem.

• The problem started when we upgraded 2.6.5 to 2.7.6.
• I did not observe the issue on my MacOS (this is where I’m developing).
• In production we use Windows, and it hangs on close. CPU utilization goes to 100% and the web service doesn’t respond anymore.
• We open multiple browsers. We use launch and real Chrome 89.0.4389.90.
• Reverting back to 2.6.5 fixes the problem.

I tested again my script with 2.7.4 version and I’m getting same error with 2.6.7 works fine

Gotcha. Thanks for the test case, we’ll look into it. 😃