unlighthouse: Unlighthouse stuck at "Waiting for routes..." screen

Describe the bug

unlighthouse dashboard does not load any data, and no error message at all is ever displayed, when using on a Linux distribution with Chromium installed. The default browser on this machine is Firefox.

The problem happens regardless of the website scanned.

unlighthouse-puppeteer behaves exactly the same way.

How to reproduce

Run unlighthouse as instructed in the documentation:

npx unlighthouse --site https://my-website.com

Confirm the npx prompt to install required package.

Observed result

In the command line, messages confirm the successful connection to the website with status 200, the eventual detection of a sitemap, and number of routes initially discovered.
In the command line, a deprecation warning is displayed:

  Puppeteer old Headless deprecation warning:
    In the near feature `headless: true` will default to the new Headless mode
    for Chrome instead of the old Headless implementation. For more
    information, please see https://developer.chrome.com/articles/new-headless/.
    Consider opting in early by passing `headless: "new"` to `puppeteer.launch()`
    If you encounter any bugs, please report them to https://github.com/puppeteer/puppeteer/issues/new/choose.

A tab open in the default web browser and loads unlighthouse dashboard
The dashboard stays empty with Waiting for routes... spinner displayed. Worker progress value remains at 0%, even minutes after the launch.

Expected result

In the command line, messages confirm the successful connection to the website with status 200, the eventual detection of a sitemap, and number of routes initially discovered.
A tab open in the default web browser and loads unlighthouse dashboard
The dashboard content gets populated progressively as the crawling progresses.

Reproduction

c.f. bug description

System / Nuxt Info

System:
    OS: Linux 4.19 BunsenLabs GNU/Linux 10.5 (Lithium)
    CPU: (8) x64 Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz
    Memory: 9.83 GB / 15.35 GB
    Container: Yes
    Shell: 5.0.3 - /bin/bash
  Binaries:
    Node: 16.19.1 - ~/.nvm/versions/node/v16.19.1/bin/node
    npm: 8.19.3 - ~/.nvm/versions/node/v16.19.1/bin/npm
  Browsers:
    Chromium: 90.0.4430.212
    Firefox: 112.0.2

About this issue

Original URL
State: closed
Created a year ago
Reactions: 9
Comments: 23 (6 by maintainers)

Most upvoted comments

Hey, can you try out version 0.7.0 with the --debug option and provide the output? I am not able to replicate

Thanks for the follow-up. Here is the debug log output:

$ npx unlighthouse@0.7.1 --site example.com --debug --no-cache
Need to install the following packages:
  unlighthouse@0.7.1
Ok to proceed? (y) 
(#########⠂⠂⠂⠂⠂⠂⠂⠂⠂) ⠹ idealTree:f748e5d196c066c9: timing idealTree:#root Completed in 57ms

(#########⠂⠂⠂⠂⠂⠂⠂⠂⠂) ⠸ idealTree:f748e5d196c066c9: timing idealTree:#root Completed in 57ms



npm WARN deprecated intl-messageformat-parser@1.8.1: We've written a new parser that's 6x faster and is backwards compatible. Please use @formatjs/icu-messageformat-parser
⚙ Starting Unlighthouse at root: /tmp cwd: /tmp                                                                      unlighthouse 18:33:35
⚙ Discovered config definition { config: undefined, sources: [] }                                                    unlighthouse 18:33:35
⚙ Unable to locale page files, disabling route discovery.                                                            unlighthouse 18:33:35
ℹ Using system chrome located at: /usr/bin/chromium.                                                                 unlighthouse 18:33:35
⚙ Post config resolution { routerPrefix: '/',                                                                        unlighthouse 18:33:35
  apiPrefix: '/api',
  cache: false,
  client:
   { groupRoutesKey: 'route.definition.name',
     columns:
      { overview: [Array],
        performance: [Array],
        accessibility: [Array],
        'best-practices': [Array],
        seo: [Array] } },
  scanner:
   { customSampling: {},
     ignoreI18nPages: true,
     maxRoutes: 200,
     skipJavascript: true,
     samples: 1,
     throttle: false,
     crawler: true,
     dynamicSampling: 5,
     sitemap: true,
     robotsTxt: true,
     device: 'mobile' },
  server: { port: 5678, showURL: false, open: true },
  discovery: false,
  root: '/tmp',
  outputPath: '.unlighthouse',
  debug: true,
  puppeteerOptions:
   { headless: true,
     ignoreHTTPSErrors: true,
     defaultViewport:
      { mobile: true, width: 412, height: 823, deviceScaleFactor: 1.75, disabled: false },
     executablePath: '/usr/bin/chromium' },
  puppeteerClusterOptions:
   { monitor: true,
     workerCreationDelay: 500,
     retryLimit: 3,
     timeout: 300000,
     maxConcurrency: 8,
     skipDuplicateUrls: false,
     retryDelay: 2000,
     concurrency: 3,
     puppeteer:
      PuppeteerNode {
        _changedProduct: false,
        _isPuppeteerCore: true,
        connect: [Function: bound bound connect],
        configuration: [Object],
        defaultBrowserRevision: '113.0.5672.63',
        launch: [Function: bound launch],
        executablePath: [Function: bound executablePath],
        defaultArgs: [Function: bound defaultArgs] } },
  lighthouseOptions:
   { onlyCategories: [ 'performance', 'accessibility', 'best-practices', 'seo' ],
     throttlingMethod: 'provided',
     throttling:
      { rttMs: 0,
        throughputKbps: 0,
        cpuSlowdownMultiplier: 1,
        requestLatencyMs: 0,
        downloadThroughputKbps: 0,
        uploadThroughputKbps: 0 },
     formFactor: 'mobile',
     screenEmulation:
      { mobile: true, width: 412, height: 823, deviceScaleFactor: 1.75, disabled: false } },
  urls: [],
  site: 'https://example.com',
  hooks: { 'resolved-config': [AsyncFunction: resolved-config] },
  chrome:
   { useSystem: true,
     useDownloadFallback: true,
     downloadFallbackVersion: 1095492,
     downloadFallbackCacheDir: '/home/user/.unlighthouse' } }
⚙ Testing Site https://example.com is valid.                                                                         unlighthouse 18:33:35
✔ Successfully connected to https://example.com. (Status: 200).                                                      unlighthouse 18:33:36
⚙ Creating Unlighthouse                                                                                              unlighthouse 18:33:36
⚙ Setting Unlighthouse Site URL [Site: https://example.com]                                                          unlighthouse 18:33:36
⚙ Setting Unlighthouse Server Context [Server: http://localhost:5678/]                                               unlighthouse 18:33:36
› Creating new Unrouted ctx: unlighthouse-api {                                                                  unlighthouse-api 18:33:36
  debug: true,
  prefix: '/',
  middleware: [],
  hooks: {
    'serve:before-route': [Function: serve:before-route]
  },
  plugins: [
    {
      meta: [Object],
      setup: [AsyncFunction: setup]
    }
  ],
  presets: [
    {
      meta: [Object],
      setup: [AsyncFunction: setup]
    },
    {
      meta: [Object],
      setup: [AsyncFunction: setup]
    }
  ],
  name: 'unlighthouse-api',
  app: {
    use: [Function: use],
    handler: [AsyncFunction (anonymous)] {
      __is_handler__: true
    },
    stack: [],
    options: {}
  }
}
› Using preset: @unrouted/preset-api@0.5.0                                                                       unlighthouse-api 18:33:36
› Using preset: @unrouted/preset-node@0.5.0                                                                      unlighthouse-api 18:33:36
› Using plugin: lazyRouteHandles                                                                                 unlighthouse-api 18:33:36
› Registering route get /__lighthouse.                                                                           unlighthouse-api 18:33:36
› Registering route post /api/reports/rescan.                                                                    unlighthouse-api 18:33:36
› Registering route post /api/reports/:id/rescan.                                                                unlighthouse-api 18:33:36
› Registering route get,head /api/__launch.                                                                      unlighthouse-api 18:33:36
› Registering route get,head /api/ws.                                                                            unlighthouse-api 18:33:36
› Registering route get,head /api/reports.                                                                       unlighthouse-api 18:33:36
› Registering route get,head /api/scan-meta.                                                                     unlighthouse-api 18:33:36
› Registering route get /**.                                                                                     unlighthouse-api 18:33:36
› Registering route get /.                                                                                       unlighthouse-api 18:33:36
› Setting up 9 routes.                                                                                           unlighthouse-api 18:33:36
⚙ cache is disabled, deleting cache folder: .unlighthouse                                                            unlighthouse 18:33:36
⚙ Starting Unlighthouse [Server: http://localhost:5678/ Site: https://example.com Debug: true]                       unlighthouse 18:33:36
⚙ Scanning https://example.com/robots.txt                                                                            unlighthouse 18:33:36

 WARN  You seem to be missing a robots.txt.                                                                          unlighthouse 18:33:37

⚙ Attempting to fetch sitemap at https://example.com/sitemap.xml                                                     unlighthouse 18:33:37
Error occurred during "crawl('https://example.com/sitemap.xml')":
 Error: HTTP Error occurred: Response code 404 (Not Found)
⚙ Fetched sitemap with 0 URLs.                                                                                       unlighthouse 18:33:37
ℹ Sitemap appears to be missing, falling back to crawler mode.                                                       unlighthouse 18:33:37
⚙ Route has been queued. Path: / Name: _index.                                                                       unlighthouse 18:33:37
› GET /api/scan-meta 200 object - 0ms                                                                            unlighthouse-api 18:33:38

   ╭───────────────────────────────────────────────────────────────────────────────────────────╮
   │                                                                                           │
   │   ⛵  Unlighthouse cli @ v0.7.1                                                           │
   │                                                                                           │
   │   ▸ Scanning: https://example.com                                                         │
   │   ▸ Route Discovery: Crawler                                                              │
   │                                                                                           │
   │    💖 Like Unlighthouse? Support the development: https://github.com/sponsors/harlan-zw   │
   │                                                                                           │
   │   Report: http://localhost:5678/                                                          │
   │                                                                                           │
   ╰───────────────────────────────────────────────────────────────────────────────────────────╯

› GET /favicon.ico 200 undefined - 1ms                                                                           unlighthouse-api 18:33:39
› GET /api/scan-meta 200 object - 0ms                                                                            unlighthouse-api 18:33:39
› GET /api/reports 200 object - 0ms                                                                              unlighthouse-api 18:33:39
› GET /api/scan-meta 200 object - 0ms                                                                            unlighthouse-api 18:33:43
› GET /api/scan-meta 200 object - 0ms                                                                            unlighthouse-api 18:33:44
› GET /api/scan-meta 200 object - 1ms                                                                            unlighthouse-api 18:33:48
› GET /api/scan-meta 200 object - 0ms                                                                            unlighthouse-api 18:33:49

mef on May 14, 2023

Still happens to me with ⛵ Unlighthouse cli @ v0.10.6

› GET /api/scan-meta 200 object - 0ms                                                                   unlighthouse-api 15:07:20
› GET /api/scan-meta 200 object - 0ms                                                                   unlighthouse-api 15:07:21
› GET /api/scan-meta 200 object - 0ms                                                                   unlighthouse-api 15:07:25
› GET /api/scan-meta 200 object - 1ms                                                                   unlighthouse-api 15:07:26
› GET /api/scan-meta 200 object - 0ms                                                                   unlighthouse-api 15:07:30
› GET /api/scan-meta 200 object - 0ms                                                                   unlighthouse-api 15:07:31

nothing else happens.

Time remaining is stuck on 5 secs.

Maybe an option to specify browser to use? Chromium is commonly installed on Ubuntu, but yet it seems that Unlighthouse fails without Chrome installed.

nekromoff on Feb 7, 2024