chromium: [BUG] No longer working in Lambda

package.json:

    "@sparticuz/chromium": "123.0.1",
    "puppeteer-core": "22.6.4"

code:

  chromium.setGraphicsMode = false;

  console.log("## Launching browser");
  const browser = await puppeteer.launch({
    args: chromium.args,
    defaultViewport: chromium.defaultViewport,
    executablePath: await chromium.executablePath(),
    headless: chromium.headless,
    ignoreHTTPSErrors: true,
  });

  console.log("## Creating new page");
  const page = await browser.newPage();

  console.log("## Setting user agent");
  await page.setUserAgent("Lambda PDF");

  console.log("## Going to " + url);
  await page.goto(url, {
    waitUntil: ["domcontentloaded", "networkidle0"],
  });

It’s freezing after “Creating new page” appears in the console logs.

Lambda 18.x (tried with 20.x too) with 2GB memory Runtime version ARN: arn:aws:lambda:eu-west-2::runtime:b475b23763329123d9e6f79f51886d0e1054f727f5b90ec945fcb2a3ec09afdd

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 11
  • Comments: 48 (3 by maintainers)

Commits related to this issue

Most upvoted comments

Downgrading to:

    "@sparticuz/chromium": "122.0.0",
    "puppeteer-core": "22.5.0"

Seems to have worked. Leaving this open just to verify I didnt make a mistake selecting the wrong package versions?

Did AWS change something with lambdas and lambda layers that is biting all of us?

Yes, AWS updated some libraries that are causing issues with the libraries included in this package. I’m in contact with the team at AWS and we will be working on a solution.

I would recommend pinning the lambda execution environment while this issue gets sorted.

Yes, AWS updated some libraries that are causing issues with the libraries included in this package. I’m in contact with the team at AWS and we will be working on a solution.

I would recommend pinning the lambda execution environment while this issue gets sorted.

@Sparticuz any news on this? Pinning to older version for a long run is not perfect solution for security reasons.

We’ve worked around it by rolling back to the previous Lambda Runtime ARN.

How do you do this with Serverless framework?

@ryan-concora Specify the run time management at the “provider” or “function” level in your serverless.yml file:

functions:
  hello:
    runtimeManagement:
      mode: manual
      arn: arn:aws:lambda:us-east-1::runtime:0cdcfbdefbc5e7d3343f73c2e2dd3cba17d61dea0686b404502a0c9ce83931b9

The ARN version in the example above works for me (The offending ARN is mentioned in OP: https://github.com/Sparticuz/chromium/issues/271#issue-2271372222) Reference: https://www.serverless.com/framework/docs-providers-aws-guide-functions

This works for us.

We migrated to a previous version of Runtime on lambda(Node.Js 18.*)

ARN: arn:aws:lambda:ap-south-1::runtime:0cdcfbdefbc5e7d3343f73c2e2dd3cba17d61dea0686b404502a0c9ce83931b9

package.json for compatibility.

"@sparticuz/chromium": "^119.0.0",
"puppeteer": "^21.5.2"

setting this works chromium.setGraphicsMode = true;

INIT_START Runtime Version: nodejs:18.v24	Runtime Version ARN: arn:aws:lambda:eu-central-1::runtime:c09960ad0af4321e1a7cf013174f7c0d7169bf09af823ca2ad2f93c72ade708a

INIT_START Runtime Version: nodejs:18.v24	Runtime Version ARN: arn:aws:lambda:ap-southeast-1::runtime:c09960ad0af4321e1a7cf013174f7c0d7169bf09af823ca2ad2f93c72ade708a

Working with runtime above by configure lambda runtime manually

@raducretu you can specify the exact version of the runtime under something called “runtime management configuration”. If you look at your logs from when things worked (a few days ago) you should see which arn they are using as it’s printed in the INIT_START. Look for your old arn that worked, and then go in and change it to that for your Lambda.

More info here: https://docs.aws.amazon.com/lambda/latest/dg/runtimes-update.html

FWIW I was able to get it working with:

"@sparticuz/chromium": "122.0.0",
"puppeteer-core": "^21.1.1",
"puppeteer-extra": "^3.3.6",
"puppeteer-extra-plugin-stealth": "^2.11.2",

Facing the same issue.

Error: spawn Unknown system error -8

"@sparticuz/chromium": "^110.0.0",
"puppeteer": "19.6.0"
const browser = await puppeteer.launch({
    dumpio: true,
    args: chromium.args,
    executablePath: await chromium.executablePath(),
    headless: true,
    ignoreHTTPSErrors: true,
})

Tried upgrading to

Downgrading to:

    "@sparticuz/chromium": "122.0.0",
    "puppeteer-core": "22.5.0"

Seems to have worked. Leaving this open just to verify I didnt make a mistake selecting the wrong package versions?

This still did not work. Any other ideas ??

Our solution:

  • Added lambda layer, pulling from s3 doesn’t seem to be working
  • Used this as executable path (examples need updating chromium.executablePath(‘/opt/nodejs/node_modules/@sparticuz/chromium/bin’;
  • Removed “@sparticuz/chromium-min”: “122.0.0”
  • Added as dev dep “@sparticuz/chromium”: “122.0.0”
  • “puppeteer-core”: “22.5.0”
  • Used v122.0.0 lambda layer

This works for us.

We migrated to a previous version of Runtime on lambda(Node.Js 18.*)

ARN: arn:aws:lambda:ap-south-1::runtime:0cdcfbdefbc5e7d3343f73c2e2dd3cba17d61dea0686b404502a0c9ce83931b9

package.json for compatibility.

"@sparticuz/chromium": "^119.0.0",
"puppeteer": "^21.5.2"

This worked, thankyou

I rolled back manualy the runtime version and it worked

We’ve worked around it by rolling back to the previous Lambda Runtime ARN.

We have also started facing same issue. We have not update package version from 3 months. While trying to figure out the issue, we found that it is not working only when html contains https URLs as we had some https URLs in img tag and link tag. With http urls, it is working fine.

We had the following package.json

 "@sparticuz/chromium": "^109.0.5",
 "puppeteer-core": "^19.4.0",

Same thing is working fine with https URLs after updating package version to (as mentioned by @fire015 )

"@sparticuz/chromium": "122.0.0",
"puppeteer-core": "22.5.0"

version 123.0.1 seems to be working again on my end. not sure what the root cause is. I have tested generating more than 100 PDFs today and it works properly.

lib version: https://github.com/Sparticuz/chromium/releases/tag/v123.0.1

package.json

"@sparticuz/chromium": "123.0.1"
"puppeteer-core": "22.7.1"

code:

import puppeteer, { Browser } from 'puppeteer-core';
import chromium from '@sparticuz/chromium';

chromium.setGraphicsMode = true;

const browser = await puppeteer.launch({
  args: [...chromium.args, '--disable-gpu'],
  defaultViewport: chromium.defaultViewport,
  executablePath: await chromium.executablePath(
    '/opt/nodejs/node_modules/@sparticuz/chromium/bin'
  ),
  headless: chromium.headless,
});

const page = await browser.newPage(); // <-- was causing lambda timeout 3 weeks ago


const signedUrl = await getSignedUrl(s3, new GetObjectCommand({
    Bucket: props.bucket,
    Key: props.htmlFilename,
  }));

await page.goto(signedUrl, { waitUntil: 'networkidle0' });
await page.pdf({
    format: 'A4'
});
  
await page.close()

Lambda runtime

INIT_START Runtime Version: nodejs:18.v30	Runtime Version ARN: arn:aws:lambda:ap-southeast-2::runtime:f89c264158db39a1cfcbb5f9b3741413df1cfce4d550c9a475a67d923e19e2f4

edit: set graphics mode to true seems fix the problem

To everyone who’s reading this thread looking for answers for Vercel, what eventually worked for me:

(I’m not sure what here is responsible for a fix)

  1. Set engine in project settings to 20.x
  2. Switched from pnpm to npm
  3. Updated @sparticuz/chromium-min to newest (123.0.1 as of writing)
  4. Updated chromium tar link also to newest (chromium-v123.0.1-pack.tar as of writing)

Just in case: puppeteer-core is “^21.3.8”

We’ve worked around it by rolling back to the previous Lambda Runtime ARN.

How do you do this with Serverless framework?

serverless: maybe need version 3.x+ since it is a new feature last year, https://www.serverless.com/framework/docs-providers-aws-guide-functions aws cli: https://docs.aws.amazon.com/cli/latest/reference/lambda/put-runtime-management-config.html You should use lambda runtime management config, and find the old lambda arn in previous log which you could success running. “Runtime Version ARN: xxxx”


import { custom } from './custom';
import { functions } from './functions';
import { plugins } from './plugins';
import { stepFunctions } from './step-functions';

export const main = {
    service: some-report-fn',
    frameworkVersion: '3',
    useDotenv: true,
    configValidationMode: 'error',
    package: {
        individually: true,
    },
    provider: {
        name: 'aws',
        runtime: 'nodejs18.x',
        runtimeManagement: {
            mode: 'manual',
            arn: 'arn:aws:lambda:${opt:region}::runtime:${ssm:/configs/LAMBDA_NODEJS18_RUNTIME_ARN}',
        },

how we do now is store the ARN into SSM then map to runtimemanagement > arn

For everyone that is still experiencing issues after pinning the runtime version: check if you’re properly closing the browser at the end of your lambda.

Our function was running fine for over a year with puppeteer-core 19.7.4 and chromium 111.0.0. After pinning the runtime version, it was kind of working again, but crashed regularly under load. Turns out we never closed the browser, which hasn’t been an issue before. Changing

try {
    const browser = await puppeteer.launch({
        // ...
    });
    const page = await browser.newPage();
    // ... actions
    await page.close();
} catch {
    // ... error handling
}

to

var browser;
try {
    browser = await puppeteer.launch({
        // ...
    });
    const page = await browser.newPage();
    // ... actions
    for (const page of await browser.pages()) {
        await page.close();
    }
} catch {
    // ... error handling
} finally {
    if (browser) {
        await Promise.race([browser.close(), browser.close(), browser.close()]);
    }
}

along with pinning the version fixed the issues for us, it’s been running fine for a few hours now.

I am curious about the root cause, seems it relate to SSL lib update, but why GraphicsMode can make it solved.

Removing our disabling of the graphics mode solved this for us:

- chromium.setGraphicsMode = false
+ // FIXME: re-enable in Chromium 124
+ // ref: https://github.com/Sparticuz/chromium/issues/271#issuecomment-2085784736
+ // chromium.setGraphicsMode = false

with the following versions:

    "@sparticuz/chromium": "123.0.1",
    "playwright-core": "1.43.1",

edit: actually, not sure about this, we’re still seeing some errors

Make sure to leave graphics mode enabled. The option will be fixed in the next version. I’ve been traveling so I haven’t had time to update to chromium 124.