chrome-aws-lambda: The current Chromium Binary doesn't work for AWS nodejs10.x runtime

(same problem as OP), but I also want to express my gratitude to the author for publishing the package and the very rapid response above.

@TimotheeJeannin 👍 That’s my goal, I’ll try to get to it during this weekend.

Version 1.20.1 is published (requires this layer) and should be compatible with Node 10. 😌

Finally put something together to show a quick example of combining this repo with the dependencies for chromedriver in order to run selenium-based tests in lambda using node10: https://github.com/jensbodal/lambda-layer-chromedriver/tree/master/examples/webdriver.io. The relevance to this issue would be that it IS possible to get chrome-aws-lambda working with AL2 and node10.

@FreddyJD I could only find the time to work on it now, but I’m happy to say that I finally have a 7.8MiB zipped Layer that makes this package compatible with Node 10. 😃 I will try to reduce dependencies as much as possible and better structure the overall code, after that I will release a new bundle.

@transitive-bullshit 😱 We can’t have ZEIT Now not being supported. @Bnaya Probably interesting for you too:

Version 1.20.2 now transparently supports nodejs10.x, no layer needed. 😃

Closing this issue now, if you find any problems with the current approach feel free to reopen.

@bhasto I did opt in to the new execution environment on a Node 8 Lambda with Layer arn:aws:lambda:::awslayer:AmazonLinux1803 and things worked normally.

However I do also share the concern about a potential update to Linux 2 on all environments, but if that time comes I’ll definitely be creating and maintaining support Layers for it.

By using Roboto and getting rid of some extra shared objects, I was able to get the layer to 2.5MB.

Not only that, but it seems that the AWS_Lambda_nodejs10.x was upgraded from Node v10.15.1 to v10.16.3 - that means that Brotli support is natively supported, so iltorb is no longer necessary! 🎉

Also, the overall performance seems to be much better, the same code on:

• Node 8: 569.54 ms
• Node 10: 359.38 ms

@alixaxel First of all, appreciate the awesome library you’ve built here.

Just some feedback, I upgraded our lambda functions to nodejs 10.x and upgraded to version 1.20.3 of your library. All seems to be working as normal but will keep an eye on it and report if we start experiencing issues.

Thanks again for your awesome work!

Cheers!

@exentrich That is very very helpful, thanks for sharing! 🙇‍♂️

This is sort a unknown Lambda behavior to me, but I think I know what might be happening. Let me do a little research, I think I should be able to get a fix by EOD today.

It looks like AWS is going to force the new, lightweight environment for all lambdas pretty soon: https://aws.amazon.com/blogs/compute/upcoming-updates-to-the-aws-lambda-execution-environment/

If I’m not mistaken, this will cause problems similar to the ones mentioned above even on Node 8 lambdas. Has anyone managed to make this work in the new environment?

Really glad to see this work coming together 😄

Unfortunately ZEIT now doesn’t currently support adding lambda layers, so if there’s a workaround that doesn’t require using them, I’d be very interested.

Thanks!

@nickychow

From what I could assess, the new Lambda environment is way more lightweight than the 8.10 counterpart: not only it comes with the outdated libnss dependencies you already noticed, I also wasn’t able to locate any Fontconfig configuration file, nor any single font available on the system - grepped for *.ttf, *.otf, …

So it seems like a proper packaging for Node 10 would be substantially larger:

• 1786 KiB in needed linked dependencies (uncompressed)
• 7893 KiB in DejaVu font faces (uncompressed)

Besides that, quite some patching to the environment/files would be required:

• LD_LIBRARY_PATH for shared objects
• PATH for local fonts.conf overlay

In light of all of this, I no longer think the advantages of upgrading to Node 10 outweigh the drawbacks.

I’m uploading the necessary shared objects (they should be rather stable/future-proof) in case someone wants to create a Node 10 compatibility layer on top of this, including whatever font faces are appropriate.

I’m sure this is already known, but LD_LIBRARY_PATH variable includes /opt/lib in the path for the AL2 execution environment. So for that a layer would just need to be added which includes a lib folder and the necessary shared objects.

As for the font, you can set the FONTCONFIG_PATH to /opt/usr/share/fontconfig (or whatever path) and then include a /usr/share/fontconfig/font.conf file in the layer with the following:

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
    <dir>/opt/usr/share/fonts</dir>
    <dir>/tmp/.fonts</dir>
</fontconfig>

This should restore the previous functionality you have for asynchronously loading fonts into /tmp/.fonts as well as allowing others to create layers which include fonts.

I’m testing out preparing a chrome headless binary for AL2 in a different way than the approach in this repo (using amazonlinux docker image to build shared objects and non-chrome binaries), but came across this same error and resolved it by setting the font config path. Not sure yet if everything else is working but I plan on publishing my repo if it works out.

@bhasto pretty sure that the forced update is to 2018.03 (of amazon linux 1), which is an update for sure, but not one that should impact this. That’s not to say that there wont be another update later in the year that forces linux 2 on all lambda environments.

@hasansaghir Node 12.x support is only implemented in version 2.0.1 onwards.

This is what I’m getting with latest version:

AWS_EXECUTION_ENV: 'AWS_Lambda_nodejs12.x',
  FONTCONFIG_PATH: undefined,
  LD_LIBRARY_PATH: '/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib',
  '/tmp/aws/lib': false

UPDATE: it does work when I’m trying to use node10.x, but can’t make it work with node12.x

@Bnaya Including 2.5MB of that extra (Node 10 specific) layer in the NPM package doesn’t seem ideal.

This layer is mostly static, it’s not going to change from version to version, so you should be able to copy it into your project dependencies and reference it from there. In the next minor version I might make this process more transparent and get rid of the layer altogether (still need to hash out some details).

Also to keep in mind is the fact Node8 TLS maintenance will finish the next December 2019.

I don’t think AWS deprecate it in the next months, but for sure will not recommend continuing using node8 after the deadline

@exentrich It seems that your Lambda is in a hot state, but after a while it resets the custom ENV variables somehow… I looked and couldn’t find this behavior documented anywhere unfortunately.

So far I wasn’t able to reproduce this behavior either, so I’m a bit blindsided here.

I’ve published 1.20.3 with a tentative workaround for this - would you mind giving it a try?

I had the same issue when using the module in a layer with Node10.x runtime, but after I updated the layer to 1.20.3, the issue has gone and now everything is working fine!

Thank you!

It’s for other propose: i’m using serverless framework, and i can specify zip file to be uploaded as a layer If the zip will be already inside node_modules/chrome-aws-lambda/xxx i can simply point to there

To avoid the need to maintain a layer in multiple regions, maybe having just a script or a make target that builds the layer would be a good solution ?

Oh wow that was unfortunate, I did everything but forgot to copy my fonts.conf file when building the libs as I described above. I can confirm that my original method does in fact work. Sorry about the noise 😐

I have not yet tested in lambda but have it working with the AL2018 docker image.

My /opt directory looks like

opt:
lib  usr

opt/lib:
libX11.so.6  libXau.so.6  libxcb.so.1

opt/usr:
share

opt/usr/share:
fontconfig  fonts

opt/usr/share/fontconfig:
conf.avail  fonts.conf

opt/usr/share/fontconfig/conf.avail:
61-stix.conf

opt/usr/share/fonts:
stix

opt/usr/share/fonts/stix:
STIX-Bold.otf  STIX-BoldItalic.otf  STIX-Italic.otf  STIX-Regular.otf

And before calling my test I run process.env.FONTCONFIG_PATH = '/opt/usr/share/fontconfig';

@SergeyAvd I don’t see why it wouldn’t work, as long as (some of) the dependencies here are installed:

https://github.com/alixaxel/chrome-aws-lambda/blob/9fe9b7014a2327402937e9570fdb3d9682ed3314/_/ansible/plays/chromium.yml#L86-L141

I see. I changed the order. Don’t know whether changing it post instantiation has any impact on an executing lambda.

I am getting a socket hangup error now with rejigged layer.

    // browser = await chromium.puppeteer.launch({
    //   args: chromium.args,
    //   defaultViewport: chromium.defaultViewport,
    //   executablePath: await chromium.executablePath,
    //   headless: chromium.headless,
    // });
    browser = await puppeteer.launch({
      args: ['--no-sandbox', '--disable-gpu', '--single-process'],
      headless:true,
      executablePath:executablePath
    })

2019-10-21T09:39:15.294Z 52c9f3fc-b4f8-4064-822c-3c71ecf64940 INFO LD_LIBRARY_PATH /opt/lib:/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib 2019-10-21T09:39:16.281Z 52c9f3fc-b4f8-4064-822c-3c71ecf64940 ERROR ErrorEvent { target: WebSocket { _events: [Object: null prototype] { open: [Function], error: [Function] }, _eventsCount: 2, _maxListeners: undefined, readyState: 3, protocol: ‘’, _binaryType: ‘nodebuffer’, _closeFrameReceived: false, _closeFrameSent: false, _closeMessage: ‘’, _closeTimer: null, _closeCode: 1006, _extensions: {}, _receiver: null, _sender: null, _socket: null, _isServer: false, _redirects: 0, url: ‘ws://127.0.0.1:37633/devtools/browser/b00164bd-cfa9-4d2f-9dce-bf1965b29fcd’, _req: null }, type: ‘error’, message: ‘socket hang up’, error: { Error: socket hang up at createHangUpError (_http_client.js:323:15) at Socket.socketOnEnd (_http_client.js:426:23) at Socket.emit (events.js:203:15) at endReadableNT (_stream_readable.js:1145:12) at process._tickCallback (internal/process/next_tick.js:63:19) code: ‘ECONNRESET’ } } 2019-10-21T09:39:16.290Z 52c9f3fc-b4f8-4064-822c-3c71ecf64940 ERROR Invoke Error {“errorType”:“Error”,“errorMessage”:“[object Object]”,“stack”:[“Error: [object Object]”," at _homogeneousError (/var/runtime/CallbackContext.js:13:12)“,” at postError (/var/runtime/CallbackContext.js:30:51)“,” at done (/var/runtime/CallbackContext.js:57:7)“,” at fail (/var/runtime/CallbackContext.js:69:7)“,” at Object.fail (/var/runtime/CallbackContext.js:105:16)“,” at Runtime.exports.handler (/var/task/tests/testchromium.js:37:20)“,” at process._tickCallback (internal/process/next_tick.js:68:7)"]}

@exentrich It seems that your Lambda is in a hot state, but after a while it resets the custom ENV variables somehow… I looked and couldn’t find this behavior documented anywhere unfortunately.

So far I wasn’t able to reproduce this behavior either, so I’m a bit blindsided here.

I’ve published 1.20.3 with a tentative workaround for this - would you mind giving it a try?

@Bnaya Rather simple with a custom Layer, but you also have the option of using runtime fonts.

@soukicz It seems like DejaVu has a slightly wider ex (character width) than Roboto, so the behavior you’re describing shouldn’t happen. I think that Roboto might be the right choice here:

• 3.3MB smaller than DejaVu
• friendly license (I actually don’t know which license DejaVu has)
• very comprehensive coverage of different scripts
• developed by Google and used by default on Android (screenshots will look more consistent)
• the most requested font on Google Fonts

I’d be interested in learning why not having Node 10 is holding you back: is it simply a matter of upgrading for the sake of it or is there another critical reason? If so, I’d be happy to reconsider this.

@alixaxel my project depends on Node packages which require:

"engines": {
    "node": ">=9.11.2"
 },

…which prevents me from using this package. I’d be happy to provide more details if that would be helpful. But for me, there is a real need for Node 10 support.

Thank you for your work on this very helpful project!

Not a critical reason for me at the moment but I’m running Google Lighthouse in a lambda and since version 5 node 10 is required (I’m currently using version 4 and node 8).

Not having node10 support is really holding us back.

While this issue is not solved we created a docker image and are now running the puppeteer logic as a AWS Fargate service with an API called by lambda. Then we were able to move to node 10.

But to tell the truth, even when this issue is solved we will keep the logic on Fargate to not risk having the same problem when they launch Amazon Linux 3, 4, 5…

This project is awesome, but all this trouble made me think that maybe it’s better to keep OS dependant logic on a controlled environment.

@LiamBateman

I don’t think so. The General Update happening on July 16th changes Amazon Linux 2017.09 to Amazon Linux 2018.03. I’ve tested my use case on 2018.03 and chrome-aws-lambda appears to work. This issue as about Amazon Linux 2. While General Update:

does not apply to the recently announced Node.js v10 runtime which today runs on Amazon Linux 2

@jensbodal This binary needs to be compiled with use_bundled_fontconfig = true:

https://github.com/alixaxel/chrome-aws-lambda/blob/8351d433bf2402a0a979f629c800a84ef684b4ad/_/ansible/plays/chromium.yml#L247

I’m not sure is that skips the FONTCONFIG_PATH lookup or not (and I don’t have time to check right now). I suppose setting PATH=$PATH:$FONTCONFIG_PATH should work, did you try that?

So I also tried to use the binary bundled with this repo but it doesn’t appear to look at the FONTCONFIG_PATH for fonts. Any idea if the path is hardcoded to /etc/fonts or something similar? With both chrome installed via RPM and chromium downloaded from chromium.org I’m able to alleviate the font issue with my method above, but with the decompressed chrome binary that you’re building it doesn’t change anything.

I’m able to get the latest RPM as well as chromium Linux_x64 working in both AL2 and AL2018 docker containers, but get other errors when I try to do the same in lambda.