WeasyPrint: Python 3.8 AWS Lambda incompatibility (Amazon Linux 2)

Did anyone find a solution to make WeasyPrint work on AWS Lambda with Python 3.8?

Hi, I have open sourced our solution https://github.com/kotify/cloud-print-utils, it builds weasyprint layer for amazon linux 2, here test results of the report sample, feedback would be great.

I believe that problem in @cpmech solution is hard links used to create .so files while that is working solution in general it doesn’t work with zip files, zip doesn’t preserve links so lambda server end up with two different files for the same library and weasyprint loads .so files while the rest of the stack loads .so.N files that leads to memory problems. We fixed that by patching weasyprint https://github.com/kotify/cloud-print-utils/blob/2c64672d0f20d3b4bae85dff5cfd976eb03b5b0f/layers/weasyprint/builder.sh#L9, I’m going to create pull requests with rationale why it’s the right thing to load .so.N files instead of .so.

Another non-trivial thing is to configure pixbuf https://github.com/kotify/cloud-print-utils/blob/2c64672d0f20d3b4bae85dff5cfd976eb03b5b0f/layers/cairo-pixbuf-libffi-pango/builder.sh#L23-L30, otherwise images doesn’t work.

Hi Again, unfortunately, the code DOES SEGFAULT from time to time… So, it’s not working well. Sorry.

So, going back to Python 3.7.

Here you go the Dockefile:

Dockerfile

FROM lambci/lambda:build-python3.7

# download libraries
RUN yum install -y yum-utils rpmdevtools
WORKDIR /tmp
RUN yumdownloader libffi libffi-devel cairo pango \
    && rpmdev-extract *rpm

# install libraries and set links
RUN mkdir /opt/lib
WORKDIR /opt/lib
RUN cp -P -R /tmp/*/usr/lib64/* /opt/lib
RUN ln libpango-1.0.so.0 pango-1.0
RUN ln libpangocairo-1.0.so.0 pangocairo-1.0

# install weasyprint and dependencies
WORKDIR /opt
RUN pipenv install weasyprint
RUN mkdir -p python/lib/python3.7/site-packages
RUN pipenv lock -r > requirements.txt
RUN pip install -r requirements.txt --no-deps -t python/lib/python3.7/site-packages

# remove warning about cairo < 1.15.4
WORKDIR /opt/python/lib/python3.7/site-packages/weasyprint
RUN sed -i.bak '34,40d' document.py

# run test
WORKDIR /opt
ADD test.py .
RUN pipenv run python test.py

# package lambda layer
WORKDIR /opt
RUN zip -r weasyprint-py37.zip lib python

Sure, here you go.

requirements.txt

-i https://pypi.org/simple
cairocffi==1.1.0
cairosvg==2.4.2
cffi==1.14.0
cssselect2==0.2.2
defusedxml==0.6.0
html5lib==1.0.1
pillow==7.0.0
pycparser==2.20
pyphen==0.9.5
six==1.14.0
tinycss2==1.0.2
weasyprint==51
webencodings==0.5.1

Thanks @Tontyna and @liZe . I tried copied the default fonts from the older Amazon Linux based lambda:build-python3.7 image to the built layer. It got rid of a “No fonts configured” error but still getting the segfaults.

I have to put this away for now but I’ll try to get back to it when time permits. If anyone wants to keep digging in the mean time here’s the Dockerfile I currently have:

FROM lambci/lambda:build-python3.7 AS py37
FROM lambci/lambda:build-python3.8

# Based on https://aws.amazon.com/premiumsupport/knowledge-center/lambda-linux-binary-package/
RUN yum install -y yum-utils rpmdevtools
WORKDIR /tmp
RUN yumdownloader --resolve \
    expat \
    glib2 \
    libffi \
    libffi-devel \
    cairo \
    pango && \
    rpmdev-extract *rpm

RUN mkdir /opt/lib
WORKDIR /opt/lib
RUN cp -P -R /tmp/*/usr/lib64/* /opt/lib
RUN ln libgobject-2.0.so.0 libgobject-2.0.so && \
    ln libcairo.so.2 libcairo.so && \
    ln libpango-1.0.so.0 pango-1.0 && \
    ln libpangoft2-1.0.so.0 pangoft2-1.0 && \
    ln libpangocairo-1.0.so.0 pangocairo-1.0

WORKDIR /opt
RUN pip3 install jinja2 weasyprint -t python

COPY --from=py37 /usr/share/fonts/default /opt/fonts/default
COPY --from=py37 /etc/fonts/fonts.conf /opt/fonts/fonts.conf
RUN sed -i s:/usr/share/fonts:/opt/fonts: /opt/fonts/fonts.conf

RUN zip -r /tmp/weasyprint_lambda_layer.zip \
    ./lib \
    ./python \
    ./fonts

And I set FONTCONFIG_PATH to /opt/fonts

Hi All,

I’ve managed to make it work—thanks to Wenzil’s and previous work.

Here you go my files:

Dockerfile

FROM lambci/lambda:build-python3.7 AS py37
FROM lambci/lambda:build-python3.8

# download libraries
RUN yum install -y yum-utils rpmdevtools
WORKDIR /tmp
RUN yumdownloader --resolve \
    expat \
    glib2 \
    libffi \
    libffi-devel \
    cairo \
    pango && \
    rpmdev-extract *rpm

# install libraries and set links
RUN mkdir /opt/lib
WORKDIR /opt/lib
RUN cp -P -R /tmp/*/usr/lib64/* /opt/lib
RUN ln libgobject-2.0.so.0 libgobject-2.0.so && \
    ln libcairo.so.2 libcairo.so && \
    ln libpango-1.0.so.0 pango-1.0 && \
    ln libpangoft2-1.0.so.0 pangoft2-1.0 && \
    ln libpangocairo-1.0.so.0 pangocairo-1.0

# copy fonts and set environment variable
COPY --from=py37 /usr/share/fonts/default /opt/fonts/default
COPY --from=py37 /etc/fonts/fonts.conf /opt/fonts/fonts.conf
RUN sed -i s:/usr/share/fonts:/opt/fonts: /opt/fonts/fonts.conf
ENV FONTCONFIG_PATH="/opt/fonts"

# install weasyprint and dependencies
WORKDIR /opt
RUN pipenv install weasyprint
RUN mkdir -p python/lib/python3.8/site-packages
RUN pipenv lock -r > requirements.txt
RUN pip install -r requirements.txt --no-deps -t python/lib/python3.8/site-packages

# remove warning about cairo < 1.15.4
WORKDIR /opt/python/lib/python3.8/site-packages/weasyprint
RUN sed -i.bak '34,40d' document.py

# run test
WORKDIR /opt
ADD test.py .
RUN pipenv run python test.py

# package lambda layer
WORKDIR /opt
RUN zip -r weasyprint-py38x.zip fonts lib python

test.py

from weasyprint import html

data = """
<!DOCTYPE html>
<html>
<body>
<h1>Hello World</h1>
<p>Just Testing.</p>
</body>
</html>
"""

HTML(string=data).write_pdf("output.pdf")

Bash script to generate lambda-layer

#!/bin/bash

set -e

PYVER=py38x

docker image build -f Dockerfile-$PYVER -t weasyprint-$PYVER .
docker create -ti --name dummy-$PYVER weasyprint-$PYVER bash
docker cp dummy-$PYVER:/opt/weasyprint-$PYVER.zip .
docker cp dummy-$PYVER:/opt/output.pdf ./output-$PYVER.pdf
docker rm dummy-$PYVER

aws lambda publish-layer-version --layer-name weasyprint-$PYVER --zip-file fileb://weasyprint-$PYVER.zip

Lambda function

import json
from weasyprint import HTML

data = """
<!DOCTYPE html>
<html>
<body>
<h1>Hello World</h1>
<p>Just Testing.</p>
</body>
</html>
"""

def lambda_handler(event, context):
    HTML(string=data).write_pdf("/tmp/output.pdf")
    return {
        'statusCode': 200,
        'body': json.dumps('Success!')
    }

Output of my Lambda function

{
  "statusCode": 200,
  "body": "\"Success!\""
}

😄 😄 😄

I hope it helps! And thanks again.

Thanks @Tontyna and @liZe . I tried copied the default fonts from the older Amazon Linux based lambda:build-python3.7 image to the built layer. It got rid of a “No fonts configured” error but still getting the segfaults.

I have to put this away for now but I’ll try to get back to it when time permits. If anyone wants to keep digging in the mean time here’s the Dockerfile I currently have:

FROM lambci/lambda:build-python3.7 AS py37
FROM lambci/lambda:build-python3.8

# Based on https://aws.amazon.com/premiumsupport/knowledge-center/lambda-linux-binary-package/
RUN yum install -y yum-utils rpmdevtools
WORKDIR /tmp
RUN yumdownloader --resolve \
    expat \
    glib2 \
    libffi \
    libffi-devel \
    cairo \
    pango && \
    rpmdev-extract *rpm

RUN mkdir /opt/lib
WORKDIR /opt/lib
RUN cp -P -R /tmp/*/usr/lib64/* /opt/lib
RUN ln libgobject-2.0.so.0 libgobject-2.0.so && \
    ln libcairo.so.2 libcairo.so && \
    ln libpango-1.0.so.0 pango-1.0 && \
    ln libpangoft2-1.0.so.0 pangoft2-1.0 && \
    ln libpangocairo-1.0.so.0 pangocairo-1.0

WORKDIR /opt
RUN pip3 install jinja2 weasyprint -t python

COPY --from=py37 /usr/share/fonts/default /opt/fonts/default
COPY --from=py37 /etc/fonts/fonts.conf /opt/fonts/fonts.conf
RUN sed -i s:/usr/share/fonts:/opt/fonts: /opt/fonts/fonts.conf

RUN zip -r /tmp/weasyprint_lambda_layer.zip \
    ./lib \
    ./python \
    ./fonts

And I set FONTCONFIG_PATH to /opt/fonts