python-xmlsec: “lxml & xmlsec libxml2 library version mismatch” error under uWSGI

If libxml2-dev was installed when uWSGI was built, then importing xmlsec within uWSGI leads to an incorrect error xmlsec.InternalError: (-1, 'lxml & xmlsec libxml2 library version mismatch'). There’s no such error outside of uWSGI.

(I can work around this error with --no-binary=lxml --no-binary=xmlsec, but that wastes a lot more CI time and shouldn’t be necessary.)

Reproduction in a fresh container:

$ docker run --rm -it ubuntu:22.04
root@076333186bac:/# apt update
…
root@076333186bac:/# apt install -y gcc libxml2-dev python3-dev python3-venv
…
root@076333186bac:/# python3 -m venv venv
root@076333186bac:/# . venv/bin/activate
(venv) root@076333186bac:/# pip install uWSGI xmlsec
…
Successfully installed lxml-5.2.1 uWSGI-2.0.25.1 xmlsec-1.3.14
(venv) root@076333186bac:/# cat > app.py <<EOF
import xmlsec

def application(env, start_response):
    start_response("200 OK", [("Content-Type", "text/plain")])
    return [b"Hello, world!\n"]

EOF
(venv) root@076333186bac:/# python -c 'import xmlsec'
(venv) root@076333186bac:/# uwsgi --http :9090 --wsgi-file app.py
*** Starting uWSGI 2.0.25.1 (64bit) on [Mon May  6 22:18:16 2024] ***
compiled with version: 11.4.0 on 06 May 2024 22:17:53
os: Linux-6.8.7 #1-NixOS SMP PREEMPT_DYNAMIC Wed Apr 17 09:23:43 UTC 2024
nodename: 076333186bac
machine: x86_64
clock source: unix
detected number of CPU cores: 12
current working directory: /
detected binary path: /venv/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
*** WARNING: you are running uWSGI without its master process manager ***
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on :9090 fd 4
spawned uWSGI http 1 (pid: 4203)
uwsgi socket 0 bound to TCP address 127.0.0.1:46691 (port auto-assigned) fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x55a23eea0bc0
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 72904 bytes (71 KB) for 1 cores
*** Operational MODE: single process ***
Traceback (most recent call last):
  File "app.py", line 1, in <module>
    import xmlsec
xmlsec.InternalError: (-1, 'lxml & xmlsec libxml2 library version mismatch')
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. going in full dynamic mode ***
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 4202, cores: 1)

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 11
  • Comments: 21

Commits related to this issue

Most upvoted comments

In my work, we use sigxml and xmlsec. Both install lxml with binaries, causing the same error.

xmlsec.InternalError: (-1, 'lxml & xmlsec libxml2 library version mismatch')

To resolve this, I install using the --no-binary pip flag. (work on Ubuntu 22.04, Python 3.10 and Pip 22.0.2)

pip install --no-binary lxml==4.6.3 lxml==4.6.3 --force-reinstall
pip install --no-binary xmlsec==1.3.13 xmlsec==1.3.13
pip install --no-binary signxml==3.2.2 signxml==3.2.2

It seems like lxml@5.2.1 uses libxml2@2.12.6. However, libxml2@2.12.7 has been released, and python-xmlsec@1.3.14 breaks because of this libxml2 version mismatch (?)

Not sure who should be in charge of fixing this, but I suppose using libxml2@2.12.6 could work.

Encountered the “lxml & xmlsec libxml2 library version mismatch” error with following version combination (both installed using linux binary wheels from pypi.org):

lxml==5.4.0
xmlsec==1.3.14

Downgrading lxml resolved the issue:

lxml==5.3.2
xmlsec==1.3.14

@eliasmazur I have the same use case as you with python3-saml. Based on what @andersk said, I’ve restricted the package version of LXML like so

lxml >= 4.6.5, !=4.7.0, <=5.2.1 

My build is working again now.

My original report explained why I don’t want to do that. It’s expensive.

(I can work around this error with --no-binary=lxml --no-binary=xmlsec, but that wastes a lot more CI time and shouldn’t be necessary.)

Thanks @D3X - that was a really good steer. IF you don’t need XML support at all (it looks like it’s only for supporting xml configuration of uwsgi, and… who does that??), you can:

UWSGI_PROFILE_OVERRIDE="xml=no" and it’ll strip xml support altogether. From the uwsgi build system: https://github.com/unbit/uwsgi/blob/89cb161cda959697a4afe013f348b06646b960aa/uwsgiconfig.py#L1328-L1363

This is the route I’ve just taken, and everything seems fine to me.

I came up with another workaround - compile uwsgi against libexpat instead of libxml2. You can do that by setting the UWSGI_PROFILE_OVERRIDE environment variable to xml=expat (or adding that part to it if you already have it set to something) for whatever command you use to install uwsgi, for example:

UWSGI_PROFILE_OVERRIDE=xml=expat pip install uwsgi

or

UWSGI_PROFILE_OVERRIDE="ssl=false;xml=expat" pip install uwsgi

Caveat: libexpat is supposedly slower than libxml2 (based on some old benchmarks found around the Internet), so if you’re a heavy user of uWSGI features that depend on an XML parsing library, this may affect performance of your application. I am not sure what these features are, though.

Once your uwsgi is decoupled from libxml2, you should be able to use binary wheels for xmlsec and lxml. You should also be able to install the older version of xmlsec that didn’t provide binary wheels, as long as you still have libxmlsec1-dev / libxml2-dev installed.

I’m getting this same error on a container that I just re-compiled and it was working fine. I use python3-saml and that’s breaking with this error when I run the container.

Here is the log:

[2024-05-20 02:54:20 +0000] [11] [ERROR] Exception in worker process
2024-05-19 22:54:20 Traceback (most recent call last):
2024-05-19 22:54:20   File "/usr/local/lib/python3.6/dist-packages/gunicorn/arbiter.py", line 583, in spawn_worker
2024-05-19 22:54:20     worker.init_process()
2024-05-19 22:54:20   File "/usr/local/lib/python3.6/dist-packages/gunicorn/workers/ggevent.py", line 203, in init_process
2024-05-19 22:54:20     super(GeventWorker, self).init_process()
2024-05-19 22:54:20   File "/usr/local/lib/python3.6/dist-packages/gunicorn/workers/base.py", line 129, in init_process
2024-05-19 22:54:20     self.load_wsgi()
2024-05-19 22:54:20   File "/usr/local/lib/python3.6/dist-packages/gunicorn/workers/base.py", line 138, in load_wsgi
2024-05-19 22:54:20     self.wsgi = self.app.wsgi()
2024-05-19 22:54:20   File "/usr/local/lib/python3.6/dist-packages/gunicorn/app/base.py", line 67, in wsgi
2024-05-19 22:54:20     self.callable = self.load()
2024-05-19 22:54:20   File "/usr/local/lib/python3.6/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load
2024-05-19 22:54:20     return self.load_wsgiapp()
2024-05-19 22:54:20   File "/usr/local/lib/python3.6/dist-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
2024-05-19 22:54:20     return util.import_app(self.app_uri)
2024-05-19 22:54:20   File "/usr/local/lib/python3.6/dist-packages/gunicorn/util.py", line 350, in import_app
2024-05-19 22:54:20     __import__(module)
2024-05-19 22:54:20   File "/opt/app/tinyflask/__init__.py", line 11, in <module>
2024-05-19 22:54:20     import tinyflask.views
2024-05-19 22:54:20   File "/opt/app/tinyflask/views.py", line 24, in <module>
2024-05-19 22:54:20     from onelogin.saml2.auth import OneLogin_Saml2_Auth
2024-05-19 22:54:20   File "/usr/local/lib/python3.6/dist-packages/onelogin/saml2/auth.py", line 12, in <module>
2024-05-19 22:54:20     import xmlsec
2024-05-19 22:54:20 xmlsec.InternalError: (-1, 'lxml & xmlsec libxml2 library version mismatch')
2024-05-19 22:54:20 [2024-05-20 02:54:20 +0000] [11] [INFO] Worker exiting (pid: 11)

Here is my Dockerfile:

FROM ubuntu:bionic

RUN apt update
RUN apt-get update
RUN apt-get -y upgrade
RUN apt-get install -y pkg-config
RUN apt-get install -y software-properties-common apt-utils
RUN add-apt-repository -y ppa:deadsnakes/ppa
RUN apt-get install -y vim
RUN apt-get install -y python3.7
RUN apt-get install -y python3-pip

RUN apt-get install -y libxml2-dev libxmlsec1-dev

RUN mkdir -p /opt/app
WORKDIR /opt/app/
COPY . .
RUN pip3 install --upgrade pip
RUN pip3 install --no-cache-dir -r requirements.txt

RUN pip3 install python3-saml

RUN touch /var/log/cron.log

RUN echo "35 6 * * * root /usr/bin/python3 /opt/app/last_login.py >> /opt/app/audit.log" >> /etc/crontab

CMD (cron) && exec gunicorn tinyflask:app -b 0.0.0.0:80 --workers 3 -k gevent

And this is the requirements.txt:

Click==7.0
Flask==1.1.1
Flask-Cors==3.0.8
gevent==1.4.0
greenlet==0.4.15
gunicorn==19.9.0
itsdangerous==1.1.0
Jinja2==2.10.1
MarkupSafe==1.1.1
Werkzeug==0.15.5
flask-compress==1.4.0
requests==2.24.0
python-dateutil==2.8.0
pytz==2021.1
pickleshare==0.7.5
xmltodict

I’m stuck. Any help is appreciated.

Thanks