python-xmlsec: “lxml & xmlsec libxml2 library version mismatch” error under uWSGI
If libxml2-dev was installed when uWSGI was built, then importing xmlsec within uWSGI leads to an incorrect error xmlsec.InternalError: (-1, 'lxml & xmlsec libxml2 library version mismatch'). There’s no such error outside of uWSGI.
(I can work around this error with --no-binary=lxml --no-binary=xmlsec, but that wastes a lot more CI time and shouldn’t be necessary.)
Reproduction in a fresh container:
$ docker run --rm -it ubuntu:22.04
root@076333186bac:/# apt update
…
root@076333186bac:/# apt install -y gcc libxml2-dev python3-dev python3-venv
…
root@076333186bac:/# python3 -m venv venv
root@076333186bac:/# . venv/bin/activate
(venv) root@076333186bac:/# pip install uWSGI xmlsec
…
Successfully installed lxml-5.2.1 uWSGI-2.0.25.1 xmlsec-1.3.14
(venv) root@076333186bac:/# cat > app.py <<EOF
import xmlsec
def application(env, start_response):
start_response("200 OK", [("Content-Type", "text/plain")])
return [b"Hello, world!\n"]
EOF
(venv) root@076333186bac:/# python -c 'import xmlsec'
(venv) root@076333186bac:/# uwsgi --http :9090 --wsgi-file app.py
*** Starting uWSGI 2.0.25.1 (64bit) on [Mon May 6 22:18:16 2024] ***
compiled with version: 11.4.0 on 06 May 2024 22:17:53
os: Linux-6.8.7 #1-NixOS SMP PREEMPT_DYNAMIC Wed Apr 17 09:23:43 UTC 2024
nodename: 076333186bac
machine: x86_64
clock source: unix
detected number of CPU cores: 12
current working directory: /
detected binary path: /venv/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** WARNING: you are running uWSGI without its master process manager ***
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on :9090 fd 4
spawned uWSGI http 1 (pid: 4203)
uwsgi socket 0 bound to TCP address 127.0.0.1:46691 (port auto-assigned) fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x55a23eea0bc0
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 72904 bytes (71 KB) for 1 cores
*** Operational MODE: single process ***
Traceback (most recent call last):
File "app.py", line 1, in <module>
import xmlsec
xmlsec.InternalError: (-1, 'lxml & xmlsec libxml2 library version mismatch')
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. going in full dynamic mode ***
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 4202, cores: 1)
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 11
- Comments: 21
Commits related to this issue
- requirements: Drop the --no-binary for lxml and xmlsec. Building these libraries from source requires too much memory, and causes OOMs on a host with 4GB of RAM when still running Zulip. Building fr... — committed to alexmv/zulip by alexmv 3 years ago
- requirements: Drop the --no-binary for lxml and xmlsec. Building these libraries from source requires too much memory, and causes OOMs on a host with 4GB of RAM when still running Zulip. Building fr... — committed to zulip/zulip by alexmv 3 years ago
- :whale: Do not install libxml2-dev in docker image build The package is required when building lxml/xmlsec from source, but as of the latest releases, binary wheels can be installed. However, if thi... — committed to open-formulieren/open-forms by sergei-maertens 2 years ago
- Remove lxml dev packages from CI and Dockerfile Since bumping our lxml/xmlsec requirements, we can utilize the pre-built binaries and no longer have a need for the *-dev system packages. Apart from m... — committed to maykinmedia/open-inwoner by swrichards 2 years ago
- Remove lxml dev packages from CI and Dockerfile Since bumping our lxml/xmlsec requirements, we can utilize the pre-built binaries and no longer have a need for the *-dev system packages. Apart from m... — committed to maykinmedia/open-inwoner by swrichards 2 years ago
- Remove lxml dev packages from CI and Dockerfile Since bumping our lxml/xmlsec requirements, we can utilize the pre-built binaries and no longer have a need for the *-dev system packages. Apart from m... — committed to maykinmedia/open-inwoner by swrichards 2 years ago
- fix: Don't build uwsgi with xml support. When uwsgi runs with xml support, it throws the following error on startup with the latest version of edx-platform: ``` xmlsec.InternalError: (-1, 'lxml & xm... — committed to feanil/tutor by feanil 2 years ago
- fix: Don't build uwsgi with xml support. When uwsgi runs with xml support, it throws the following error on startup with the latest version of edx-platform: ``` xmlsec.InternalError: (-1, 'lxml & xm... — committed to feanil/tutor by feanil 2 years ago
- fix: Don't build uwsgi with xml support. When uwsgi runs with xml support, it throws the following error on startup with the latest version of edx-platform: ``` xmlsec.InternalError: (-1, 'lxml & xm... — committed to feanil/tutor by feanil 2 years ago
- fix: Don't build uwsgi with xml support When uwsgi runs with xml support, it throws the following error on startup with the latest version of edx-platform: ``` xmlsec.InternalError: (-1, 'lxml &... — committed to overhangio/tutor by feanil 2 years ago
- ansible: Updates from the Ubuntu 24.04 upgrade - lxml+xmlsec have fiddly dependencies - See https://github.com/xmlsec/python-xmlsec/issues/320 for more background - It appears that inst... — committed to tech-squares/squaresdb by dehnert 2 years ago
- Force lxml to 4.9 on linux / python 3.13 https://github.com/xmlsec/python-xmlsec/issues/320 — committed to offbyone/social-core by offbyone 2 years ago
- Limit python3-saml a bit https://github.com/xmlsec/python-xmlsec/issues/320 — committed to offbyone/social-core by offbyone 2 years ago
- Remove the xmlsec system libraries, so that we only use wheels https://github.com/xmlsec/python-xmlsec/issues/320 — committed to offbyone/social-core by offbyone 2 years ago
- Remove the xmlsec system libraries, so that we only use wheels https://github.com/xmlsec/python-xmlsec/issues/320 — committed to offbyone/social-core by offbyone 2 years ago
- Restrict lxml to 5.2 https://github.com/xmlsec/python-xmlsec/issues/320 — committed to offbyone/social-core by offbyone 2 years ago
- feat: Type Annotate the *heck* out of social-core (#986) * Add py.typed and pyright to the build This enables typing in social_core and gives us the tools for testing * Many type annotation upd... — committed to python-social-auth/social-core by offbyone 2 years ago
- set lxml version to avoid library version mismatch https://github.com/xmlsec/python-xmlsec/issues/320 — committed to petracihalova/turnpike by petracihalova 2 years ago
- chore: restrict "lxml" version The "xmlsec" library is currently suffering from an issue[1] that makes the builds fail due to some version mismatch with the "libxml" library. Applying some restricti... — committed to MikelAlejoBR/turnpike by MikelAlejoBR 2 years ago
- chore: restrict "lxml" version The "xmlsec" library is currently suffering from an issue[1] that makes the builds fail due to some version mismatch with the "libxml" library. Applying some restricti... — committed to MikelAlejoBR/turnpike by MikelAlejoBR 2 years ago
In my work, we use sigxml and xmlsec. Both install lxml with binaries, causing the same error.
xmlsec.InternalError: (-1, 'lxml & xmlsec libxml2 library version mismatch')To resolve this, I install using the
--no-binarypip flag. (work on Ubuntu 22.04, Python 3.10 and Pip 22.0.2)It seems like
lxml@5.2.1useslibxml2@2.12.6. However,libxml2@2.12.7has been released, andpython-xmlsec@1.3.14breaks because of this libxml2 version mismatch (?)Not sure who should be in charge of fixing this, but I suppose using libxml2@2.12.6 could work.
Encountered the “lxml & xmlsec libxml2 library version mismatch” error with following version combination (both installed using linux binary wheels from pypi.org):
Downgrading lxml resolved the issue:
@eliasmazur I have the same use case as you with python3-saml. Based on what @andersk said, I’ve restricted the package version of LXML like so
My build is working again now.
My original report explained why I don’t want to do that. It’s expensive.
Thanks @D3X - that was a really good steer. IF you don’t need XML support at all (it looks like it’s only for supporting xml configuration of uwsgi, and… who does that??), you can:
UWSGI_PROFILE_OVERRIDE="xml=no"and it’ll strip xml support altogether. From the uwsgi build system: https://github.com/unbit/uwsgi/blob/89cb161cda959697a4afe013f348b06646b960aa/uwsgiconfig.py#L1328-L1363This is the route I’ve just taken, and everything seems fine to me.
I came up with another workaround - compile
uwsgiagainstlibexpatinstead oflibxml2. You can do that by setting theUWSGI_PROFILE_OVERRIDEenvironment variable toxml=expat(or adding that part to it if you already have it set to something) for whatever command you use to installuwsgi, for example:or
Caveat:
libexpatis supposedly slower thanlibxml2(based on some old benchmarks found around the Internet), so if you’re a heavy user of uWSGI features that depend on an XML parsing library, this may affect performance of your application. I am not sure what these features are, though.Once your
uwsgiis decoupled fromlibxml2, you should be able to use binary wheels forxmlsecandlxml. You should also be able to install the older version ofxmlsecthat didn’t provide binary wheels, as long as you still havelibxmlsec1-dev/libxml2-devinstalled.I’m getting this same error on a container that I just re-compiled and it was working fine. I use python3-saml and that’s breaking with this error when I run the container.
Here is the log:
Here is my Dockerfile:
And this is the requirements.txt:
I’m stuck. Any help is appreciated.
Thanks