picamera2: [BUG] Cannot be used with non-system python

@adamgogacz Raspberry Pi OS follows Debian long term (currently Debian 11). Until the next version of Debian Stable is taken up by Raspberry Pi OS do not expect the picamera2 packages being based on a Python version later than 3.9. FYI Debian 12 (Bookworm) uses python 3.11.

Shouldn’t you be able to build your libcamera against python 3.9, 3.10, 3.11 and 3.12 and upload all of them to pypi? This is what other python package providers are doing as well, if they need a python-version specific code of the library. Otherwise every python project that uses the camera depends on the system python.

Well, in about a week from today the Picamera2 package will be 3 Python versions behind Python3.12 release schedule. These cameras are not ready for the Python community, who by and large operate in virtual environments other than 3.9. Why complain? I now own ~$250 worth of dust collection surface.

In a virtual env (I’m using poetry like #466 but it’s the same for a simple venv), the sys.path contains /usr/lib/python3.9, so we can copy the libcamera module and its dependencies there and it will be available in all virtual envs.

sudo cp -r /usr/lib/python3/dist-packages/libcamera /usr/lib/python3.9
sudo cp -r /usr/lib/python3/dist-packages/pykms /usr/lib/python3.9

I got a bunch of errors such as:

libcblas.so.3: cannot open shared object file: No such file or directory
libopenjp2.so.7: cannot open shared object file: No such file or directory

which disappeared after installing the relevant libs.

sudo apt install libatlas-base-dev libopenjp2-7-dev

It’s a bit insane to have to do that and I wish I understood why this package breaks virtual envs - doesn’t everyone use some sort of venv nowadays for reproducibility of deployments? That’s a big deal!

But it honestly goes over my head and since I can’t fix it I can’t complain 😃

Thanks for the solution proposal. Although its very hacky I will give it a go and test it out with pi camera 3. Does you venv use newer python version like 3.11 or sticks to the system 3.9? BTW I think since Raspberry Pi Foundation sells the cameras you actually can complain when there isn’t any proper solution for a very standard use case.

Has there been any progress on this or alternate solutions? Has anyone been able to upgrade to Python 3.10+ on a Raspberry Pi and still use Picamera2/Libcamera?

–system-site-packages does not work for me, libcamera is still not showing up in the virtual environment.

Out of interest, the python libcamera bindings are normally found here: /usr/lib/python3/dist-packages/libcamera. If you rename the .so file in there to _libcamera.so, are you able to import that successfully using Python 3.10?

I installed fresh Pi OS 64-bit along with Python3.10.9 built from source with --enable-optimizations --with-lto flags, renamed the mentioned file, rebooted, and just as suspected nothing happened. Why can’t you guys package the libcamera with python bindings and post it to pypi? If you’re having problems with that @davisking did a great job handling dlib bindings with pybind11 maybe he can lend a hand.

I haven’t installed Bookworm yet, but if I do, are you saying this isn’t currently working/updated in Raspbian?

Wasn’t 100% sure what you meant. There are no pip packages for libcamera and kms++ currently. Camera applications and Picamera2 should work on Bookworm, though there are some fairly significant updates pending which will add/improve functionality.

@JooJooBee666 Wait till the end of October. When RPi 5 will be available also Raspberry Pi OS will go to bookworm and Python 3.11. I am assuming libcamera Python bindings will also be based on Python 3.11 then for aarch64. Maybe @davidplowman can confirm this.

@adamgogacz Raspberry Pi OS follows Debian long term (currently Debian 11). Until the next version of Debian Stable is taken up by Raspberry Pi OS do not expect the picamera2 packages being based on a Python version later than 3.9. FYI Debian 12 (Bookworm) uses python 3.11.

In a virtual env (I’m using poetry like #466 but it’s the same for a simple venv), the sys.path contains /usr/lib/python3.9, so we can copy the libcamera module and its dependencies there and it will be available in all virtual envs using python 3.9. It won’t work if the venv is using another python version.

sudo cp -r /usr/lib/python3/dist-packages/libcamera /usr/lib/python3.9
sudo cp -r /usr/lib/python3/dist-packages/pykms /usr/lib/python3.9

I got a bunch of errors such as:

libcblas.so.3: cannot open shared object file: No such file or directory
libopenjp2.so.7: cannot open shared object file: No such file or directory

which disappeared after installing the relevant libs.

sudo apt install libatlas-base-dev libopenjp2-7-dev

It’s a bit insane to have to do that and I wish I understood why this package breaks virtual envs - doesn’t everyone use some sort of venv nowadays for reproducibility of deployments? That’s a big deal!

But it honestly goes over my head and since I can’t fix it I can’t complain 😃

Hi, thanks for the question. You probably need your virtual environment to copy the system packages because not all are available via pip, for example:

python -m venv --system-site-packages my-env

This won’t work since the --system-site-packages adds the packages from the python version that is used inside virtualenv This solved #341 since venv package creates only isolated packages and still uses system python virtualenv on the other hand creates an isolated packages and python environment.

I’ve confirmed that it won’t work on a clean Pi OS 64-bit with source-compiled python 3.10

BUT to actually check if you can use picamera2 with python different that which the system provides I’ve done a simple test

1. installed clean Pi OS 64-bit
2. updated & upgraded
3. upgraded pip for system python
4. installed python 3.10.8 from source
5. upgraded pip for installed python
6. run a sanity check on system python without virtualenv

Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import picamera2
>>> picamera2.__file__
'/usr/lib/python3/dist-packages/picamera2/__init__.py'
>>> import libcamera
>>> libcamera.__file__
'/usr/lib/python3/dist-packages/libcamera/__init__.py'

1. created and virtualenv for system python with said flag

pip install virtualenv
virtualenv venv --system-site-packages 
source venv/bin/activate
python

1. checked if the proposed works for system python (it works)

Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import picamera2
>>> picamera2.__file__
'/usr/lib/python3/dist-packages/picamera2/__init__.py'
>>> import libcamera
>>> libcamera.__file__
'/usr/lib/python3/dist-packages/libcamera/__init__.py'

1. deactivated the virtualenv
2. created a virtualenv for python3.10

virtualenv venv10 --system-site-packages --python /usr/local/bin/python3.10
source venv10/bin/activate
python

1. checked for libcamera

Python 3.10.8 (main, Nov 26 2022, 04:52:14) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libcamera
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'libcamera'

1. checked for the same without virtualenv (same error)

SO No matter what, the libcamera would have to be prepackaged and build duing picamera2 build. Otherwise the users are forced to install libcamera on their own from source which I personally find toubling as when buying Raspberry Pi camera v2 I was hoping for a more native and easier way to get it up and running on newer python versions. Now the only hope is the python-opencv

Please consider chaning the default installation to pip as it supposed to be in the python community. Im happy to help and run some more tests if you’d like. Im desperate to get it running on picamera2 and at least python3.10