meta-updater: Python execution very slow

Python scripts/programs appear to be very slow. Already running the Python interpreter is slower than usual. Using python3 -v shows that Python detects the bytecode files (pyc) as stale:

$ python3 -v
import _frozen_importlib # frozen
import _imp # builtin
import sys # builtin
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'posix' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
# bytecode is stale for 'encodings'
# code object from /usr/lib/python3.5/encodings/__init__.py
# could not create '/usr/lib/python3.5/encodings/__pycache__/__init__.cpython-35.pyc': OSError(30, 'Read-only file system')
# wrote '/usr/lib/python3.5/encodings/__pycache__/__init__.cpython-35.pyc'
# bytecode is stale for 'codecs'
# code object from /usr/lib/python3.5/codecs.py
# could not create '/usr/lib/python3.5/__pycache__/codecs.cpython-35.pyc': OSError(30, 'Read-only file system')
# wrote '/usr/lib/python3.5/__pycache__/codecs.cpython-35.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0xb6b5c130>
# bytecode is stale for 'encodings.aliases'
# code object from /usr/lib/python3.5/encodings/aliases.py
# could not create '/usr/lib/python3.5/encodings/__pycache__/aliases.cpython-35.pyc': OSError(30, 'Read-only file system')
# wrote '/usr/lib/python3.5/encodings/__pycache__/aliases.cpython-35.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0xb6b669d0>
...

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 15 (14 by maintainers)

Most upvoted comments

At this point we might as well just use reproducible builds for the whole rootfs. There are probably even more advantages: E.g. currently, when clearing tmp/sstate and rebuilding, the resulting OSTree contains a lot of differences to the earlier version, just due to timestamp etc. So for OSTree, using reproducible builds also helps keeping the deltas small even when not reusing sstate…

I like the sound of that! What are the drawbacks to that approach? I haven’t read up on the topic in a while.

pattivacek on Jan 29, 2019

My first try was adding a recipes-devtools/python/python3_%.bbappend which directly sets SOURCE_DATE_EPOCH:

export SOURCE_DATE_EPOCH ?= "0"

This works for the core libraries. However, all libraries (e.g. python3-urllib3) still suffer from the issue. For my test program docker-compose, this cuts down stale bytecode warnings from 459 to 326 files. And execution time from 12.5s to 9.5s. With all files fixed (using the above script), I get an execution time of 4.5s…

I then tired setting export SOURCE_DATE_EPOCH ?= "0" in openembedded-core/meta/classes/python3-dir.bbclass. This uncovered another interesting issue: The recipe python3-vcversioner builds an Python egg file, which uses zip. Zip seems to have a restriction on mtime it supports:

|   File "/home/ags/torizoncore/build-colibri-imx7/tmp-torizon/work/x86_64-linux/python3-vcversio
ner-native/2.16.0.0-r0/recipe-sysroot-native/usr/lib/python3.5/zipfile.py", line 338, in __init__
|     raise ValueError('ZIP does not support timestamps before 1980')
| ValueError: ZIP does not support timestamps before 1980

I am not sure why python3-vcversioner needs/is building this egg file. Removing the do_compile_append and do_install_append from the recipe seems to work fine and builds the rootfs successfully. I am not sure how to solve this properly. I think it would be good if OSTree has a way to define the mtime it is using, so we could use a timestamp somewhat more recent…

With that, I have 0 bytecode warnings, and docker-compose starts in 4.2s.

However, this is not ideal since we cannot easily alter python3-dir.bbclass from meta-updater. If anybody has an idea how to hook into all Python recipe from meta-updater I would be interested to hear.

At this point we might as well just use reproducible builds for the whole rootfs. There are probably even more advantages: E.g. currently, when clearing tmp/sstate and rebuilding, the resulting OSTree contains a lot of differences to the earlier version, just due to timestamp etc. So for OSTree, using reproducible builds also helps keeping the deltas small even when not reusing sstate…

agners on Jan 29, 2019

Another option would be to just not deploy the py source files. This seems a valid Python3 deployment method and saves space too…

That would be nice to see as well.

ricardosalveti on Jan 28, 2019