python-for-android: Comprehensive list of broken python3 recipes
Just like it was done before for python2 and python3crystax, we need a list of broken python3 recipes https://github.com/kivy/python-for-android/issues/1402
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 33 (28 by maintainers)
Commits related to this issue
- Update of Recipes for python3 test (#1622) Update of Recipes for test of broken recipes because of invalid dependency graphs, refs #1514 — committed to kivy/python-for-android by strubbi77 5 years ago
- Update broken recipes list, refs #1514 Also bumps few versions. — committed to AndreMiras/python-for-android by AndreMiras 5 years ago
- Update broken recipes list, refs #1514 Removed gevent, cffi and pycryptodome recipes from it since they now compile fine. Also bumped few versions. — committed to AndreMiras/python-for-android by AndreMiras 5 years ago
- Update broken recipes list, refs #1514 Removed gevent, cffi and pycryptodome recipes from it since they now compile fine. Also bumped few versions. — committed to AndreMiras/python-for-android by AndreMiras 5 years ago
- Updates broken recipes list, refs #1514 — committed to AndreMiras/python-for-android by AndreMiras 5 years ago
- Merge pull request #1658 from AndreMiras/feature/feature/ticket1514_update_broken_recipes_list_bis Updates broken recipes list, refs #1514 — committed to kivy/python-for-android by AndreMiras 5 years ago
- Further recipe changes for python3 #1514 — committed to strubbi77/python-for-android by strubbi77 5 years ago
- Origin (#1681) * update to add dependency on python3 to opencv recipe numpy/opencv fails with latest p4a at runtime (import) #1678 * Further recipe changes for python3 #1514 * Update __init__.... — committed to kivy/python-for-android by strubbi77 5 years ago
A little more detail…
Build environment:
sudo apt-get install -y autoconf libtool cmake libffi-dev pkg-config libjpeg9-dev python3-pilUsing python-for-android - master, installed 1/19/2019.p4a
Build command:
p4a apk --requirements=<module>,python3,kivyResulting module builds can be divided into four groups:These respectively contain:
Pillow : site-packages/PIL/_imaging.so site-packages/PIL/_imagingft.so site-packages/PIL/_imagingmath.so site-packages/PIL/_imagingmorph.so audiostream : site-packages/audiostream/core.so site-packages/audiostream/platform/plat_android.so site-packages/audiostream/sources/puredata.so site-packages/audiostream/sources/thread.so site-packages/audiostream/sources/wave.so cffi : modules/_ctypes.cpython-37m.so site-packages/_cffi_backend.so cryptography : modules/_ctypes.cpython-37m.so modules/_hashlib.cpython-37m.so modules/_ssl.cpython-37m.so site-packages/cryptography/hazmat/bindings/_constant_time.so site-packages/cryptography/hazmat/bindings/_openssl.so site-packages/cryptography/hazmat/bindings/_padding.so site-packages/_cffi_backend.so libcurl : modules/_hashlib.cpython-37m.so modules/_ssl.cpython-37m.so libffi : modules/_ctypes.cpython-37m.so lxml : site-packages/lxml/builder.so site-packages/lxml/etree.so site-packages/lxml/html/clean.so site-packages/lxml/html/diff.so site-packages/lxml/objectify.so site-packages/lxml/_elementpath.so netifaces : site-packages/netifaces.so
numpy : site-packages/numpy/core/multiarray.so site-packages/numpy/core/umath.so site-packages/numpy/core/_dummy.so site-packages/numpy/core/_multiarray_tests.so site-packages/numpy/core/_operand_flag_tests.so site-packages/numpy/core/_rational_tests.so site-packages/numpy/core/_struct_ufunc_tests.so site-packages/numpy/core/_umath_tests.so site-packages/numpy/fft/fftpack_lite.so site-packages/numpy/linalg/lapack_lite.so site-packages/numpy/linalg/_umath_linalg.so site-packages/numpy/random/mtrand.so openssl : modules/_hashlib.cpython-37m.so modules/_ssl.cpython-37m.so pycrypto : modules/_hashlib.cpython-37m.so modules/_ssl.cpython-37m.so site-packages/Crypto/Cipher/_AES.so site-packages/Crypto/Cipher/_ARC2.so site-packages/Crypto/Cipher/_ARC4.so site-packages/Crypto/Cipher/_Blowfish.so site-packages/Crypto/Cipher/_CAST.so site-packages/Crypto/Cipher/_DES.so site-packages/Crypto/Cipher/_DES3.so site-packages/Crypto/Cipher/_XOR.so site-packages/Crypto/Hash/MD2.so site-packages/Crypto/Hash/MD4.so site-packages/Crypto/Hash/RIPEMD160.so site-packages/Crypto/Hash/SHA224.so site-packages/Crypto/Hash/SHA256.so site-packages/Crypto/Hash/SHA384.so site-packages/Crypto/Hash/SHA512.so site-packages/Crypto/Util/cpuid.so site-packages/Crypto/Util/galois.so site-packages/Crypto/Util/strxor.so site-packages/Crypto/Util/_counter.so site-packages/netifaces.so pycryptodome : modules/_ctypes.cpython-37m.so site-packages/Crypto/Cipher/_ARC4.so site-packages/Crypto/Cipher/_chacha20.so site-packages/Crypto/Cipher/_raw_aes.so site-packages/Crypto/Cipher/_raw_arc2.so site-packages/Crypto/Cipher/_raw_blowfish.so site-packages/Crypto/Cipher/_raw_cast.so site-packages/Crypto/Cipher/_raw_cbc.so site-packages/Crypto/Cipher/_raw_cfb.so site-packages/Crypto/Cipher/_raw_ctr.so site-packages/Crypto/Cipher/_raw_des.so site-packages/Crypto/Cipher/_raw_des3.so site-packages/Crypto/Cipher/_raw_ecb.so site-packages/Crypto/Cipher/_raw_ocb.so site-packages/Crypto/Cipher/_raw_ofb.so site-packages/Crypto/Cipher/_Salsa20.so site-packages/Crypto/Hash/_BLAKE2b.so site-packages/Crypto/Hash/_BLAKE2s.so site-packages/Crypto/Hash/_keccak.so site-packages/Crypto/Hash/_MD2.so site-packages/Crypto/Hash/_MD4.so site-packages/Crypto/Hash/_RIPEMD160.so site-packages/Crypto/Hash/_SHA224.so site-packages/Crypto/Hash/_SHA256.so site-packages/Crypto/Hash/_SHA384.so site-packages/Crypto/Hash/_SHA512.so site-packages/Crypto/Protocol/_scrypt.so site-packages/Crypto/Util/_cpuid.so site-packages/Crypto/Util/_galois.so site-packages/Crypto/Util/_strxor.so site-packages/_cffi_backend.so pymunk : modules/_ctypes.cpython-37m.so site-packages/pymunk/libchipmunk.so site-packages/_cffi_backend.so reportlab : site-packages/numpy/core/multiarray.so site-packages/numpy/core/umath.so site-packages/numpy/core/_dummy.so site-packages/numpy/core/_multiarray_tests.so site-packages/numpy/core/_operand_flag_tests.so site-packages/numpy/core/_rational_tests.so site-packages/numpy/core/_struct_ufunc_tests.so site-packages/numpy/core/_umath_tests.so site-packages/numpy/fft/fftpack_lite.so site-packages/numpy/linalg/lapack_lite.so site-packages/numpy/linalg/_umath_linalg.so site-packages/numpy/random/mtrand.so site-packages/reportlab/graphics/_renderPM.so site-packages/reportlab/lib/_rl_accel.so
spine : site-packages/spine/animation/animation.so site-packages/spine/animation/animationstate.so site-packages/spine/animation/attachmenttimeline.so site-packages/spine/animation/colortimeline.so site-packages/spine/animation/curvetimeline.so site-packages/spine/animation/drawordertimeline.so site-packages/spine/animation/eventtimeline.so site-packages/spine/animation/ffdtimeline.so site-packages/spine/animation/flipxtimeline.so site-packages/spine/animation/flipytimeline.so site-packages/spine/animation/ikconstrainttimeline.so site-packages/spine/animation/rotatetimeline.so site-packages/spine/animation/scaletimeline.so site-packages/spine/animation/timeline.so site-packages/spine/animation/trackentry.so site-packages/spine/animation/translatetimeline.so site-packages/spine/attachment/attachment.so site-packages/spine/attachment/boundingboxattachment.so site-packages/spine/attachment/meshattachment.so site-packages/spine/attachment/regionattachment.so site-packages/spine/attachment/skinnedmeshattachment.so site-packages/spine/bone.so site-packages/spine/ikconstraint.so site-packages/spine/skeleton/skeleton.so site-packages/spine/skeleton/skeletonbounds.so site-packages/spine/skeleton/skeletondata.so site-packages/spine/slot.so site-packages/spine/spineevent.so site-packages/spine/utils.so sqlite3 : modules/_sqlite3.cpython-37m.so
['enum34', 'ffmpeg', 'ffpyplayer_codecs', 'freetype', 'harfbuzz', 'idna', 'ipaddress', 'jedi', 'jpeg', 'leveldb', 'libexpat', 'libiconv', 'libogg', 'libsecp256k1', 'libshine', 'libx264', 'libxml2', 'libxslt', 'openal', 'pbkdf2', 'png', 'pycparser', 'pyjnius', 'pysdl2', 'requests', 'sdl2_image', 'sdl2_mixer', 'sdl2_ttf', 'six', 'snappy','zeroconf']Lots of reasons here, not clear to me I can correctly diagnose, but I see things like: enum34 bypasses build on python3 (this might be OK in this case?)) libshine builds .a but it is not copied for apk libxml2 builds .al but not .a, then tries to copy .a for apk
Build Log Files:
log_no_object.zip log_no_apk.zip log_has_object.zip
Getting better. Results of my attempt to build the recipes follow. If anybody knows how to build any of the failed recipes please let me know.
Recipes looking good:
['Pillow', 'android', 'apsw', 'atom', 'audiostream', 'babel', 'boost', 'cffi', 'coverage', 'cryptography', 'cymunk', 'dateutil', 'decorator', 'doubleratchet', 'enum34', 'ffmpeg', 'ffpyplayer_codecs', 'freetype', 'gevent', 'gevent-websocket', 'greenlet', 'harfbuzz', 'idna', 'ifaddrs', 'ipaddress', 'jedi', 'jpeg', 'leveldb', 'libcurl', 'libexpat', 'libffi', 'libiconv', 'libnacl', 'libogg', 'libpq', 'libsecp256k1', 'libshine', 'libsodium', 'libtorrent', 'libx264', 'libxml2', 'libxslt', 'lxml', 'msgpack-python', 'netifaces', 'numpy', 'openal', 'opencv', 'openssl', 'pbkdf2', 'png', 'preppy', 'protobuf_cpp', 'pyaml', 'pyasn1', 'pycparser', 'pycrypto', 'pyjnius', 'pymunk', 'pynacl', 'pyogg', 'pyopenal', 'pyopenssl', 'pysdl2', 'pysha3', 'pytz', 'pyusb', 'pyyaml', 'reportlab', 'requests', 'scrypt', 'sdl2', 'sdl2_image', 'sdl2_mixer', 'sdl2_ttf', 'secp256k1', 'simple-crypt', 'six', 'snappy', 'spine', 'sqlite3', 'vispy', 'websocket-client', 'wsaccel', 'zeroconf']Recipes with no APK created:
['cdecimal', 'cherrypy', 'enaml', 'ethash', 'evdev', 'feedparser', 'ffpyplayer', 'flask', 'fontconfig', 'groestlcoin_hash', 'icu', 'kivent_core', 'kivent_cymunk', 'kivent_particles', 'kivent_polygen', 'kivykiwisolver', 'libgeos', 'libglob', 'libmysqlclient', 'libtribler', 'libvorbis', 'libzbar', 'libzmq', 'm2crypto', 'mysqldb', 'omemo', 'omemo-backend-signal', 'pil', 'psycopg2', 'pycryptodome', 'pyethereum', 'pyicu', 'pyleveldb', 'pyproj', 'pyrxp', 'pyzmq', 'regex', 'ruamel.yaml', 'shapely', 'sqlalchemy', 'storm', 'sympy', 'twisted', 'ujson', 'vlc', 'x3dh', 'xeddsa', 'zbar', 'zope', 'zope_interface']Test .p4a
Test depends
sudo apt-get install -y autoconf libtool cmake libffi-dev pkg-config libjpeg9-dev python3-pilTest log from failed builds:
Errors like
sqlalchemy : Build failed: Didn’t find any valid dependency graphs
are easy to overcome, the dependencies in every file under recipes/packagename/_init.py
depends = [(‘python2’, ‘python3crystax’), ‘setuptools’]
have to be changed to: depends = [‘setuptools’]
Then probably other errors will occur
Next I looked in the .apk for .so files in addition to the default set of .so I found that most builds silently create an incomplete apk in my setup (the NONE items below). Silently, is to my mind an issue! Just starting to dig through why the apks are missing extra .so
EDIT: some are in ‘site-packages’ directory, I was not looking there.
Pillow NONE audiostream NONE cffi _ctypes.cpython-37m.so cryptography _ctypes.cpython-37m.so _hashlib.cpython-37m.so _ssl.cpython-37m.so enum34 NONE ffmpeg NONE ffpyplayer_codecs NONE freetype NONE harfbuzz NONE idna NONE ipaddress NONE jedi NONE jpeg NONE leveldb NONE libcurl _hashlib.cpython-37m.so _ssl.cpython-37m.so libexpat NONE libffi _ctypes.cpython-37m.so libiconv NONE libogg NONE libsecp256k1 NONE libshine NONE libvorbis FAIL libx264 NONE libxml2 NONE libxslt NONE lxml NONE numpy NONE openal NONE openssl _hashlib.cpython-37m.so _ssl.cpython-37m.so pbkdf2 NONE png NONE pycparser NONE pycryptodome _ctypes.cpython-37m.so pyjnius FAIL pysdl2 NONE requests NONE sdl2_image NONE sdl2_mixer NONE sdl2_ttf NONE snappy NONE spine NONE sqlite3 _sqlite3.cpython-37m.so
[Previously posted in #1576 , suggested more appropriate here]
I tried to build all the user modules for python3, kivy. Of course a successful apk build does not necessarily imply the apk contents are correct!
The following dependencies are required in addition to those for a simple .py build (for which I use https://groups.google.com/forum/#!topic/kivy-users/Lf4zlYmLVPo )
sudo apt-get install -y autoconf libtool cmake libffi-dev pkg-configI use python-for-android (master, installed 1/12/2019) I tested each with a placeholder main.py and:
p4a apk --requirements=<module>,python3,kivy --arch=armeabi-v7a --ndk_version 17c --android_api 28 --ndk-api 21The following modules successfully build an apk:
BUT, of these, the following fail with error “… _ctypes…”" if module order is changed --requirements=python3,kivy,<module>
Which confirms that order can be important.
The following do not build because they have no dependency on ‘python3’, but do have a dependency on some combination of ‘python2’/‘hostpython2’/ ‘python3crystax’/‘hostpython3’
The following does not build because of a dependency I can’t identify:
['flask']The following do not build because they have a dependency on ‘kivy’ !['kivent_core','kivent_cymunk','kivent_particles','kivent_polygen']The following does not build because: ‘p4a.cmake’: No such file or directory
['libmysqlclient']The following does not build because of cmake errors
['opencv']The following does not build because (seems strange): crypt.h: No such file or directory
['reportlab']The following does not build because has dependency on ‘incremental’, ‘constantly’
['twisted']The following does not build because NDK v18 needed
['vlc']sqlalchemy is broken with python 3.7.1
cwd is /home/strubbi77/.local/share/python-for-android/build/other_builds/sqlalchemy-python3-python3/armeabi-v7a__ndk_target_21/sqlalchemy
Traceback (most recent call last): File “/usr/local/bin/p4a”, line 10, in <module> sys.exit(main()) File “/usr/local/lib/python3.7/dist-packages/pythonforandroid/toolchain.py”, line 1033, in main ToolchainCL() File “/usr/local/lib/python3.7/dist-packages/pythonforandroid/toolchain.py”, line 553, in init getattr(self, args.subparser_name.replace(‘-’, ‘_’))(args) File “/usr/local/lib/python3.7/dist-packages/pythonforandroid/toolchain.py”, line 150, in wrapper_func build_dist_from_args(ctx, dist, args) File “/usr/local/lib/python3.7/dist-packages/pythonforandroid/toolchain.py”, line 191, in build_dist_from_args build_recipes(build_order, python_modules, ctx) File “/usr/local/lib/python3.7/dist-packages/pythonforandroid/build.py”, line 633, in build_recipes recipe.build_arch(arch) File “/usr/local/lib/python3.7/dist-packages/pythonforandroid/recipe.py”, line 867, in build_arch self.build_compiled_components(arch) File “/usr/local/lib/python3.7/dist-packages/pythonforandroid/recipe.py”, line 884, in build_compiled_components build_dir = glob.glob(‘lib.*’)[0] IndexError: list index out of range