pyjulia: PyJulia does not work with Julia 1.0 & 0.7 with Python installed via Conda/Ubuntu/(what else?)

This problem is not going to go away, Debian is not going to change it and lots and lots of people are going to try this as debian distros are the most popular (though I prefer Arch I still have to use them often myself). My first question is how other languages solve this problem. We can’t be the first people ever to want to use a dynamically linked Python executable. It would be good to see how Scala and Java solve this, though I haven’t started looking yet.

I think at a minimum what we need is a very clear set of the simplest possible steps that one would take to get pyjulia running on debian in a very visible location right in the packages readme or documentation. Thanks to @tkf to taking the first steps to this with his pyenv setup, I’m going to try it myself when I get a chance.

I think there are three ways to use Python with Julia 1.0 on Ubuntu/Debian/etc. at the moment:

• If you just want to use your Python module, you can import it from PyCall.jl and call the functions in it. See, e.g., https://github.com/tkf/IPython.jl/blob/d5b27ac6e38f1b96ba42099da1098975cdd8b9be/src/core.jl#L31 and https://github.com/tkf/IPython.jl/blob/d5b27ac6e38f1b96ba42099da1098975cdd8b9be/src/core.jl#L20 in IPython.jl/src/core.jl.
• If you just want an interactive Python session, you can use my IPython.jl to drop into IPython through Julia REPL.
• If you want fully functional PyJulia in Python, one way to do it is though docker. Check out: https://github.com/tkf/docker-ipyjulia

Will this issue be solved in near future

I think I know how to fix it #173. But likely not in next weeks. I can’t grant anything 😃 If I were to do it I want to do it after dropping Julia 0.6 (which requires PyJulia to be released soon).

I merged a workaround for statically linked Python #200. Basically, you “just” need to run python-jl script.py (notice the -jl suffix) instead of python script.py if you use Debian or conda. You can try it by:

pip install https://github.com/JuliaPy/pyjulia/archive/master.zip#egg=julia

Or you can switch to Arch Linux 😆

It uses --enable-shared: https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/python&id=68e188a1f975dcc6798072bcda4ef35abfe162ed#n57

What’s the status of this issue? I am trying to run PyJulia in Ubuntu 22 but python is statically linked.

It works in Arch Linux 😃

IIRC, the official python docker works. It’s a bit old, but here is an example for setting up PyJulia in Docker: https://github.com/tkf/docker-pyjulia

Also, just FYI, I started working on compile-time preference integration for PyCall https://github.com/JuliaPy/PyCall.jl/pull/835. If it works as I expected, PyJulia should just work without any hacks in Julia 1.6 and above. Obviously, there are still some unknowns as I haven’t finished implementing all. So, let’s see 😉

It’s a bit strange. 3.7.0 from conda-forge is statically linked:

$ conda create --prefix conda-forge-python conda-forge::python
...
The following NEW packages will be INSTALLED:
...
    python:          3.7.0-h5001a0f_4        conda-forge
...
$ conda-forge-python/bin/python
Python 3.7.0 | packaged by conda-forge | (default, Sep 30 2018, 14:56:18)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
$ ldd conda-forge-python/bin/python
        linux-vdso.so.1 (0x00007ffec5dce000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f86eb290000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f86eb28b000)
        libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f86eb286000)
        librt.so.1 => /usr/lib/librt.so.1 (0x00007f86eb27c000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f86eb0f7000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f86eaf33000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f86eb2e4000)
$ conda-forge-python/bin/python -c 'from julia.find_libpython import linked_libpython; print(linked_libpython())'
None

But conda-forge::python=3.6 is dynamically linked.

$ rm -r conda-forge-python
$ conda create --prefix conda-forge-python conda-forge::python=3.6
...
The following NEW packages will be INSTALLED:
...
    python:          3.6.6-h5001a0f_3        conda-forge
...
$ ldd conda-forge-python/bin/python
        linux-vdso.so.1 (0x00007ffff0f7e000)
        libpython3.6m.so.1.0 => /home/takafumi/repos/pyjulia/conda-forge-python/bin/../lib/libpython3.6m.so.1.0 (0x00007f1f758a8000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f1f75856000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f1f75851000)
        libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f1f7584c000)
        librt.so.1 => /usr/lib/librt.so.1 (0x00007f1f75842000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f1f756bd000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f1f754f7000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f1f75de7000)
$ conda-forge-python/bin/python -c 'from julia.find_libpython import linked_libpython; print(linked_libpython())'
/home/takafumi/repos/pyjulia/conda-forge-python/lib/libpython3.6m.so.1.0

Yep, README is the only blocker before the release ATM.

@tkf ，thanks for your awesome work. The new package works on Ubuntu 16.04 with anaconda. It really helps.

There are some minor suggestions (or some misunderstanding of me).

third-party packages need to be re-compiled/re-installed

for test purpose, I test from julia import NPZ, while NPZ is already added in original julia environment. But it failed.

and the following code works:

from julia import Pkg
Pkg.add("NPZ")
#... some install outputs
from julia import NPZ
data = NPZ.npzread("path")

It seems that pyjulia and julia of system is using different package cache? I think re-install packages is not a problem and it might be a valuable feature. I only suggest add some doc for this

convenient broadcast call

broadcast is commonly used in julia, but Main.abs.([1,2,3]) is a SyntaxError in python. in README.md of pyjulia, there is

Main.eval("sin.(xs)")

which seems that pyjulia will not expose broadcast method to python level. I’m not sure if I am right, but if it is true, I’m thinking is it possible to adding some special syntax like Main.abs.broadcast([1,2,3]) or in short Main.abs.b([1,2,3]) (although may introduce some confusing to sum_b for python version sum!).

InterruptException() handling

When I press ctrl-C in python REPL, there would be a small problem. If I’m using a pure python REPL, it should work like this

>>> # presss ctrl-C here
KeyboardInterrupt
>>> 

But in a python REPL which loaded julia, it would work like this:

>>> # press ctrl-C here

and nothing happens, after that if I enter next command

>>> # Ctrl - C, and nothing happens
>>> Main.xs = [1,2,3] # Enter, change line and type new command and Enter I got:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 140, in __setattr__
    self._julia.eval(setter)(value)
  File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 606, in eval
    ans = self._call(src)
  File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 562, in _call
    self.check_exception(src)
  File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 589, in check_exception
    .format(exception, src))
julia.core.JuliaError: Exception 'InterruptException' occurred while calling julia code:

            PyCall.pyfunctionret(
                (x) -> eval(Main, :(xs = $x)),
                Any,
                PyCall.PyAny)

It seems the InterruptException is send to julia in one delayed statement?