I have the same problem with python 3.9

Dug a little more at the Python 3.3 vs 3.4 differences, and there was a change to default all file descriptors to be non-inheritable, including sockets (see the addition of F_DUPFD_CLOEXEC, and https://docs.python.org/3/library/os.html#fd-inheritance). So it’s looking like the socket gets closed when os.exec is called.

So a different invasive workaround within IntelliJ’s pydev code, is to just set the client socket to be inheritable: ~/Library/Application Support/IntelliJIdea2019.1/python/helpers/pydev/_pydev_bundle/pydev_comm.py

def start_client(host, port):
    """ connects to a host/port """
    pydevd_log(1, "Connecting to ", host, ":", str(port))

    s = socket(AF_INET, SOCK_STREAM)
    # FIX: Set to inheritable to handle changes in Python 3.4
    if hasattr(s, 'set_inheritable'):
        s.set_inheritable(True)

Hopefully IntelliJ will provide a fix of their own sometime soon, but if not, then maybe something on BlazePyDebugRunner that is able to handle the socket being closed and a new one being opened following the os.exec() call.

I’m still seeing this with python 3.6.8.

Spent a little time looking at the symlink issue, and it looks like there’s code for converting the symlink to the workspace file, but that it isn’t being used on the file path it receives from the pydev stack frame: https://github.com/bazelbuild/intellij/blob/89e0b25e5ab51b4cf0523a14f5e3e6a3528e99b1/python/src/com/google/idea/blaze/python/run/BlazePyPositionConverter.java#L50-L53

So maybe need to wrap the file path with a call to BlazePyPositionConverter.convertFilePath().

I think this is the IntelliJ upstream call into this method, when parsing the pydev stack frame: https://github.com/JetBrains/intellij-community/blob/be6247932aa9414ddf7831c0e3becba6940f4839/python/pydevSrc/com/jetbrains/python/debugger/pydev/ProtocolParser.java#L215

So the remaining issue on the Bazel plugin side of things, to open the workspace file instead of the bazel-out file, looks like it’s just an issue with symlink resolution.

pydev is passing the symlink path back to the IDE when it triggers the breakpoint. This happens within NetCommandFactory.make_thread_suspend_str() found in the same file as the socket workaround: ~/Library/Application Support/IntelliJIdea2019.1/python/helpers/pydev/_pydev_bundle/pydev_comm.py

Note that the function get_abs_path_real_path_and_base_from_frame returns a tuple of: (os.abspath(file), os.realpath(file), basename(file))

So just swapping the index used by pydev from 0 to 1 is another invasive workaround.

abs_path_real_path_and_base = get_abs_path_real_path_and_base_from_frame(curr_frame)

# Workaround: Changed from 0 to 1 to pass realpath instead of symlinks
my_file = abs_path_real_path_and_base[1]

if is_real_file(my_file):
    # if filename is Jupyter cell id
    # Workaround: Changed from 0 to 1 to pass realpath instead of symlinks
    my_file = norm_file_to_client(abs_path_real_path_and_base[1])

So with the previous hack of pydev start_client() and this one, the IntelliJ debugger appears to be working correctly.

It seems reasonable to have the IntelliJ team fix the socket issue introduced with Python 3.4, but the symlink issue feels like it probably needs to be handled within the Bazel plugin.

Still haven’t dug through this code, but I would hope there’s somewhere to resolve the symlink that it receives from pydev.

On Mac, there seems to be a couple issues at play with the os.execv call within Bazel’s python_stub_template.txt.

Digging around on the IntelliJ side of things, I encountered this issue with breakpoints not being triggered by IntelliJ following os.execv (note that my workaround for that still triggers SIGKILL in the Bazel scenario): https://youtrack.jetbrains.com/issue/PY-37960

I haven’t dug through BlazePyDebugRunner at all, but my wild guess would be something involving the timing of the exec call being interpreted as a hangup. Like maybe some socket or file handle getting closed (e.g. FD_CLOEXEC).

A not-so-great workaround is to set IntelliJ to use Python 3.3, which has some different os.execv behavior that doesn’t trigger any of these bugs. But if you’re using any Python 3.4+ syntax or features, then this won’t work for you.

If rules_python decides to implement their proposal for customizing python_stub_template.txt, that would allow a potential workaround by switching os.execv to subprocess.call: https://github.com/bazelbuild/rules_python/blob/master/proposals/2018-11-08-customizing-the-python-stub-template.md

For an invasive workaround as things currently stand, you can modify your IntelliJ pydev code: ~/Library/Application Support/IntelliJIdea2019.1/python/helpers/pydev/_pydev_bundle/pydev_monkey.py

Change this:

def create_execv(original_name):
    def new_execv(path, args):
        """
        os.execv(path, args)
        os.execvp(file, args)
        """
        import os
        if is_python_args(args):
            send_process_will_be_substituted()
        return getattr(os, original_name)(patch_path(path), patch_args(args))
    return new_execv

To this:

def create_execv(original_name):
    def new_execv(path, args):
        """
        os.execv(path, args)
        os.execvp(file, args)
        """
        from subprocess import call
        call(patch_args(args))
    return new_execv

The remaining downside with this approach for me, is that the debugger opens a new tab with the bazel-out version of the file you’re debugging.