runtime: DllNotFoundException after successful manual dlopen on MacOS

I am trying to embed Python 3.6 into my cross-platform C# app.

Since I want the app to rely on an existing Python installation, I let the user pick the Python interpreter, then preload its the dynamic library with LoadLibrary/dlopen, and rely on DllImport to use the preloaded library.

This approach works no problem on Windows and Linux (though in the later case I only tried with the library actually being in the load path), however on MacOS I get Unhandled Exception: System.DllNotFoundException: Unable to load shared library 'python3.6m' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libpython3.6m, 1): image not found

Here’s the repro example:

using System;
using System.Runtime.InteropServices;

namespace DynPreload
{
    class Program
    {
        static void Main(string[] args)
        {
            const string pythonLib = "/usr/local/opt/python36/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin/libpython3.6m.dylib";
            if (dlopen(pythonLib, DLOpenFlags.RTLD_NOW) == IntPtr.Zero) throw new Exception();
            Console.WriteLine("Preloaded!");
            Py_SetProgramName("42"); // <- this throws DllNotFoundException
        }

        enum DLOpenFlags {
            RTLD_NOW = 2,
        }
        [DllImport("dl")]
        static extern IntPtr dlopen(string path, DLOpenFlags flag);
        [DllImport("python3.6m")]
        static extern void Py_SetProgramName(string name);
    }
}

(to get Python 3.6 installed into the specified folder using brew, run brew install sashkab/python/python36)

This happens despite dlopen successfully preloading libpython3.6m.dylib in the second line.

TargetFramework is netcoreapp2.0

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 18 (13 by maintainers)

Most upvoted comments

but 1.5x difference is quite a lot in the numerical compute scenario, which the Python binding is often be used for.

@lostmsu I previously worked at The MathWorks (makers of MATLAB) and this was an argument we would hear a lot. What is being missed is the fact that the call should be noise relative to the work the function is actually doing. If during a perf measurement the dispatch of the call via some FFI is profound then more than likely the call isn’t justified in the first place and the overhead of doing the work in the calling language/runtime is probably warranted - if perf is the greatest concern.

What is the relative cost of the raw PInvoke vs. the cost of calling empty python function via the Python bindings?

I would expect that that the latter will be orders of magnitude more expensive that makes the cost of the raw PInvoke not something to worry about, ie. if the raw PInvoke became 5x faster by a miracle, you would not really notice that on the Python binding performace.

As for the GetDelegateForFunctionPointer approach. Is there a way to get rid of the delegate invocation overhead?

The invocation overhead of the delegate returned by GetDelegateForFunctionPointer is comparable to the DllImport PInvoke overhead.

I would recommend using the GetDelegateForFunctionPointer for what you are trying to do.