pyodide: numpy fails to import

I can’t import numpy, has anyone ever had this problem? am i doing anything wrong? Browser: Chrome Version 67.0.3396.87 (Official Build) (64-bit) My code


languagePluginLoader.then(() => {
    // pyodide is now ready to use...
   pyodide.loadPackage('numpy').then(() => {
        pyodide.runPython('import numpy as np');
       
  });
});

the error

Error: Traceback (most recent call last):
  File "/lib/python3.6/site-packages/numpy/core/__init__.py", line 16, in <module>
    from . import multiarray
ImportError: Could not evaluate dynamic lib: /lib/python3.6/site-packages/numpy/core/multiarray.so
RangeError: WebAssembly.Compile is disallowed on the main thread, if the buffer size is larger than 4KB. Use WebAssembly.compile, or compile on a worker thread.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/lib/python3.6/site-packages/numpy/__init__.py", line 142, in <module>
    from . import add_newdocs
  File "/lib/python3.6/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/lib/python3.6/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/lib/python3.6/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/lib/python3.6/site-packages/numpy/core/__init__.py", line 26, in <module>
    raise ImportError(msg)
ImportError: 
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

Original error was: Could not evaluate dynamic lib: /lib/python3.6/site-packages/numpy/core/multiarray.so
RangeError: WebAssembly.Compile is disallowed on the main thread, if the buffer size is larger than 4KB. Use WebAssembly.compile, or compile on a worker thread.


    at _hiwire_throw_error (https://iodide.io/pyodide-demo/pyodide.asm.js:4:34764)
    at wasm-function[330]:1321
    at wasm-function[333]:621
    at Object.Module.__runPython (https://iodide.io/pyodide-demo/pyodide.asm.js:4:1853747)
    at Object.Module.runPython (https://iodide.io/pyodide-demo/pyodide.asm.js:4:37861)
    at t (https://iodide.io/dist/iodide.pyodide-20180605.js:12:191465)
    at https://iodide.io/dist/iodide.pyodide-20180605.js:110:12423
    at dispatch (https://iodide.io/dist/iodide.pyodide-20180605.js:46:27803)
    at code.cellType.p.then (https://iodide.io/dist/iodide.pyodide-20180605.js:12:191240)

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 24

Most upvoted comments

When you pass an array from Javascript to Python, it becomes a raw memory pointer, represented as a Uint8ClampedArray. To get the type back, you need to use np.frombuffer (which initializes a numpy array from existing memory), rather than np.array, which would interpret each byte as an array element. So try:

np.frombuffer(buffer, dtype=float32)

(This is the sort of low-level detail that we need to hide inside a helper library at some point – that just doesn’t exist yet…)

mdboom on Jun 26, 2018

Thanks for your patience with the rough edges on this very new project.

I’m not surprised to hear that Chrome is slower than Firefox – that’s been my experience thus far. It also is less memory efficient since at the moment we are force to load all dynamic libraries whether they are used or not in Chrome. Plus, we just got it working at all yesterday 😃

But, in general, a slowdown of 5x is not outside the range I’ve seen in my own benchmarking. See http://droettboom.com/blog/2018/04/11/profiling-webassembly/

That said, there are a few places here that might help (though it’s hard to say by how much without measuring):

Create a function in Python, import that to Javascript (using pyodide.pyimport(...)) and just call it. That will prevent the need to parse the Python each time.
This code is converting the audioBuffer array to a string and back when it’s passed from Javascript to Python. Creating a Python function and calling it should convert the buffer in a much more efficient way. Going even further, you could place the audioBuffer on the C/wasm heap and then just create the Numpy array to point at that, avoiding the copy altogether. I keep meaning to create some helper functions for that, but haven’t got to it yet. Are you able to share your work thus far (including the Javascript side) so I could use it as a starting point?

mdboom on Jun 26, 2018

See docs/new_packages.md for info on building new packages. Pure python tends to be pretty straightforward, but things with C or Cython extensions can be trickier.

mdboom on Jun 26, 2018

This looks like the Chrome-incompatibility issue in #22 and #55 which was just merged and deployed a few hours ago. Would you mind clearing your browser cache, trying again and reporting back?

mdboom on Jun 25, 2018