neuralcoref: binary incompatibility
I’m using the current spaCy from the master branch, and getting this error:
RuntimeWarning: spacy.tokens.span.Span size changed, may indicate binary incompatibility. Expected 72 from C header, got 80 from PyObject
I’m assuming this happens because span.pxd
has changed after the 2.1 release: https://github.com/explosion/spaCy/commits/master/spacy/tokens/span.pxd
I tried reinstalling with
pip install neuralcoref --no-binary neuralcoref
But the warning remains and the program crashes when I run nlp(doc)
:
Process finished with exit code -1073741819 (0xC0000005)
Any idea on how to fix this? I’m compiling spaCy from sources too, so I was hoping not to have to do the same for neuralcoref …
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 7
- Comments: 29 (2 by maintainers)
Yes, downgrading to 2.1.0 does seems to work
The
--no-binary
flag will tell pip to install thesdist
, rather than the binary wheel. The problem is that thesdist
of a Cython package like neuralcoref isn’t really the source: it already has some compilation — the Python-to-Cython processing stage.The problem is that the sdist includes .cpp files that were generated using the .pxd files of a particular spaCy version. These are basically headers that get copied into the .cpp source when you run Cython. Those headers specify a different size for some objects, e.g. Span.
So what’s needed is a new Neuralcoref release that pins against the new spaCy. As soon as the new sdist is generated, everything should work correctly — because Cython just needs to be rerun with the new spaCy. That’s why building from master works, as @svlandeg reports above.
Yep, when I build neuralcoref’s master branch (4.0) from source, in combination with the current spaCy master branch (2.1.6) from source, it does work.
@thomwolf: So just to summarize, I think there are two main issues discussed in this thread:
Using
neuralcoref
out-of-the-box will install the latestspaCy
version because ofspacy>=2.1.0
in therequirements.txt
, but the latest version of spaCy has changes in some of its core Cython files (such asspan.pxd
) so this raises incompatibility issues. For most users, downgrading to spaCy 2.1.0 will be fine for now, so the PR by @BramVanroy makes sense to address this issue as a sort of temporary fix: https://github.com/huggingface/neuralcoref/pull/186When you do want to use a newer spaCy version > 2.1.0, like I do, a
binary incompatibility
RuntimeWarning is thrown by neuralcoref after installing it withpip install neuralcoref
. According to the docs, this should be fixed by installing withpip install neuralcoref --no-binary neuralcoref
instead, but that doesn’t seem to fix it for me. The docs do mention this as a fix for a warning aroundspacy.strings.StringStore size changed
while I might add that the current incompatibility is caused byspacy.tokens.span.Span
. Then when I instead buildneuralcoref
’s currentmaster
branch from source, it does work. So I feel like the issue is with thepip install neuralcoref --no-binary neuralcoref
command.Does not fix the issue for me. Going to try to build from sources…
Edit: compiling from source cleared up the error.
That fixes it for me.
How is Java involved in this?
Yes, this is not fixed because I currently don’t have a clear understanding of the root of the problem and how way to fix it. I will try to discuss with @honnibal and @ines and see if we can work out the source of the problem.